liquibase-ddm-ext library

🌐 This document is available in both English and Ukrainian. Use the language toggle in the top right corner to switch between versions.

1. Problem statement

We use Liquibase to create a registry physical data model for PostgreSQL.

By default, Liquibase supports objects deployment and versioning in DB, meaning the creation/deletion of tables, creation of connections between these tables, creation of views, configuration of constraints, etc.

For this purpose Liquibase has a number of constructions — change types, each one defining a certain version of changes to the DB, and is formed with XML-tags. For example, <createTable>, <dropTable>, etc.

Within the Registry Platform, Liquibase is used as the sole instrument for operations with physical data model in PostgreSQL, and its standard functionality is insufficient to cover all the possible scenarios of work with registry regulations.

For this reason, the following solutions have been developed:

  • liquibase-ext-schema Liquibase scheme that expands the standard Liquibase tags and attributes list with new Platform-specific ones

  • liquibase-ddm-ext Liquibase extensions library that includes program processing of tags and attributes added in liquibase-ext-schema, and modifies the behaviour for default tags and attributes

2. General description

liquibase-ddm-ext is a library that expands Liquibase functionality, and is responsible for the processing of XML-tags and attributes that may be used in registry regulations modeling.

This library adds the processing of Platform-specific new tags and attributes, and also changes the processing logic for some of the tags included in Liquibase by default.

3. Implementation principles

The extension uses all the main principles provided to Liquibase developers and users (details), and uses the mechanisms provided to extensions developers (examples of other extensions created by Liquibase developers for open use: https://github.com/orgs/liquibase/repositories).

4. Usage scenarios

The solution is used within regulations publication pipeline (registry-regulation-publication-pipeline) on data model deployment stage (create-schema stage of the registry-regulations-data-model pipe) for the filling of registry DB scheme, and creation of the metadata required to generate data-model services.

5. Adjacent components

  • registry-regulation-publication-pipeline - calls the Liquibase-utility on regulations publication stage

  • dataplatform-jenkins-agent - imports the library to pom.xml for further usage by regulations publication pipeline

  • liquibase-ext-schema - extended XSD-scheme of Liquibase tags and attributes that may be used in Platform registry development. It is included in all regulations files processed by Liquibase using liquibase-ddm-ext

  • service-generation-utility - generates the registry-model, registry-rest-api, registry-kafka-api, and registry-soap-api services, using database scheme, filled by Liquibase using liquibase-ddm-ext

6. List of extensions

Tag name

Included in standard Liquibase

Description

addColumn

yes

extends the logic of Liquibase-tag for adding columns to existing tables

createAnalyticsIndex

no

allows for the creation of an index for a previously created view on a DB replica

createAnalyticsView

no

for the creation of analytical view on a DB replica

createCompositeEntity

no

for the creation of metadata required to generate the endpoint for storing of several entities within one transaction, in registry-rest-api

createDomain

no

for the creation of user data types (domains)

createMany2Many

no

for the creation of many-to-many inter-table connections, used for storing and receiving data

createSearchCondition

no

for the creation of search condition from the DB with certain parameters

createSimpleSearchCondition

no

for the creation of a simple search condition (search by one field in one table)

createTable

yes

extends the logic of Liquibase-tag for new table creation

createType

no

for the creation of ENUM data types

distributeTable

no

for the creation of distributed tables

dropAnalyticsView

no

for the deletion of analytical view from the DB

dropColumn

yes

рextends the logic of Liquibase-tag for the deletion of a column from a table in DB

dropDomain

no

for the deletion of user data types (domains)

dropSearchCondition

no

for the deletion of search conditions from DB

dropType

no

for the deletion of ENUM data types

exposeSearchCondition

no

for the creation of metadata required to generate the endpoint for search conditions, accessible from external systems, in registry-rest-api

grantAll

no

for the provision of rights for all analytical views

grant

no

for the provision of rights for a separate view

makeObject

no

for the defining of a existing table as an object (added with links subject table, which was previously created)

modifyDataType

yes

extends the logic of Liquibase-tag for the changing of data type in table column

partialUpdate

no

for the creation of metadata required to generate partial entity update endpoints in registry-rest-api (for certain defined columns)

rbac

no

for the creation of metadata required to control the access to registry-rest-api endpoints for the defined roles

referenceTable

no

for the creation of reference-table

renameColumn

yes

extends the logic of Liquibase-tag for table column renaming

revokeAll

no

for the deletion of rights for all analytical views

revoke

no

for the deletion of rights for a separate view

rls

no

for the application of Row-Level Security rules to work with data in a table

tableReadParameters

no

or the creation of metadata required to generate proper data read requests in registry-rest-api (with/without attached entities, etc.)

truncateLocalDataAfterDistributingTable

no

for local data deletion after the creation of a distributed table

undistributeTable

no

for the cancellation of table distribution function

7. Diagrams

In the general process of data model deployment by the regulations publication pipeline, the execution of required Liquibase-scripts by the extension is performed on the create-schema stage.

datamodel

Example of a pipe for the publication of data model in Jenkins. liquibase-ddm-ext is called on the create-schema stage

data model pipe

The diagram that displays the approximate structure of packets in an extension, and the main points that comprise the extension of Liquibase default functionality (AbstractChange, CreateTableChange classes, etc.)

classes

8. DB

Besides the creation of DB scheme, liquibase-ddm-ext also fills the tables with metadata required for the correct generation of registry-services.

Tables where the metadata goes: ddm_liquibase-metadata, ddm_rls_metadata, ddm_role_permission.

8.1. ddm_liquibase_metadata

Table for metadata that doesn’t require a structure described in a separate table.

8.1.1. Table structure

CREATE TABLE public.ddm_liquibase_metadata (
    metadata_id INTEGER GENERATED BY DEFAULT AS IDENTITY NOT NULL,
    change_type TEXT NOT NULL,
    change_name TEXT NOT NULL,
    attribute_name TEXT NOT NULL,
    attribute_value TEXT NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW() NOT NULL,
    CONSTRAINT pk_ddm_liquibase_metadata PRIMARY KEY (metadata_id)
);

8.1.2. Table columns function

Column name

Function

Example

change_type

metadata type

search_condition

change_name

the object that owns the metadata

search conditions names

attribute_name

name of one attribute set by change

pagination

attribute_value

value for the attribute

limit

8.2. ddm_rls_metadata

The table that contains information on Row-Level Security rules that must be applied to registry-rest-api requests.

8.2.1. Table structure

CREATE TABLE public.ddm_rls_metadata (
    rls_id INTEGER GENERATED BY DEFAULT AS IDENTITY NOT NULL,
    name TEXT NOT NULL,
    type TEXT NOT NULL,
    jwt_attribute TEXT NOT NULL,
    check_column TEXT NOT NULL,
    check_table TEXT NOT NULL,
    CONSTRAINT pk_ddm_rls_metadata PRIMARY KEY (rls_id)
);

8.2.2. Table columns function

Column name

Function

Example

name

name of rule change

write_rls_katottg

type

operation to which the rule will apply

write

jwt_attribute

jwt-token attribute, where all the information required for checking is contained

katottg_jwt_attr

check_table

table, which the RLS is applied to

katottg_table

check_column

table column, where the values will be checked for the RLS

katottg_value

8.3. ddm_role_permission

The table that contains information for RBAC data access parameters

8.3.1. Table structure

CREATE TABLE public.ddm_role_permission (
    permission_id INTEGER GENERATED BY DEFAULT AS IDENTITY NOT NULL,
    role_name TEXT NOT NULL,
    object_name TEXT NOT NULL,
    column_name TEXT,
    operation TYPE_OPERATION NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW() NOT NULL,
    CONSTRAINT pk_ddm_role_permission PRIMARY KEY (permission_id)
);

8.3.2. Table columns function

Column name

Function

Example

role_name

user role that is being configured

officer

object_name

object, access to which is being restricted

table name, or search condition name

column_name

an attribute of the object, access to which is being restricted

table column, or search condition column

operation

operation type (read, write, etc.)

S / I / U / D