Saving UI form schemes and validating user data

🌐 This document is available in both English and Ukrainian. Use the language toggle in the top right corner to switch between versions.

1. Functional scenarios

  • Getting the data schemes of UI forms to display in the citizen portals.

  • Validation of data entered by users using the UI forms of tasks according to the configured rules.

  • Validation of files uploaded by users using the UI forms of tasks according to the configured rules.

  • Validation of data used as input parameters to initiate business processes by external systems.

2. Basic principles and motivation for redesigning

  • Providing for a unified approach to the implementation of cross-cutting requirements imposed on the registry Edge services with a publicly available API, through the use of a single technology stack and, as a result, the reuse of a standard set of techniques, libraries, etc.

  • Ensuring a high level of сohesion in the design of the platform services.

  • Standardization of types of data storages operated by the Platform.

  • The data storage type must meet non-functional requirements for operations with this data.

3. Scope of services redesigning

As part of the redesign, the following new components must be introduced to implement the functional scenarios:

  • UI form provider service (form-schema-provider)

  • UI form data validation service (form-submission-validation)

Each component must be integrated into the registry ecosystems and fully comply with all the cross-cutting requirements.

Components to be changed:

  • User processes management service (user-process-management).

  • User tasks management service (user-task-management).

  • Digital documents service (digital-documents).

  • Regulations publication pipeline (jenkins).

Components to be replaced/removed after data migration:

  • UI forms provider service (form-management-provider).

  • UI form scheme data storage (form-management-provider-db)

4. Target system design

The following diagram shows the target state of the platform services and the interaction between them. In addition, it shows essential features that need to be taken into account as part of the implementation.

form-schema-provider

4.1. System components and their designation in the target design

Component Name Use cases

UI form data validation service

form-submission-validation

Validation of data and files, entered by users using the UI form of business process tasks, for compliance with the configured rules

UI forms provider service

form-schema-provider

Processing requests for UI form schemes from user portals

User tasks management service

user-task-management

Storage of the user data and signatures obtained during the execution of business processes and validation for compliance with configured rules

User processes management service

user-process-management

Storage of the input data received from the user using the initial UI form of the business process and validation for compliance with configured rules

Gateway service for integration with external systems

bp-webservice-gateway

Storage of the input data required to initiate a business process when called by external systems

Digital documents service

digital-documents

Storage and validation of files uploaded by users

Regulations publication pipeline

jenkins

Publication of changes in the UI form schemes of business process tasks to the target registry environment

Distributed in-memory data warehouse

redis

Storing/caching UI form scheme data

4.2. Configuration of inter-service policies

As part of the requirements implementation, it is required to add the appropriate NetworkPolicy policies that allow interoperability for the following components:

  • kong → form-schema-provider

  • form-submission-validation → form-schema-provider

  • user-process-management → form-submission-validation

  • user-task-management → form-submission-validation

  • digital-documents → form-submission-validation

  • form-schema-provider → redis

4.3. UI form schemes provider service

This Java-servіs is responsible for processing requests for getting the UI form schemes for business process tasks. Redis Sentinel is used as the data store.

You can find more information about Redis Sentinel in the Fault-tolerant key-value data storage based on Redis Sentinel section.

4.3.1. Storing data of UI forms

UI forms are stored using the Redis Hash structure with the object segregation approach using the Keyspaces prefixes (<keyspace>:<key>).

bpm-form-schemas keyspace is used for the UI form schema objects.

Example of the key generation pattern for writing/reading an object:
bpm-form-schemas:{form-key}

4.3.2. API to access UI form schemes

Access to the API is possible only under a request from an authenticated user in the system.
4.3.2.1. Receiving UI-form scheme using ID [Public]
This API route is public and must be published for external access via a separate Kong Route.

GET /api/forms/{form-key}

Parameter Type Request part Description

X-Access-Token

JWT

HTTP header

User access token

form-key

Text

Request parameter

Unique identifier of the UI form scheme
Response example
{
  "type": "form",
  "display": "form",
  "title": "Task form name",
  "name": "form-key",
  "path": "form-key",
  "components": [
  ]
}
Table 1. Error codes
Code Description

200

OK, returning the result of the UI-form scheme data

400

Incorrectly generated request

401

Authentication error (no access token)

404

No UI form scheme for the specified {form-key}

500

Server request processing error
4.3.2.2. Creating a new UI form scheme [Internal]
The purpose of the API rout is the service use by the Regulations publication pipeline to inform the data storage of the UI form schemes with data (redis). The rout is not available for external access via Kong.

POST /api/forms/

Parameter Type Request part Description

X-Access-Token

JWT

HTTP header

User access token
Example of the request body
{
  "type": "form",
  "display": "form",
  "title": "Task form name",
  "name": "form-key",
  "path": "form-key",
  "components": [
  ]
}
Table 2. Error codes
Code Description

201

Created, returning the result of the UI form scheme data

400

Incorrectly generated request

401

Authentication error (no access token)

500

Server request processing error
4.3.2.3. Making changes to an existing UI form scheme [Internal]
The purpose of the API rout is the service use by the Regulations publication pipeline to inform the data storage of the UI form schemes with data (redis). The rout is not available for external access via Kong.

PUT /api/forms/{form-key}

Parameter Type Request part Description

X-Access-Token

JWT

HTTP header

User access token

form-key

Text

Request parameter

Unique identifier of the UI form scheme
Example of the request body
{
  "type": "form",
  "display": "form",
  "title": "Task form name",
  "name": "form-key",
  "path": "form-key",
  "components": [
  ]
}
Table 3. Error codes
Code Description

200

OK, returning the result of the UI-form scheme data

400

Incorrectly generated request

401

Authentication error (no access token)

404

No UI form scheme for the specified {form-key}

500

Server request processing error

4.4. UI form data validation service

This NodeJS service is responsible for data validation based on the rules defined in the UI form scheme using the formio.js library.

In addition, it supports server validation for the files uploaded by users.

You can find out more details in the Form.IO documentation by following this link.

4.4.1. UI form data validation API

Access to the API is possible only as part of execution of a request by an authenticated user in the system. API routs are not available for external access via Kong and are used only by the internal registry services.

4.4.2. Data validation according to the rules defined in the UI form scheme (Internal)

POST /api/form-submissions/{form-key}/validate

Parameter Type Request part Description

X-Access-Token

JWT

HTTP header

User access token

form-key

Text

Request parameter

Unique identifier of the UI form scheme for which data validation is required
Example of the request form containing the UI form data
{
  "data": {
    "field-key1": "Joe",
    "field-key2": "joe@example.com",
    "field-key3": "123123123"
  }
}

  1. Example of a response in case of data validation errors

{
  "name": "ValidationError",
  "details": [
    {
      "message": "Name is required",
      "level": "error",
      "path": [
        "name"
      ],
      "context": {
        "validator": "required",
        "setting": true,
        "key": "name",
        "label": "Name",
        "value": ""
      }
    },
    {
      "message": "Edrpou must have at least 2 characters.",
      "level": "error",
      "path": [
        "edrpou"
      ],
      "context": {
        "validator": "minLength",
        "setting": 2,
        "key": "edrpou",
        "label": "Edrpou",
        "value": "1"
      }
    }
  ]
}
Table 4. Error codes
Code Description

200

OK

400

Incorrectly generated request

401

Authentication error (no access token)

422

Error of UI form scheme data validation

500

Server request processing error
After the redesign of services, it is necessary to redesign the validation API to ensure consistency of contracts on returning validation errors.

4.4.3. Data validation of individual fields according to the rules defined in the UI form scheme (Internal)

POST /api/form-submissions/{form-key}/fields/{field-key}/validate

Currently only the file field validation is supported. If {field-key} belongs to a field of another type, the system must return 501 (NOT_IMPLEMENTED).
Parameter Type Request part Description

X-Access-Token

JWT

HTTP header

User access token

form-key

Text

Request parameter

Unique identifier of the UI form scheme

field-key

Text

Request parameter

Unique field identifier within the UI form
Example of the request body with the data on the downloaded UI form file for validation
{
  "documentKey": "",
  "filename": "",
  "contentType": "",
  "size": 0
}
Example of a response for validation of file fields
{
  "traceId": "<trace-id>",
  "code": 422,
  "message": "The type of the downloaded file is not supported.",
  "details": []
}
Table 5. Error codes
Code Description

200

OK, returning the result of the UI form scheme data

400

Incorrectly generated request

401

Authentication error (no access token)

404

No UI form schema for the specified {form-key}

500

Server request processing error

501

System does not support this operation

4.5. Regulations publication pipeline

To fill the new bpm-form-schemas Redis keyspace repository of UI forms with data in the framework of the regulations publication, it is necessary to make changes to -Stage upload-form-changes using the internal API of the newly created UI forms provider service (form-schema-provider).

5. Backing up and restoration of the UI form scheme data

Backing up and restoration of the data stored in the Redis storage is performed according to the general procedure using a secure backup storage.

For details, see Registry backing up and restoration.

6. Migration of the UI form scheme data

Currently, the registry UI form schemes are stored in the MongoDB form-management-provider-db repository.

As part of the transition to the new version, it is required to fill the new bpm-form-schemas Redis keyspace storage with data for each registry.

6.1. Migration procedure, option # 1:

  • Get all files of the registry regulations UI forms schemes from the <registry-regulation>/forms directory.

  • Create corresponding records of the UI form schemes through the internal API of the UI forms provider service using the algorithm defined in UploadFormChanges.groovy of the corresponding step of the Regulations publication pipeline.

6.2. Migration procedure, option # 2:

  • Generate Gerrit MR with type changes: touch <<registry-regulation>/forms/*.json>

  • Integrate Gerrit MR into the master branch, thus initiate the launch of the "Regulations publication pipeline"

  • Wait for completion of the upload-form-changes step and check for entries in bpm-form-schemas Redis keyspace