Interim data of business processes

🌐 This document is available in both English and Ukrainian. Use the language toggle in the top right corner to switch between versions.

The current technical design is focused on the general aspects of implementing the requirements for temporary storage of intermediate data received from the user or generated within the execution of business processes.

1. Description of components intermediate data BP

Intermediate BP data is a logical group of data that belongs to a separate instance of a business process and consists of the following components:

Data received through task UI forms from users
Files that have been uploaded through the task UI forms by users
Signature data received through task UI forms from users
Data used as input when initiating a business process by the user
Data used as input when initiating a business process by external systems
Data of system signatures that were generated on derived data
Data that was generated as a result of interprocess interaction

Intermediate BP data is an integral part of the state of a business process instance, which is vital for its recovery in the event of failures and subsequent successful completion.

2. Scenarios for using intermediate data

Storage and use of intermediate BP data
Exchange of intermediate BP data during interservice interaction
Exchange intermediate BP data during interprocess interaction
Removal of intermediate BP data upon completion of business processes

3. Basic principles

Проміжні дані БП повинні зберігатися у сховищі, яке забезпечує вимоги надійності зберігання
Проміжні дані БП повинні зберігатися у сховищах, які є оптимальними у розрізі типів цих даних та сценаріїв їх використання задля забезпечення швидкодії системи
Сховище проміжних даних БП повинно забезпечувати вимоги відмовостійкості та масштабування
Проміжні дані БП є даними, по відношенню до яких необхідно забезпечити вимоги бекапування та відновлення
Цілісність зберігання проміжних даних БП забезпечується відсутністю можливостей адміністратора регламенту для внесення прямих змін, окрім сценаріїв, які надає система в автоматичному режимі
Взаємодія зі сховищем проміжних даних БП повинна бути реалізовано з використанням захищеного протоколу обміну даними
Проміжні дані БП повинні зберігатися у сховищі у зашифрованому вигляді
Сховище проміжних даних БП має експортувати всі необхідні метрики для його моніторингу та обслуговування

4. Technical solution design

This diagram shows the platform services involved in implementing the requirements and the interaction between them. In addition, important features that must be taken into account in the framework of implementation are depicted.

Two types of storage are used to save and exchange intermediate data of business processes:

Redis: Hash structure with data segregation by keyspaces
Ceph: "lowcode-file-storage" bucket for temporary storage of documents

4.1. System components and their purpose within the solution design

This section provides a list of system components that are involved or need to be changed/created as part of the implementation of functional requirements according to the technical design of the solution.

Component	Official title	Usage scenarios
Business processes execution service	bpms	- Saving and reading data and user signature obtained through UI-forms of business process tasks. - Storage of derived data and system signature. - Saving and reading payloads of messages formed during interprocess interaction. - Deletion of temporary data upon completion of business processes.
User tasks management service	user-task-management	- Storage of user and signature data obtained during the execution of business process tasks. - Reading user data obtained in the process of performing tasks of business processes
User processes management service	user-process-management	Saving the input data received from the user through the start UI form of the business process
Gateway service for integration with external systems	bp-webservice-gateway	Storage of input data required to initiate a business process when called by external systems
Digital documents service	digital-documents	- Storage of intermediate data / documents during the execution of business processes - Deletion of intermediate data / documents upon completion of business processes.
Service for requests for changes to registry data	registry-rest-api	- Obtaining primary user data with a superimposed signature. - Get the signature generated on derived data by the system
Service for servicing requests for excerpt generation	excerpt-service-api	- Obtaining primary user data with a superimposed signature. - Get the signature generated on derived data by the system
Distributed in-memory data storage	redis	Storage of intermediate data during the execution of business processes
Distributed object data storage	ceph	Storage of intermediate data / documents during the execution of business processes

Component

Official title

Usage scenarios

Business processes execution service

bpms

- Saving and reading data and user signature obtained through UI-forms of business process tasks.

- Storage of derived data and system signature.

- Saving and reading payloads of messages formed during interprocess interaction.

- Deletion of temporary data upon completion of business processes.

User tasks management service

user-task-management

- Storage of user and signature data obtained during the execution of business process tasks.

- Reading user data obtained in the process of performing tasks of business processes

User processes management service

user-process-management

Saving the input data received from the user through the start UI form of the business process

Gateway service for integration with external systems

bp-webservice-gateway

Storage of input data required to initiate a business process when called by external systems

Digital documents service

digital-documents

- Storage of intermediate data / documents during the execution of business processes

- Deletion of intermediate data / documents upon completion of business processes.

Service for requests for changes to registry data

registry-rest-api

- Obtaining primary user data with a superimposed signature.

- Get the signature generated on derived data by the system

Service for servicing requests for excerpt generation

excerpt-service-api

- Obtaining primary user data with a superimposed signature.

- Get the signature generated on derived data by the system

Distributed in-memory data storage

redis

Storage of intermediate data during the execution of business processes

Distributed object data storage

ceph

Storage of intermediate data / documents during the execution of business processes

4.2. Configuring interservice interaction policies

As part of the implementation of the requirements, it is necessary to add the appropriate network policies NetworkPolicy, which allow interaction for the following components:

bpms → redis
bpms → digital-documents
user-task-management → redis
user-process-management → redis
bp-webservice-gateway → redis
registry-rest-api → redis
excerpt-service-api → redis

Updating and removing outdated network access policies, if necessary, should be carried out on N+1 updates of the Platform version.

4.3. Structural diagram of system components

Ensuring standardization in working with intermediate data between Platform services, fixing contracts and code reuse is implemented through the introduction of the following modules:

ddm-form-data-storage
ddm-file-storage
ddm-bpm-message-payload-storage

This structural diagram shows the services of the platform and their interaction with temporary data stores using the service modules listed above.

To save data using the Redis Hash-structure, the approach of segregation of objects through Keyspaces-prefixes is used (<keyspace>:<original-key>):

bpm-form-submissions
bpm-message-payloads

At this stage, the <original-key> generation algorithm remains unchanged to ensure the correctness of data migration when switching from Ceph to Redis.

An example of a key pattern for the task data:

bpm-form-submissions:process/${processInstanceId}/task/{taskDefinitionKey}

An example of a key pattern for the body data of an interprocess communication message:

bpm-message-payloads:process-definition/{processDefinitionKey}/start-message/{uuid}

4.4. Data structure of bpm-form-submissions keyspace

bpm-form-submissions

{
  "x-access-token": "...",
  "data": {
    ...
  },
  "signature": "..."
}

4.5. Data structure of bpm-message-payloads keyspace

bpm-message-payloads

{
  "data": {
    ...
  }
}

4.6. Automatic deletion of intermediate business process data

The system must automatically remove intermediate data from the storage upon completion of the business processes (transition to the COMPLETED or EXTERNALLY_TERMINATED state) within which they were received or generated.

4.6.1. lowcode-file-storage Ceph-bucket

Files that have been uploaded through the task UI forms by users

4.6.2. bpm-form-submissions Redis keyspace

Data received through task UI forms from users
Signature data received through task UI forms from users
Data used as input when initiating a business process by the user
Data used as input when initiating a business process by external systems
Data of system signatures that were generated on derived data

4.6.3. bpm-message-payloads Redis keyspace

Data that was generated as a result of interprocess interaction

5. Fault-tolerant key-value data storage based on Redis Sentinel

Redis is used as a key-value storage of intermediate BP data, and fault tolerance is ensured by means of the Redis Sentinel mechanism.

Redis Sentinel is a distributed system that consists of N instances of Sentinel processes that interact with each other.

Redis Sentinel has the following features:

the fact of failure of the master node is confirmed by several instances of Sentinel, which form a quorum, which reduces the number of false positives
Sentinel itself is a fault-tolerant system that can perform its functions even if some Sentinel instances are down.

Redis Sentinel provides the following capabilities:

Monitoring - Sentinel monitors that instances of Redis-master and replica work correctly
Alerting - Sentinel provides the ability to send messages to the administrator in case of identification of failures of Redis instances
Automatic recovery - In the event that the Redis-master instance starts working incorrectly, Sentinel initiates the process of determining a new Redis-master instance and reconfiguring other Redis-replicas to interact with the new master.

Detailed information can be found in the official technical documentation Redis Sentinel.

5.1. Deploying Redis Sentinel

A separate Kubernetes operator https://github.com/spotahome/redis-operator [Redis Operator by Spotahome] is used to automate the deployment and management of the Redis Sentinel repository.

Deployment features:

Interaction with Redis Sentinel requires client authentication
Interaction with Redis requires client authentication
Data exchange between Sentinel and Redis instances is secured using TLS
Developed 3 possible Redis Sentinel deployment configurations depending on fault tolerance requirements and available resources:
- Minimal
- Recommended
- CI/CD

5.2. Redis Sentinel Configuration

Settings	Value	Description
sentinel.replicas	3	The number of Sentinel process instances
redis.replicas	2	The number of Redis-replica instances
sentinel.quorum	2	The number of Sentinel-processes, which is necessary to confirm the inoperability / unavailability of the Redis-master
sentinel.customConfig."down-after-milliseconds"	60000	The time, in milliseconds, that a Redis instance must be unavailable for Sentinel to consider it down
sentinel.customConfig."failover-timeout"	180000	The time in milliseconds used as a delay when confirming the unavailability of the Redis-master
sentinel.customConfig."parallel-syncs"	1	The number of Redis replicas that are simultaneously unavailable while reconfiguring to use a new Redis master in case of automatic recovery
sentinel.customConfig."min-replicas-to-write"	1	The minimum number of replicas available to the Redis master to accept data modification operations
sentinel.customConfig."min-replicas-max-lag"	10	The minimum time, in seconds, used as a replica unavailability indicator when performing asynchronous replication or availability checks

Settings

Value

Description

sentinel.replicas

The number of Sentinel process instances

redis.replicas

The number of Redis-replica instances

sentinel.quorum

The number of Sentinel-processes, which is necessary to confirm the inoperability / unavailability of the Redis-master

sentinel.customConfig."down-after-milliseconds"

60000

The time, in milliseconds, that a Redis instance must be unavailable for Sentinel to consider it down

sentinel.customConfig."failover-timeout"

180000

The time in milliseconds used as a delay when confirming the unavailability of the Redis-master

sentinel.customConfig."parallel-syncs"

The number of Redis replicas that are simultaneously unavailable while reconfiguring to use a new Redis master in case of automatic recovery

sentinel.customConfig."min-replicas-to-write"

The minimum number of replicas available to the Redis master to accept data modification operations

sentinel.customConfig."min-replicas-max-lag"

The minimum time, in seconds, used as a replica unavailability indicator when performing asynchronous replication or availability checks

5.3. Configuration of client services

Redis Sentinel Connection Configuration Example for the Registry Change Request Service

lowcode-form-data-storage:
  type: redis
  backend:
    redis:
      keyspace: 'bpm-form-submissions'
      username: ${redis-secret:redis.username} [redis data node auth]
      password: ${redis-secret:redis.password} [redis data node auth]
      sentinel:
        master: <sentinel-redis-master-group-name>
        nodes: <redis-sentinel-service>:<redis-sentinel:port>
        username: ${redis-secret:redis.sentinel.username} [sentinel auth]
        password: ${redis-secret:redis.sentinel.password} [sentinel auth]
lowcode-file-storage:
  type: ceph
  backend:
    ceph:
      http-endpoint: ${lowcode-file-ceph.http-endpoint}
      access-key: ${lowcode-file-ceph.access-key}
      secret-key: ${lowcode-file-ceph.secret-key}
      bucket: ${lowcode-file-ceph.bucketName}
datafactory-form-data-storage:
  type: ceph
  backend:
    ceph:
      http-endpoint: ${datafactoryceph.http-endpoint}
      access-key: ${datafactoryceph.access-key}
      secret-key: ${datafactoryceph.secret-key}
      bucket: ${datafactoryceph.bucketName}

Example configuration of connection to Redis Sentinel for Business Process Execution Service

storage:
  form-data:
    type: redis
    keyspace: 'bpm-form-submissions'
  message-payload:
    type: redis
    keyspace: 'bpm-message-payloads'
  backend:
    redis:
      username: ${redis-secret:redis.username} [redis data node auth]
      password: ${redis-secret:redis.password} [redis data node auth]
      sentinel:
        master: <sentinel-redis-master-group-name>
        nodes: <redis-sentinel-service>:<redis-sentinel:port>
        username: ${redis-secret:redis.sentinel.username} [sentinel auth]
        password: ${redis-secret:redis.sentinel.password} [sentinel auth]