Business processes execution history

🌐 This document is available in both English and Ukrainian. Use the language toggle in the top right corner to switch between versions.

1. General context

The Business Process Management Service (BPMS) keeps a minimal set of necessary data on the state of running business process (BP) instances inside the database. We provisionally call this group of dedicated tables the Runtime Database.

Additionally, to meet audit requirements, the system creates a separate audit log of meaningful events we call the History Event Stream. By default, this event stream is saved to the History Database group of database tables.

The HistoryLevel setting controls the amount and level of events generated. HistoryLevel is defined using the camunda.bpm.history-level property.

The camunda.bpm.history-level property is defined once during the initial run of the Camunda application. To change this property, you also need to change the history level in the Camunda database:

UPDATE ACT_GE_PROPERTY SET VALUE_ = ?
WHERE NAME_ = 'historyLevel';
The camunda.bpm.history-level property can have the following properties:

  • NONE (VALUE_ = 0) — No history events are logged into the database, which minimizes impact on performance.

  • ACTIVITY (VALUE_ = 1) — Meaningful history events are logged for these objects: PROCESS, ACTIVITY, TASK.

  • AUDIT (VALUE_ = 2) — In addition to the events provided by the ACTIVITY history level, BP variables events are logged.

  • FULL (VALUE_ = 3) — In addition to the events provided by the AUDIT history level, BP variables change history is logged. Not recommended due to maximum impact on performance.

Saving history data to the History Database works synchronously, while the amount and level of events generated is controlled by HistoryLevel. Note that history data accumulation is not limited by default.

The following sequence diagram presents the algorithm of logging history events when running BPs:

Diagram

2. Anti-patterns of history usage

Several anti-patterns of history usage directly influence the Business Process Management Service performance:

  • Using History Database for long-term storage and further querying.

  • Setting HistoryLevel to a redundant level of events logging leads to a considerable increase in the amount of synchronous storage operations and volume of history data.

  • Having no control over the accumulation of history data in the History Database.

  • Using history data from the History Database to serve operational scenarios of user interactions through the cabinet.

  • Using History Database to serve scenarios of viewing history data through the cabinet.

3. Principles built into the solution design to support proper history usage

  • Implementing operational and history data usage scenarios separately on the level of individual components and databases.

  • Setting HistoryLevel to the minimal level of logging sufficient for administrators and support engineers to maintain the system.

  • Enabling automatic history cleanup to limit the accumulation of business processes run history data in the BPMS database.

  • Limiting history data’s time to live (TTL) to the corresponding BP execution time when providing this data to support engineers.

  • Generating a separate stream of meaningful BP run history events and publishing them asynchronously through the Kafka message broker for further processing and storage.

  • Processing BP history events messages from the Kafka message broker and saving them to a separate Process Execution History Database in a denormalized form.

  • Making sure history data from the History Database is not used as utility data for operational scenarios.

  • Implementing history data viewing scenarios using the Process Execution History Database.

4. Solution technical design

The following diagram presents Platform services involved in the implementation and their interactions. The diagram also outlines aspects that are important to consider during the implementation.

bpm history

4.1. Components of history maintenance

4.1.1. Publishing history events

To minimize the impact on the business processes performance and generate a separate history data storage, we need to implement the Process Engine Plugin with the Process History Event Publisher component that will handle events with AUDIT history level from the BPMN Core Engine and publish them to a separate topic of the Kafka message broker.

We need to consider implementing a custom level of history events logging to publish messages to Kafka using the following rules:

Resource Event type Resource identifier Save operation

Process Instance

START, UPDATE, END

-

INSERT OR UPDATE BPM_HISTORY_PROCESS BY PROCESS_INSTANCE_ID

Task Instance

CREATE, UPDATE, COMPLETE

-

INSERT OR UPDATE BPM_HISTORY_TASK BY ACTIVITY_INSTANCE_ID

Variable Instance

CREATE, UPDATE, DELETE

System variables: sys-var-process-completion-result, sys-var-process-excerpt-id

UPDATE BPM_HISTORY_PROCESS BY PROCESS_INSTANCE_ID

4.1.2. Saving published historical events

To save business processes run history, we need to implement the User Process History Event Subscriber component that will handle the messages from the history events topic of the Kafka message broker and save them into a separate database in a denormalized form.

4.1.3. History data access API

To give cabinet users access to their personal history of business processes and tasks, we need to implement a separate User Process History Management component that will provide the API necessary to support historical querying by authenticated users.

4.2. System components interaction

The following sequence diagram presents the algorithm of logging history events when running a BP:

Diagram

5. Get user’s business processes history data API

5.1. Get currently initiated business processes

Data access is limited to the requests from authenticated users.

The user ID obtained from the request’s X-Access-Token HTTP header is mandatory when generating a data sample using the startUserId field.

When generating a business processes data sample request, a criterion for obtaining a top-level BP is added unconditionally (SUPER_PROCESS_INSTANCE_ID IS NULL).

GET /api/process-instances

Parameter Type Request part Optional Default value Description

X-Access-Token

JWT

HTTP header

No

-

User access token

offset

Number

Request parameter

Yes

0

Record offset

limit

Number

Request parameter

Yes

10

Records limit

sort

String

Request parameter

Yes

desc(endTime)

Field to sort by and sort order.

Example: asc(<field>) / desc(<field>)
Sample response
[
    {
      "processInstanceId":  "",
      "superProcessInstanceId": "",
      "processDefinitionId": "",
      "processDefinitionKey": "",
      "processDefinitionName": "",
      "businessKey": "",
      "startTime": "",
      "startUserId": "",
      "status": {
        "code": "",
        "title": ""
      }
    }
]
Table 1. Error codes
Code Description

200

OK with the request results in the message body

400

Incorrect request (wrong data format)

401

Authentication error (no access token)

500

Server-side error when processing the request
Sequence diagram of the request to business processes current data
Figure 1. Sequence diagram of the request to business processes current data
Table 2. Statuses localization
Status Localized status (Ukrainian)

ACTIVE

У виконанні

PENDING

Очікує виконання задачі

SUSPENDED

Призупинено адміністратором

5.2. Get initiated business processes history

Data access is limited to the requests from authenticated users.

The user ID obtained from the request’s X-Access-Token HTTP header is mandatory when generating a data sample using the startUserId field.

When generating a business processes data sample request, a criterion for obtaining a top-level BP is added unconditionally (SUPER_PROCESS_INSTANCE_ID IS NULL).

GET /api/history/process-instances

Parameter Type Request part Optional Default value Description

X-Access-Token

JWT

HTTP header

No

-

User access token

offset

Number

Request parameter

Yes

0

Record offset

limit

Number

Request parameter

Yes

10

Records limit

sort

String

Request parameter

Yes

desc(endTime)

Field to sort by and sort order.

Example: asc(<field>) / desc(<field>)
Sample response
[
    {
      "processInstanceId":  "",
      "superProcessInstanceId": "",
      "processDefinitionId": "",
      "processDefinitionKey": "",
      "processDefinitionName": "",
      "businessKey": "",
      "startTime": "",
      "endTime": "",
      "startUserId": "",
      "excerptId": "",
      "status": {
        "code": "",
        "title": ""
      }
    }
]
Table 3. Error codes
Code Description

200

OK with the request results in the message body

400

Incorrect request (wrong data format)

401

Authentication error (no access token)

500

Server-side error when processing the request
Sequence diagram of the request to business processes history data
Figure 2. Sequence diagram of the request to business processes history data
Table 4. Statuses localization
Status Localized status (Ukrainian)

completionResult != null

Значення completionResult

COMPLETED

Надання послуги завершено

EXTERNALLY_TERMINATED

Відмінено адміністратором

5.3. Get completed business process tasks history

Data access is limited to the requests from authenticated users.

The user ID obtained from the request’s X-Access-Token HTTP header is mandatory when generating a data sample using the assignee field.

GET /api/history/tasks

Parameter Type Request part Optional Default value Description

X-Access-Token

JWT

HTTP header

No

-

User access token

offset

Number

Request parameter

Yes

0

Record offset

limit

Number

Request parameter

Yes

10

Records limit

sort

String

Request parameter

Yes

desc(endTime)

Field to sort by and sort order.

Example: asc(<field>) / desc(<field>)
Sample response
[
    {
      "activityInstanceId":  "",
      "taskDefinitionKey": "",
      "taskDefinitionName": "",
      "processInstanceId": "",
      "processDefinitionId": "",
      "processDefinitionKey": "",
      "processDefinitionName": "",
      "startTime": "",
      "endTime": "",
      "assignee": ""
    }
]
Table 5. Error codes
Code Description

200

OK with the request results in the message body

400

Incorrect request (wrong data format)

401

Authentication error (no access token)

500

Server-side error when processing the request

6. Configuring history data in Business Process Management Service

6.1. Logging business processes history events

When working with the system, users may require the support team to investigate errors and find the reasons why the users' business processes have stopped. To fully utilize the Camunda Cockpit admin interface to monitor business process status and variables, we recommend setting the required level of history events logging using the camunda.bpm.database-history-level property.

The camunda.bpm.database-history-level property can have the following values:

  • NONE — No history events are logged into the database, which minimizes impact on performance.

  • ACTIVITY — Meaningful history events are logged for these objects: PROCESS, ACTIVITY, TASK.

  • AUDIT — In addition to the events provided by the ACTIVITY history level, BP variables events are logged.

  • FULL — In addition to the events provided by the AUDIT history level, BP variables change history is logged. Not recommended due to maximum impact on performance.

The following default settings are recommended:

  • camunda.bpm.history-level: AUDIT

  • camunda.bpm.database-history-level: ACTIVITY

The settings need to be adjusted depending on the system’s stability and the need to improve performance or increase the level of events detail.
To further improve performance, it is possible to provide a custom level of history events logging by implementing the TypeBasedHistoryLevel interface and registering it in the Process Engine configuration.
The camunda.bpm.history-level property should not be used to define the history events logging level because this property defines the level of generating history events, not the level of their filtering prior to processing. The camunda.bpm.database-history-level custom property should be used instead.

6.2. Automatic history cleanup

The suggested mechanism of business processes history cleanup is intended for process instances and has no impact on the metadata that belongs to the currently installed and outdated versions of Deployment. Should the need arise, removing outdated versions can be implemented separately.

To improve performance and reduce the accumulation of history data, it is necessary to apply the following settings to the Business Process Management Service. This implements the automatic process of deleting outdated data using the Removal-Time-based strategy:

Setting Value Description

historyCleanupEnabled

true

Enable history cleanup execution on a regular basis.

historyCleanupStrategy

removalTimeBased

Enable the Removal-Time-based history cleanup strategy (removal time = base time + TTL).

historyRemovalTimeStrategy

end

Configure base time to define removal time for BP history cleanup.

historyTimeToLive

P1D

Configure TTL to define removal time for BP history cleanup.

historyCleanupBatchWindowStartTime

20:00

Specify the start time of the batch window during which daily cleanup should run.

historyCleanupBatchWindowEndTime

22:00

Specify the end time of the batch window during which daily cleanup should run.

historyCleanupDegreeOfParallelism

1

Set the degree of parallel execution for history cleanup (the number of job executor threads).

historyCleanupBatchSize

500

Set the number of BP instances removed in one cleanup transaction.

7. Business processes history data model

When working with history data, there are two main scenarios of user interaction through the cabinet:

  • Getting the history of business processes started and finished by the user

  • Getting the history of user’s completed tasks

To optimize these requests, history data should be stored in a denormalized form in a separate storage:

  • BPM_HISTORY_PROCESS - Business processes history

  • BPM_HISTORY_TASK - Tasks history

A relationship between these tables was not defined on purpose. After denormalization both tables contain all the necessary attributes to serve historical requests and get data independently.
Diagram