Partial entity update in data factory

On this page:

1. Overview
2. Usage and configuration in a business process
3. API implementation
4. Data model level implementation

1. Overview

The Platform uses 4 main HTTP-methods for all the tables in Data Factory, which the services interact with: POST, GET, PUT and DELETE.

To update an entity in Data Factory, we use the PUT method and the corresponding Update entity in data factory* implemented delegate.

When using PUT, it is imperative to enter the values of all table fields in the request body, even those that don’t require updating. This behavior is default for PUT method.

Example 1. Example

For example, let’s take a table with 5 fields: First name, Last name, Middle name, Birth date, Work place. We need to update only the first field, which is First name.

When using PUT method, we must set values for each of the five fields of the object.

Otherwise, all unfilled fields will autofill with NULL.

partialUpdate functionality solves this proble. Partial update uses PATCH HTTP-method on the API level. This method ignores all unfilled fields, not defined in request body. It only processes the parameters that require updating.

To work with the PATCH method on the API level there is a dedicated endpoint that allows for the partial database updates (see API implementation).

On the business process level, a typical extension has been developed to interact with the API and allow for the sending of business data to the database — the ${dataFactoryConnectorPartialUpdateDelegate} delegate. The Update entity in data factory partially template was implemented for it, represented by dataFactoryConnectorPartialUpdateDelegate.json JSON file.

This delegate was developed to update certain parameter values in the database table.

2. Usage and configuration in a business process

Let’s look at an example of business process modelling using the partial (selective) entity parameters update delegate.

The puprose of this business process is the creation of an entity in Data Factory, search for this entity by ID, and updating of certain parameters in the entity, meaning its partial update.

2.1. Creating a pool for the business process

Найперше, необхідно змоделювати пул для бізнес-процесу. Для цього виконайте такі налаштування: To model a pool for the business process, take the following steps:

Open Camunda Modeler and create a new BPMN diagram by clicking File → New File → BPMN Diagram in the top left corner.

On the left panel, find the Create pool/Participant element and drag it to the modelling canvas.

Fill in the following fields with the corresponding values:

In the Participant Name field, enter the name of the pool — Partial data update.
In the Process id field, enter business process ID — partially-data-update.
In the Process Name field, enter business process name — Partial data update.

2.2. Modelling start form

On this stage we need to model the start form for data enteroing.

This business process is initiated by a start form that is filled with user data, not a start event.

To configure the start form, take the following steps:

On the left panel, find the CreateStartEvent element, and drag it to the modelling canvas.
On the right panel, open the General tab:
- In the Id field, enter element ID. For example, addPersonalProfile.
- In the Name field, enter start event name — Entering student data
- In the Initiator field, enter initiator.
  
  initiator — is a special variable set for the user who started the process.
Navigate to the Forms tab. In the Form Key field, enter the key for the business process form — add-person-profile.

2.3. Modelling a form for signing the data with Qualified Electronic Signature

Now we need to model a user form for the signing of entered data with an e-Signature. The data is transferred from data entering form to the signing form via the submission() function in the Form data pre-population field.

Model a User form user task for the signing of user profile data with e-Signature, and connect it with the business process form using the Form key parameter.
On the right panel, configure the following parameters:

In the Id field, enter task ID — signPersonalProfile. That’s the task definition key.
In the Name field, enter task name. For example, Signing of student data.
In the Form key field, enter business process form key — sign-person-profile.
In the Assignee field, enter the variable of the user who initiated the process instance — ${initiator}.
In the Form data pre-population field, enter the data that needs to be transferred from the start form for signing. Use the submission() function for this — ${submission('addPersonalProfile').formData}.

2.4. Modelling "Preparing data for writing (transient var)" script task

The data entered into the form, and signed with an e-Signature are transferred to a Script task, where a groovy-script is used to form a JSON-object from the data, and write it into the createPersonPayload variable.

Create a new task, define its type by clicking the key icon and selecting Script Task from the menu.

On the right panel, fill in the following fields:

In the Name field, enter task name — Preparing data for writing (transient var).
In the Script Format field, enter script format — groovy.
In the Script Type field, enter script type — Inline Script.
In the Script field, enter the groovy-script:

Example 2. The groovy-script that forms the JSON-object to be written to the database

def formData = submission('signPersonalProfile').formData

def cephData = [:]
        cephData['secondName'] = 'Tiberius'
        cephData['lastName'] = formData.prop('lastName').value()
        cephData['firstName'] = formData.prop('firstName').value()
        cephData['birthday'] = formData.prop('birthday').value()

def createPersonPayload = S(cephData, 'application/json')
execution.removeVariable('createPersonPayload')
set_transient_variable('createPersonPayload', createPersonPayload)

bp partial update 5

As a result of the task execution we get a formed JSON, stored in the createPersonPayload variable to be used in the business process later.

Example 3. A formed JSON-object, stored in the createPersonPayload variable

{
"secondName": "string",
"firstName": "string",
"lastName": "string",
"birthday": "2022-02-16T13:17:10.952Z"
}

2.5. Modelling a service task for entity creation in the database

The received data is used in a service task for user profile creation.

It is necessary to use the Create entity in data factory delegate for entity creation, using the Payload from the ${createPersonPayload} variable, and send a request to the corresponding person-profile API-endpoint.

The resource access token, e-Signature, and system signature key are sent with the data.

Model a new task.
Define its type by clicking the key icon and selecting Service Task from the menu.
Navigate to the right panel and apply the Create entity in data factory delegate by selecting the corresponding template from the Open Catalog.
Perform the following configurations:

In the Name field, enter task name. For example, Save data into the DB
In the Resource field, enter the resource (API-endpoint), where the request will be performed — person-profile.

On the API level, the endpoint looks like this: /<resource name>, where <resource name> is the name of the resource. In the Resource field, the value after slash (/) must be entered.

In the Payload field, enter request body — the JSON-object, meaning the data from ${createPersonPayload} variable, which needs to be saved in the Data Factory.

Please be advised that this JSON-object (the payload) must be created earlier within the Script Task.

In the X-Access-Token field, enter resource access token — ${completer('signPersonalProfile').accessToken}.

The access token is taken from EITHER the initiator (for example, ${initiator().accessToken}), OR task completer (for example, ${completer('taskDefinitionId').accessToken}):

If there is no other user task before the service task in the business proces, we use an initiator token.

If there is another user task before the service task, we use the completer token.

This way we create an entity in the database EITHER on behalf of the business process initiator, OR on behalf of the completer of the previous user task.

In the X-Digital-Signature-source field, enter e-Signature source — ${sign_submission('signPersonalProfile').signatureDocumentId}.

In the X-Digital-Signature-Derived-source field, enter the source of the system signature, meaning the variable where the system key is taken from — ${createPersonPayloadDerivedKey}.

In the Result variable field, enter variable name, where the API response will be written to — response.

As a result, a transaction that creates the entity with the user profile data in the database, is performed.

2.6. Modelling the entity search service task

Now we need to find the data written to the DB, using a search condition, looking for a person with the newest last name written into the DB. So we are trying to search for a user ID using lastName as the search condition. The result will be written into the response variable,

Model a new task.
Define its type by clicking the key icon and selecting Service Task from the menu.
Navigate to the right panel and apply the Create entity in data factory delegate by selecting the corresponding template from the Open Catalog.

Perform the following configurations:

In the Name field, enter task name. For example, Defining record ID
Expand the Resource section:
- In the Local Variable Assigment field, set the option for defining local variables — On.
- In the Variable Assignment Type field, select the type of variable assignment from the menu — String or Expression.
- In the Variable Assignment Value field, enter the value for the local variable — person-profile-equal-last-name. This is the name of our search condition for the resource on Data Factory level, for the corresponding view.

Expand the Search variables section:
- In the Local Variable Assigment field, set the option for defining local variables — On.
- In the Variable Assignment Type field, select the type of variable assignment from the menu — Map, which is a "key-value" pair.
- Click Add Entry (+) and add a new pair:
  - in the Key field, enter lastName, meaning the key for parameter search in the DB. This allows us to send the search parameter to the resource (API-endpoint for data search).
  - in the Value field, enter data from the user form, where the lastName was entered — ${submission('signPersonalProfile').formData.prop('lastName').value()}.

Expand the Access Token section. Enter resource access token — ${completer('signPersonalProfile').accessToken}.

The access token is taken from EITHER the initiator (for example, ${initiator().accessToken}), OR task completer (for example, ${completer('taskDefinitionId').accessToken}):

If there is no other user task before the service task in the business proces, we use an initiator token.

If there is another user task before the service task, we use the completer token.

This way we create an entity in the database EITHER on behalf of the business process initiator, OR on behalf of the completer of the previous user task.

In the Result Variable field, enter the name for the transit variable, where the request result will be written to — `response.

2.7. Modelling script task for getting student data

On this stage we need to fet element ID from the response transit variable from the previous task, using a script. This action is required to write the result to another, non-transit variable, and store the ID there. The new variable will be used further during the partial entity update.

Create a new task, define its type by clicking the key icon and selecting Script Task from the menu.

On the right panel, fill in the following fields:

In the Name field, enter task name — Getting student data.
In the Script Format field, enter script format — groovy.
In the Script Type field, enter script type — Inline Script.
In the Script field, enter the groovy-script:

response.responseBody.elements().get(0).prop('personProfileId').value()

+ TIP: The script gets the value from the first element of the response variable.

In the Result Variable, enter value for the new variable for ID rewriting.

2.8. Modelling user form for data editing

Now we need to model a form, where the user will be able to enter updated information on the student profile.

Model a User Form for the signing of user profile data with e-Signature, and connect it with the business process form using Form key parameter.

On the right panel, configure the following parameters:

In the Id field, enter task ID — editPersonalProfile. It is the task definition key.
In the Name field, enter task name. For example, Student data editing.
In the Form key field, enter business process form key — edit-person-profile.
In the Assignee field, enter the variable of the user who initiated the process instance — ${initiator}.
In the Form data pre-population field, enter the data that needs to be edited — ${submission('addPersonalProfile').formData}.

2.9. Modelling user form for the signing of updated data with e-Signature

Now we need to model the user form for the signing of updated data with e-Signature.

Model a User Form for the signing of user profile data with e-Signature, and connect it with the business process form using Form key parameter.

On the right panel, configure the following parameters:

In the Id field, enter task ID — signEditedPersonalProfile. It is the task definition key.
In the Name field, enter task name. For example, Sign updated data
In the Form key field, enter business process form key — sign-edited-person-profile.
In the Assignee field, enter the variable of the user who initiated the process instance — ${initiator}.
In the Form data pre-population field, enter the edited data that needs to be signed — ${submission('editPersonalProfile').formData}.

2.10. Modelling script task for the forming of an object with updated data

Create a new task, define its type by clicking the key icon and selecting Script Task from the menu.

On the right panel, fill in the following fields:

In the Name field, enter task name — Preparing data for writing (transient var). -* In the Script Format field, enter script format — groovy.
In the Script Type field, enter script type — Inline Script.
In the Script field, enter the groovy-script:

Example 4. The groovy-script that forms the JSON-object to be written to the database

def formData = submission('signEditedPersonalProfile').formData
def cephData = [:]

        cephData['lastName'] = formData.prop('lastName').value()
        cephData['firstName'] = formData.prop('firstName').value()
        cephData['birthday'] = formData.prop('birthday').value()

        set_transient_variable('updatePersonPayload', S(cephData, 'application/json'))

+ bp partial update 11

2.11. Modelling the service task for the partial entity update according to the changes made

Now we need to model the service task for entity update according to the changes made into the form. This is done via a dedicated delegate.

The ]Update entity in data factory partially* extension is a delegate for the partial update of an entity in Data Factory, which is configured through the developed Update entity in data factory partially template (dataFactoryConnectorPartialUpdateDelegate.json).

Before configuring the template in Camunda Modeler, ensure that the application folder resources → element-templates includes the dataFactoryConnectorPartialUpdateDelegate.json file.

Create a Service Task.

On the right panel, click Open Catalog, select the corresponding Update entity in data factory partially template from the list, and click Apply to confirm.

Configure the selected template:

In the Name field, enter task name. For example, Partial update completed.

In the Resource field, enter the resource, meaning the name of the endpoint to send the request to — person-profile.

On the API level, the endpoint looks like this: /partial/<resource-name>/<resource-id>, where <resource name> is the name of the and <resource-id> is the resource Data Factory ID. In the Resource field, the value between /partial and /<resource-id> must be entered without slash (/).

In the Resource id field, enter the ID of the resource, meaning the entity in Data Factory that needs to be updated. For example, {id}.

Resource ID is defined in UUID format. It can be transferred as a variable taken from previous tasks of the business process, or plain — f7dc68fe-98e1-4d95-b80f-df5ce42cebb9.

In the Payload field, enter request payload — a JSON-structure with the parameters that need to be updated in Data Factory. For example, ${updatePersonPayload}.

In the X-Access-Token field, enter resource access token. For example, ${completer('signEditedPersonalProfile').accessToken}.

The access token is taken from EITHER the initiator (for example, ${initiator().accessToken}), OR task completer (for example, ${completer('taskDefinitionId').accessToken}).

In the X-Digital-Signature source field, enter the source for the Ceph-document, where the user’s signature, used on the UI-form data during the entering, is stored — ${sign_submission('signEditedPersonalProfile').signatureDocumentId}.

In the X-Digital-Signature-Derived source field, enter the source for the Ceph-document, where the system signature, automatically attached to request body, is stored — ${updatePersonPayloadDerivedKey}.

In the Result variable field, enter the process variable name, where the result will be written (default — response).

2.12. Modelling process end event

End the business process by modelling an end event and entering Process end in the Name field.

bp partial update 13

3. API implementation

On the API level, Data Factory has a dedicated endpoint for processing PATCH method for partial entity update.

Example 5. Partial entity update request parameters

Method and resource:

PATCH /partial/<resource-name>/<resource-id>

Request payload:

{
	"firstName":"John",
	"lastName":"Doe",
	"birthday":"2020-01-01"
}

4. Data model level implementation

On the data structure level, add the corresponding changeSet with the <ext:partialUpdate> tag to the createSearchConditions.xml file. This way you’ll automatically create a dedicated PATCH-endpoing on the API level to support partial entity update.

The <ext:partialUpdate> tag must be added to each table.

Example 6. changeSet for the partial entity update in the DB

<changeSet author="registry owner" id="partial update person_profile">
	<ext:partialUpdate>
		<ext:table name="person_profile">
			<ext:column name="last_name"/>
			<ext:column name="first_name"/>
			<ext:column name="birthday"/>
		</ext:table>
	</ext:partialUpdate>
</changeSet>