Asynchronous messaging subsystem

🌐 This document is available in both English and Ukrainian. Use the language toggle in the top right corner to switch between versions.

1. Overview

The Asynchronous messaging subsystem ensures asynchronous interaction between the registry components with guaranteed message storage, processing, and delivery.

2. Subsystem functions

Asynchronous message exchange.
Ensuring message delivery guarantees.
Long-term message storage.

The Asynchronous messaging subsystem is built on Apache Kafka, a stream processing and distributed message storage platform. Kafka employs a publish/subscribe model, also known as pub/sub, where a publisher sends messages to topics, and users subscribe to topics to receive those messages.

Apache Kafka clusters are managed and deployed on the Platform using Strimzi Cluster Operator. This tool provides an automated way to configure, scale, and manage Kafka clusters in an OpenShift environment.

4. Subsystem components

4.1. Registry

Component name Environment representation Source Function

Component name	Environment representation	Source	Function
Kafka broker	`kafka-cluster-kafka`	3rd-party	The main component of Apache Kafka that acts as an intermediary for message exchange between publishers and subscribers. It is the central part of the Kafka architecture responsible for storing and processing data streams.
ZooKeeper	`kafka-cluster-zookeeper`	3rd-party	A centralized service that provides coordination and synchronization for distributed systems. Kafka brokers use ZooKeeper to coordinate and manage their clusters.
Entity Operator	`kafka-cluster-entity-operator`	3rd-party	This component consists of the Topic Operator and User Operator. The Topic Operator manages topics in a Kafka cluster using Kubernetes resources. The User Operator manages Kafka users using Kubernetes resources.
Kafka Exporter	`kafka-cluster-kafka-exporter`	3rd-party	A tool that publishes Kafka cluster metrics for the Event monitoring and notification subsystem.

Kafka broker

kafka-cluster-kafka

3rd-party

The main component of Apache Kafka that acts as an intermediary for message exchange between publishers and subscribers. It is the central part of the Kafka architecture responsible for storing and processing data streams.

ZooKeeper

kafka-cluster-zookeeper

3rd-party

A centralized service that provides coordination and synchronization for distributed systems. Kafka brokers use ZooKeeper to coordinate and manage their clusters.

Entity Operator

kafka-cluster-entity-operator

3rd-party

This component consists of the Topic Operator and User Operator. The Topic Operator manages topics in a Kafka cluster using Kubernetes resources. The User Operator manages Kafka users using Kubernetes resources.

Kafka Exporter

kafka-cluster-kafka-exporter

3rd-party

A tool that publishes Kafka cluster metrics for the Event monitoring and notification subsystem.

4.2. Platform

Component name Namespace Environment representation Source Repository Function

Component name	Namespace	Environment representation	Source	Repository	Function
Strimzi Cluster Operator	kafka-operator	`strimzi-cluster-operator`	3rd-party	github:/epam/edp-ddm-strimzi-kafka-operator	This tool provides an automated way to configure, scale, and manage Kafka clusters in an OpenShift environment.
Strimzi Drain Cleaner	kafka-operator	`strimzi-drain-cleaner`	3rd-party	This tool takes over the staged upgrade management during the migration process of Kafka cluster pods between OpenShift cluster nodes, ensuring that the respective Kafka pods are migrated one at a time while maintaining the desired level of replication and availability.

Strimzi Cluster Operator

kafka-operator

strimzi-cluster-operator

3rd-party

github:/epam/edp-ddm-strimzi-kafka-operator

This tool provides an automated way to configure, scale, and manage Kafka clusters in an OpenShift environment.

Strimzi Drain Cleaner

kafka-operator

strimzi-drain-cleaner

3rd-party

This tool takes over the staged upgrade management during the migration process of Kafka cluster pods between OpenShift cluster nodes, ensuring that the respective Kafka pods are migrated one at a time while maintaining the desired level of replication and availability.

5. Technological stack

The following technologies were used when designing and developing the subsystem:

6. Subsystem quality attributes

6.1. Scalability

The Asynchronous messaging subsystem scales horizontally, enabling the distributed message storage and processing of large volumes of data. It can operate in a cluster with multiple brokers, enabling load distribution and ensuring high availability.

6.2. Performance

Kafka supports real-time message processing and streaming data analysis. It enables writing, reading, and processing large amounts of data in real time.

6.3. Security

Communication between the following components is always encrypted using TLS:

Kafka brokers
ZooKeeper nodes
Kafka brokers and operators
ZooKeeper operators and nodes
Kafka Exporter

Additionally, TLS encryption can be configured between Kafka brokers and clients.

Kafka uses authentication to ensure a secure client connection to the Kafka cluster. Kafka clusters also use authorization to control which operations are allowed on Kafka brokers by individual clients or users.

6.4. Observability

The Asynchronous messaging subsystem logs incoming requests and collects performance metrics for analysis through the web interfaces of respective Platform subsystems.

For details on the subsystem design, see:

6.5. Reliability

Kafka ensures reliable message delivery by saving messages on disk and replicating them across multiple brokers. This prevents data loss even when individual components of the system fail.

The Backup and restore subsystem handles backing up Kafka broker file systems.