Scalable Pool of Data Importers

Problem Definition

User Story

As an operations professional
I want to manage a single pool of data importers
So that I don't over-spend on infrastructure

Use Cases

• Importing from 10 connected engines using 3 identical data importers
• Maintaining a single configuration for a data importer
• Scaling up the data importer pool based on resource usage
• Supporting higher process throughput use cases (e.g. 12M/day)

PM Notes (Customer requests, context, assumptions)

From the FR (SUPPORT-11641)

"At the moment Camunda-Optimize - supports multinode/multi-instance deployments. However, in that setup, we have to commit a specific Optimize node to multiple BPM instances - "It is important to configure the cluster in the sense that only one instance at a time is actively importing from a particular Camunda Platform engine." . In such deployment, we might have to tune up how BPM instance single node should handle, this can change over time and as it does we need to reconfigure nodes.

• Currently, each engine has its own import queue.
• Because only 1 importer should pull data from an engine. The customer uses a 1:1 Engine-Importer ratio, which costs $$
• The customer wants to remove this restriction to have a single shared import queue where a few importers can share the load evenly across many engines

  Due to PaaS high tier requirements, all instances of service have to be deployed in multinode setup across availability zones otherwise we can't route user traffic to them. Each node starts with the same configuration and doesn't know about the scale of the deployment so it can't just assign a subset of BPM instances at the start. As a workaround, we had to put a data importer node in a different group that doesn't serve user traffic and can have a single instance. We will observe its resource usage and might need to create more groups with a subset of the BPM instances. All groups of instances of the service use the same image and they modify config on launch based on the group name (WebServer, dataImporter), so if we need to have more groups the more complex the launch script will get.

• A complex launch script requires additional ops time

Solution Definition

Solution Ideas

• Remove this requirement - "It is important to configure the cluster in the sense that only one instance at a time is actively importing from a particular Camunda Platform engine."
• Prevent multiple instances from pulling data from a single engine
• Prevent multiple instances from pulling data from a single import job type (e.g. variables)
• Prevent multiple instances from pulling data from a single unique record (e.g. processInstanceID 3f84...104fba)

Design Notes (Videos, mockups, guidance)

User Scenario

GIVEN
(MH/SH/NTH) WHEN
(MH/SH/NTH) THEN

Metrics (if applicable)