Optimize can write 1M Zeebe Process Instances/hr to the database

We don't currently have an accurate estimate as to how Optimize performs while importing Zeebe data for cloud deployments. For us to be able to communicate this to customers, we need to be able to measure this.

We currently run import jobs nightly for platform data, but the Zeebe import is different in how it works and theoretically more performant than this (less duplication, we make use of sharding). A good starting point for this would be to look at the existing Operate import tests to see how they set this up and the data they use.

We should strive to run a nightly import test that validates the number specified in this KR, i.e Optimize can import 5M process instances within 5 hours.

PM Notes:

• One customer deals with 600k-3M root process instance batches with 48hr SLAs. During that time they might also have a different PI with a 1hr SLA. It would be nice to have a solution that doesn't block the import of the urgent PI while there's an import backlog
• Another customer is expecting 12M process instances per day at full capacity. This may increase over time
• Anything above our defined 'peak load' can cause data to lose 'near-real-time' functionality. This is acceptable, as long as it's documented. We might expect a feature where dashboards show a warning that there's a data import backlog.

Notes:

• Would it be possible to share the dataset that Operate are already using?
• We should aim for a diverse dataset where are importing all different types of entities
• We can probably exclude DMN as this is not yet imported (a follow-up ticket might be appropriate though for later)
• We should consider different cloud configurations. Ideally, the job could be parameterized to take the number of partitions used in the Zeebe data generation so we can even compare Optimize's performance with different partition configuration
• As this ticket progresses, you may identify areas where we can improve the import pipelines. They can become subtasks or future follow-up tickets, depending on how you evaluate the potential improvement
• This will require close collaboration with infra, most likely
• https://camunda.slack.com/archives/C02GQV25LG5 - the history of this channel can also give more context into potential import issues that we may have