NOTE: This is a cloned ticket from OPT-5911 with a 10x process instance volume

We don't currently have an accurate estimate as to how Optimize performs while reading large volumes of data. We do have query performance tests but we should also have a good measure of performance when querying data of the volume specified in this KR.

There is fairly big overlap with https://jira.camunda.com/browse/OPT-5910, in that there is a requirement for large datasets in both, so it might make sense to think of the requirements for these together.

PM Notes

• If it takes more than 15 seconds to generate a report, we should review the report loading state or spinner
• If it takes more than 30 seconds to generate a report, we may want to consider data sampling (like Tableau). They offer a 2-tier data import approach. First, they provide a default sample of up to 1M records. Next, they offer the ability to "load all data".
• Errors should fail fast and provide actionable feedback

TL Notes:

• Would it be possible to share the dataset that Operate are already using?
• We should aim for a diverse dataset where are importing all different types of entities
• We can probably exclude DMN as this is not yet imported (a follow-up ticket might be appropriate though for later)
• As this ticket progresses, you may identify areas where we can improve the import pipelines. They can become subtasks or future follow-up tickets, depending on how you evaluate the potential improvement. One example is tickets such as https://jira.camunda.com/browse/OPT-4961, whereby backend pagination can mitigate poor query performance/failures
• This will require close collaboration with infra, most likely
• The work we do as part of https://jira.camunda.com/browse/OPT-5790 will also likely have a direct impact on the read (or probably write) performance of Optimize