We don't currently have an accurate estimate as to how Optimize performs while reading large volumes of data. We do have query performance tests but we should also have a good measure of performance when querying data of the volume specified in this KR.

There is fairly big overlap with https://jira.camunda.com/browse/OPT-5910, in that there is a requirement for large datasets in both, so it might make sense to think of the requirements for these together.

PM Notes

• If it takes more than 15 seconds to generate a report, we should review the report loading state or spinner
• If it takes more than 30 seconds to generate a report, we may want to consider data sampling (like Tableau). They offer a 2-tier data import approach. First, they provide a default sample of up to 1M records. Next, they offer the ability to "load all data".
• Errors should fail fast and provide actionable feedback

TL Notes:

• Would it be possible to share the dataset that Operate are already using?
• We should aim for a diverse dataset where are importing all different types of entities
• We can probably exclude DMN as this is not yet imported (a follow-up ticket might be appropriate though for later)
• As this ticket progresses, you may identify areas where we can improve the import pipelines. They can become subtasks or future follow-up tickets, depending on how you evaluate the potential improvement. One example is tickets such as https://jira.camunda.com/browse/OPT-4961, whereby backend pagination can mitigate poor query performance/failures
• This will require close collaboration with infra, most likely
• The work we do as part of https://jira.camunda.com/browse/OPT-5790 will also likely have a direct impact on the read (or probably write) performance of Optimize