Please describe the missing/desired functionality:

Optimize is currently somewhat inflexible in how it approaches ES configuration and could be better "optimized" to suit the different needs of our customers, allowing it handle data ingestion/read at large scale.

We should investigate ways in which we can improve performance and make use of ES capabilities in ways that suit all of our customers, rather than trying to find a one-size-fits all approach

Core solution

Please describe the missing/desired functionality:

We currently have a fixed number of shards for each process/decision instance index. This doesn't distinguish between the requirements of each though, instead being applied to all. This exposes a lot of redundancy and inefficiency on which we could improve. We should investigate how Optimize can handle index configuration in a smarter way based on the ingested data.

One idea would be to only have a single "warm" index for each process instance, where the currently written data is always routed. Simultaneously, "completed" instances could be periodically archived to a read-only index. For report evaluation, we can query the archive index and the warm index, but for data write, we would only ever need to update the running instances

What problem would the feature solve?

The environment.yaml's one-size-fits-all approach is inappropriate for real-world indices of varying size, and requires a complex manual effort that adds friction to the Optimize version upgrade process.
Elastic recommends splitting indices into shards with ~50GB each, but indices vary anywhere from a few MB (1 shard) to 3.4TB (70 shards).

Hint

• Configuration per process definition key is a partial solution, as manual evaluation is still necessary
• Elastic offers a feature for index autoscaling that we might be able to leverage. https://www.elastic.co/guide/en/elasticsearch/reference/7.10/index-lifecycle-management.html
• The archive indices could be rolled over based on size in line with ES recommendation
• The "warm" index could still be shard configurable as now, to allow for the different needs of different customers 
• We might be able to identify completed instances based on the timestamps/positions of the mediator
  • i.e. a completed instance is one in which we have an end date, and if the latest timestamp/position is more recent than the timestamp/position in which the end date was written (note that we don't know the position on the zeebe instance data)