Description

Currently, the importer executes a search query by position and partitionId. With a growing data set, the search query's execution time increases up to ~900ms:

Measuring the entire roundtrip from the application perspective, the overall roundtrip takes ~3s:

To achieve a steady import time and performant search query, the search query should be done by the sequence field, which will be introduced with https://github.com/camunda/zeebe/issues/10568 in Zeebe.

More details can be found here: https://github.com/camunda/product-hub/issues/102

The field sequence encodes the partitionId and a counter in it, allowing to formulate only one RangeQuery:

BooleanQuery (+RangeQuery x < sequence <= (x+200)) (size: 200, sort by sequence asc)

Limiting the range of sequences avoids a growing execution when the importer lags.

Breakdown

• Extend the import position index schema by the field sequence of type long
• The ImportPositionEntity contains the sequence field as well
• Rewrite the search query by using a RangeQuery with sequence greater than the last imported sequence value and lower equals than the last imported sequence value + the given batch size: +RangeQuery x < sequence <= (x + batchSize)
• Backward Compatibility:
• As long as the imported events don't have a sequence field set, the search query happens by position and partitionId
• Once an event with a sequence field and value is imported, the search query happens by the sequence field.
• Migration: The tasklist-import-position is migrated to the new schema

Testing Notes:

Ideally test in SaaS and CCSM. Run Optimize with a zeebe version where records have the sequence field, eg 8.2.0-alpha3 and confirm that zeebe records are still imported correctly. Also run Optimize with a zeebe version pre 8.2.0-alpha3 and confirm records are imported correctly.

Also ensure that Optimize correctly imports data when the zeebe records are mixed (ie coming from different versions), eg by:
1) start old Zeebe version and create some data in Zeebe (eg running some instances)
2) stop old Zeebe version
3) start new Zeebe version and create some data in Zeebe (eg running some instances)
4) start Optimize and let it import Zeebe records
5) confirm data from old and new Zeebe is imported correctly into Optimize