Description
Context:
Currently the `TimestampBasedImportMediator` will reimport the entitiesLastTimestamp batch over and over again if no new data is available. This causes e.g. the same activity & variable update events to be reimported continously. While this is no issue to the process_instance index as they just get merged with the existing data they get appended to the log-like indexes `optimize-variable-update-instance-` & `optimize-variable-update-instance` though. We initially accepted that as in real world scenarios new data should arrive eventually and the import pointer continue, however it makes testing harder and would continuously bloat the index if there there is no new data for a longer period, while there is an easy option to skip if the number of entities for the last timestamp is not bigger than in the previous round.
AT:
- the `TimestampBasedImportMediator` skips the entities of `entitiesLastTimestamp` if their count equals the number of entities for that timestamp of the previous execution
mgm-controller-panel
This is the controller panel for Smart Panels app
Attachments
Issue Links
- is related to
-
OPT-3459 Further improve import efficiency in terms of same timestamp entities
-
- Done
-