Details
-
Sub-task
-
Resolution: Unresolved
-
L3 - Default
-
None
-
None
-
1
-
Not defined
Description
After process instances have all of their information written to them, we can then archive them into another index that doesn't get written to. Maybe this can be done with a reindex operation from the warm index to the archive index.
Alrgorithm could be something like:
- find the timestamp of the furthest behind import (per data source)
- read all of the instances from just the test non-archive indices that:
a - have an end date timestamp
b - the end date timestamp is before the furthest behind timestamp from (a) - for each definition key, create an index (if it doesn't exist)
- Move the instances to the archive index
Edge case: what if we have NO incidents (so no last imported timestamp), we archive the instance, and then an incident comes in
Edge case: handle when there are none of a given entity, i.e. dont take epoch because then nothing will ever get archived
Notes:
- make sure the records are deleted from the source index
- block index in case the future rollover tries to run at the same time
ATs:
- Completed instances are archived after a reasonable amount of time (TBD - configurable?)
- Archived instances can still be read in the context of report evaluation
- Perhaps this operation could be scheduled on a cron trigger
- There should be no duplicate records resulting from updates to records that have already been archived