One issue related to the data generation part of these tests:
the engine seems to continue inserting data after the data generation build finished.
Locally I had a resource bottleneck issue when trying to generate 1M instances (tested with 1 CPU limit, and it constantly being ~120% used, so not really surprising).
I could see that the data after some point (around 900000 proc instances) was inserted into postgres super slowly, long after the data generation maven execute was finished.
We should assure that the data generation is completely finished before terminating. (e.g. waiting that engine/postgres stopped inserting data)
I created a new ticket for that: OPT-2358
This would explain why the process instances in the import with the SQL dump is not as expected 1_000_007, but some value below that. (similar with other data)
Another issue seems to lie in the importing process, either because of a bug in Optimize that makes it “disrupt” the import and isImporting (which the tests rely on to know if it finished) returning false for some time, even though there should be more data on fetching )
OR because of an issue on the engine/postgres side: e.g. locally I had an out of memory issue in the engine container (possibly because I used the count REST endpoints too much), which led to the history fetching taking much longer than expected.
This is probably not the issue in the CI though, since the resource limits are much higher. But, there might be something different going on in the engine?
Either way, as I see it now it seems like something we should fix in the tests:
maybe a timeout based loop that occasionally checks if the values are as expected, if yes: return successful run, if no: continue loop - until a certain set timeout, which would denote our maximum allowable import time. (and if it took longer than that, then we have a performance problem 
)
Most importantly for now:
I tested it locally with 100000 process instances and verified that everything gets imported correctly.
All the values were correct (with the exception that the engine contained many variables of type "spin://application/json", which Optimize doesn't import currently)
I tried testing it with 1000000 process instances, but I ran into performance problems locally (as described before). At the time of writing this it is still inserting data, but the imports look good so far..
Task
L3 - Default
One issue related to the data generation part of these tests:
the engine seems to continue inserting data after the data generation build finished.
Locally I had a resource bottleneck issue when trying to generate 1M instances (tested with 1 CPU limit, and it constantly being ~120% used, so not really surprising).
I could see that the data after some point (around 900000 proc instances) was inserted into postgres super slowly, long after the data generation maven execute was finished.
We should assure that the data generation is completely finished before terminating. (e.g. waiting that engine/postgres stopped inserting data)
I created a new ticket for that:
OPT-2358This would explain why the process instances in the import with the SQL dump is not as expected 1_000_007, but some value below that. (similar with other data)
Another issue seems to lie in the importing process, either because of a bug in Optimize that makes it “disrupt” the import and isImporting (which the tests rely on to know if it finished) returning false for some time, even though there should be more data on fetching )
)
OR because of an issue on the engine/postgres side: e.g. locally I had an out of memory issue in the engine container (possibly because I used the count REST endpoints too much), which led to the history fetching taking much longer than expected.
This is probably not the issue in the CI though, since the resource limits are much higher. But, there might be something different going on in the engine?
Either way, as I see it now it seems like something we should fix in the tests:
maybe a timeout based loop that occasionally checks if the values are as expected, if yes: return successful run, if no: continue loop - until a certain set timeout, which would denote our maximum allowable import time. (and if it took longer than that, then we have a performance problem
Most importantly for now:
I tested it locally with 100000 process instances and verified that everything gets imported correctly.
All the values were correct (with the exception that the engine contained many variables of type "spin://application/json", which Optimize doesn't import currently)
I tried testing it with 1000000 process instances, but I ran into performance problems locally (as described before). At the time of writing this it is still inserting data, but the imports look good so far..