Details
-
Bug Report
-
Resolution: Fixed
-
L3 - Default
-
3.4.0
-
import nested document limit
-
M
Description
Problem:
Due to the nested document structure the Optimize import can get stuck on complex process instances that exceed the nested document limit of Elasticsearch in regards to the number of activity instances, variables instances or incidents.
Steps to reproduce:
1. Run a process instance with more than 10k activity instances associated with.
2. Run the Optimize import
Actual result :
3. The activity instance import will get stuck trying to write the imported data to the instance document and logs the cause
10:55:40.322 [ImportJobExecutor-pool-}}{{8}}{{] ERROR o.c.o.s.e.j.i.CompletedActivityInstanceElasticsearchImportJob - Error while executing import to elasticsearch
org.camunda.optimize.service.exceptions.OptimizeRuntimeException: There were failures while performing bulk on Completed activity instances.
If you are experiencing failures due to too many nested documents, try carefully increasing the configured nested object limit (es.settings.index.nested_documents_limit). See Optimize documentation for details. Message: failure in bulk execution:
[}}{{821}}{{]: index [optimize-process-instance--processv6], type [_doc], id [dgffdg3453tdfgdgdf435], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=The number of nested documents has exceeded the allowed limit of [}}{{10000}}{{]. This
limit can be set by changing the [index.mapping.nested_objects.limit] index level setting.]]]
Expected result:
It would be preferable to not block the entire import in such cases. Rather the error should be detected logged and the instance skipped. It also may make sense to make this configurable to make it an actual decision of the user to ignore such cases.
What is the context of your help request?
SUPPORT-11126