Error while executing import to elastic search

Environment (Required on creation): Perf. But we can not push to prod till we do not resolve the current issue. This is a priority since we are trying to push code to Prod for our new release but this is a blocker. 

Description (Required on creation; please attach any relevant screenshots, stacktraces, log files, etc. to the ticket):

Optimize is importing data from our workflow. Optimize uses nested documents to store list information such as activities or variables belonging to a process instance that time this nested document size is going beyond the acceptable size configured in config for some of the process instances that is why this exception is thrown.

Issue is happening in optimize while importing data from workflow to optimize elasticsearch,

Since optimize code is not managed by us so can't tell the exact class or method or flow.Because of import exception we might not see some data in few reports. I have attached optimize server logs(camundaSince17Feb), splunk logs(ElasticSearch_problem) and our environment config file

We used the suggested solution for this exception by increasing the acceptable document size limit(es.settings.index.nested_documents_limit = 20000) and we are still seeing this exception 

Steps to reproduce (Required on creation): Unable to recreate

Observed Behavior (Required on creation): We are not able to import data correctly. We are missing some data in our reports

 

Additional info:{}

• What version of Camunda Platform you are using?

v7.14.16-ee.

• What version of Camunda Optimize you are using?

Version 3.4.0.

• What kind of set up are you using:

• Shared engine, Embedded engine, Spring boot based, Docker?

Spring boot based.

• • Which container / application server are you using?

Spring boot embedded tomcat.

• Do you have cluster set up (and if 'yes', is it homogenous or heterogenous)?

 No.

• Please send your process engine configuration, e.g.: application.yaml, processes.xml, bpm-platform.xml, configuration spring bean, etc.

Attached to this Support Request.

Expected behavior (Required on creation): We should not see any imperfect data in our reports

Root Cause (Required on prioritization): 

es.settings.index.nested_documents_limit = 10000

Solution Ideas (Optional):

Hints (optional):