Details
Description
Context:
On stage we observed the situation that Optimize became unusable, checking the logs it was spammed with elasticsearch client errors like these:
10:56:33.006 [ImportJobExecutor-pool-1] ERROR o.c.o.s.e.j.i.StoreIndexesElasticsearchImportJob - Error while executing import to elasticsearch java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:873) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:275) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:262) at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1628) at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1568) at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:533) at org.camunda.optimize.service.es.OptimizeElasticsearchClient.bulk(OptimizeElasticsearchClient.java:117) at org.camunda.optimize.service.es.writer.ElasticsearchWriterUtil.doBulkRequest(ElasticsearchWriterUtil.java:290) at org.camunda.optimize.service.es.writer.ElasticsearchWriterUtil.doBulkRequestWithList(ElasticsearchWriterUtil.java:179) at org.camunda.optimize.service.es.writer.ImportIndexWriter.importIndexes(ImportIndexWriter.java:46) at org.camunda.optimize.service.es.job.importing.StoreIndexesElasticsearchImportJob.persistEntities(StoreIndexesElasticsearchImportJob.java:26) at org.camunda.optimize.service.es.job.ElasticsearchImportJob.executeImport(ElasticsearchImportJob.java:56) at org.camunda.optimize.service.es.job.ElasticsearchImportJob.run(ElasticsearchImportJob.java:37) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED at org.apache.http.util.Asserts.check(Asserts.java:46) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90) at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:271)
The only way to recover in this situation is to restart Optimize.
AT:
- investigate on the cause
- seems related to https://github.com/elastic/elasticsearch/issues/49124
- find a permantn fix or workaround to recover in such a situation without an application restart