Optimize keeps open too many scrolls

Problem:

• given:
  • I have a lot of data in the engine
  • Optimize is importing the data
  • I have many users (100-200) that are interacting with Optimize
• when:
  • I open the Optimize homepage
• then:
  • I get an error:

    19:11:13.003 [qtp1946988038-52] ERROR o.c.o.r.p.GenericExceptionMapper - Mapping generic REST error
    org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
    	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
    	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1706)
    	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1683)
    	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1446)
    	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403)
    	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373)
    	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:915)
    	at org.camunda.optimize.service.es.OptimizeElasticsearchClient.search(OptimizeElasticsearchClient.java:181)
    	at org.camunda.optimize.service.es.reader.EntitiesReader.runEntitiesSearchRequest(EntitiesReader.java:266)
    	at org.camunda.optimize.service.es.reader.EntitiesReader.getAllEntitiesForCollection(EntitiesReader.java:168)
    	at org.camunda.optimize.service.security.AuthorizedEntitiesService.getAuthorizedCollectionEntities(AuthorizedEntitiesService.java:56)
    	at org.camunda.optimize.service.collection.CollectionEntityService.getAuthorizedCollectionEntities(CollectionEntityService.java:44)
    	at org.camunda.optimize.rest.CollectionRestService.getEntities(CollectionRestService.java:279)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
    	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
    	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
    	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
    	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
    	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
    	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
    	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
    	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
    	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
    	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
    	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
    	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
    	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
    	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
    	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
    	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
    	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    	at org.camunda.optimize.jetty.NoCachingFilter.doFilter(NoCachingFilter.java:50)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    	at org.camunda.optimize.jetty.SingleSignOnFilter.doFilter(SingleSignOnFilter.java:72)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    	at org.camunda.optimize.jetty.LicenseFilter.doFilter(LicenseFilter.java:85)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
    	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.Server.handle(Server.java:500)
    	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
    	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
    	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
    	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
    	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)
    	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)
    	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
    	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
    	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
    	at java.lang.Thread.run(Thread.java:748)
    	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/optimize-single-process-report,optimize-single-decision-report,optimize-combined-report,optimize-dashboard/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&scroll=60000ms&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 500 Internal Server Error]
    {"error":{"root_cause":[{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."},{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."},{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."},{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"optimize-combined-report_v3","node":"LhdNjex4QjSFXH7CUiDckA","reason":{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}},{"shard":0,"index":"optimize-dashboard_v2","node":"LhdNjex4QjSFXH7CUiDckA","reason":{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}},{"shard":0,"index":"optimize-single-decision-report_v2","node":"LhdNjex4QjSFXH7CUiDckA","reason":{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}},{"shard":0,"index":"optimize-single-process-report_v2","node":"LhdNjex4QjSFXH7CUiDckA","reason":{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}}]},"status":500}
    		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:260)
    		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:238)
    		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212)
    		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1433)
    		... 80 common frames omitted
    19:11:14.533 [ThreadPoolTaskScheduler-1] ERROR o.c.o.s.i.e.m.ProcessDefinitionXmlEngineImportMediator - Was not able to import next page, retrying after sleeping for 3375ms.
    org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
    	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
    	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1706)
    	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1683)
    	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1446)
    	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403)
    	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373)
    	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:915)
    	at org.camunda.optimize.service.es.OptimizeElasticsearchClient.search(OptimizeElasticsearchClient.java:181)
    	at org.camunda.optimize.service.importing.engine.handler.ProcessDefinitionXmlImportIndexHandler.performInitialSearchQuery(ProcessDefinitionXmlImportIndexHandler.java:115)
    	at org.camunda.optimize.service.importing.ScrollBasedImportIndexHandler.fetchNextPageOfIds(ScrollBasedImportIndexHandler.java:91)
    	at org.camunda.optimize.service.importing.ScrollBasedImportIndexHandler.getNextPage(ScrollBasedImportIndexHandler.java:56)
    	at org.camunda.optimize.service.importing.ScrollBasedImportMediator.importNextPage(ScrollBasedImportMediator.java:22)
    	at org.camunda.optimize.service.importing.BackoffImportMediator.importNextPageRetryOnError(BackoffImportMediator.java:97)
    	at org.camunda.optimize.service.importing.BackoffImportMediator.runImport(BackoffImportMediator.java:44)
    	at org.camunda.optimize.service.importing.engine.EngineImportScheduler.lambda$executeImportRound$2(EngineImportScheduler.java:118)
    	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
    	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
    	at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
    	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
    	at org.camunda.optimize.service.importing.engine.EngineImportScheduler.executeImportRound(EngineImportScheduler.java:124)
    	at org.camunda.optimize.service.importing.engine.EngineImportScheduler.runImportRound(EngineImportScheduler.java:100)
    	at org.camunda.optimize.service.importing.engine.EngineImportScheduler.runImportRound(EngineImportScheduler.java:80)
    	at org.camunda.optimize.service.importing.engine.EngineImportScheduler.run(EngineImportScheduler.java:44)
    	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
    	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    	at java.lang.Thread.run(Thread.java:748)
    

• expected:
  • Optimize does not throw any errors even if it is important data and has many users interacting with it

Hints:

• We are sometimes setting the scroll time for Elasticsearch with the method TimeValue.timeValueSeconds(configurationService.getElasticsearchScrollTimeout()). However, the scroll time in the config is given in milliseconds.
• suggestion for a solution:
  • the configuration should be in seconds.
  • The configuration is renamed to elasticsearchScrollTimeoutInSeconds
  • Everywhere we're setting the scroll timeout, we use TimeValue.timeValueSeconds