Uploaded image for project: 'Camunda Optimize'
  1. Camunda Optimize
  2. OPT-4069

Optimize keeps open too many scrolls

    XMLWordPrintable

    Details

    • Type: Bug Report
    • Status: Done
    • Priority: L3 - Default
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0, 3.2.0-alpha2
    • Component/s: backend
    • Labels:
    • Epic Link:
    • Effort:
      Not defined

      Description

      Problem:

      • given:
        • I have a lot of data in the engine
        • Optimize is importing the data
        • I have many users (100-200) that are interacting with Optimize
      • when:
        • I open the Optimize homepage
      • then:
        • I get an error:
          19:11:13.003 [qtp1946988038-52] ERROR o.c.o.r.p.GenericExceptionMapper - Mapping generic REST error
          org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
          	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
          	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1706)
          	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1683)
          	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1446)
          	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403)
          	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373)
          	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:915)
          	at org.camunda.optimize.service.es.OptimizeElasticsearchClient.search(OptimizeElasticsearchClient.java:181)
          	at org.camunda.optimize.service.es.reader.EntitiesReader.runEntitiesSearchRequest(EntitiesReader.java:266)
          	at org.camunda.optimize.service.es.reader.EntitiesReader.getAllEntitiesForCollection(EntitiesReader.java:168)
          	at org.camunda.optimize.service.security.AuthorizedEntitiesService.getAuthorizedCollectionEntities(AuthorizedEntitiesService.java:56)
          	at org.camunda.optimize.service.collection.CollectionEntityService.getAuthorizedCollectionEntities(CollectionEntityService.java:44)
          	at org.camunda.optimize.rest.CollectionRestService.getEntities(CollectionRestService.java:279)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
          	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
          	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
          	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
          	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
          	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
          	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
          	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
          	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
          	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
          	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
          	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
          	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
          	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
          	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
          	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
          	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
          	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
          	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
          	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
          	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
          	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
          	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
          	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
          	at org.camunda.optimize.jetty.NoCachingFilter.doFilter(NoCachingFilter.java:50)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
          	at org.camunda.optimize.jetty.SingleSignOnFilter.doFilter(SingleSignOnFilter.java:72)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
          	at org.camunda.optimize.jetty.LicenseFilter.doFilter(LicenseFilter.java:85)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
          	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
          	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
          	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
          	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
          	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
          	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
          	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
          	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
          	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
          	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
          	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
          	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
          	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
          	at org.eclipse.jetty.server.Server.handle(Server.java:500)
          	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
          	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
          	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
          	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
          	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
          	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
          	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)
          	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)
          	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
          	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
          	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
          	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
          	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
          	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
          	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
          	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
          	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
          	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
          	at java.lang.Thread.run(Thread.java:748)
          	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/optimize-single-process-report,optimize-single-decision-report,optimize-combined-report,optimize-dashboard/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&scroll=60000ms&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 500 Internal Server Error]
          {"error":{"root_cause":[{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."},{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."},{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."},{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"optimize-combined-report_v3","node":"LhdNjex4QjSFXH7CUiDckA","reason":{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}},{"shard":0,"index":"optimize-dashboard_v2","node":"LhdNjex4QjSFXH7CUiDckA","reason":{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}},{"shard":0,"index":"optimize-single-decision-report_v2","node":"LhdNjex4QjSFXH7CUiDckA","reason":{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}},{"shard":0,"index":"optimize-single-process-report_v2","node":"LhdNjex4QjSFXH7CUiDckA","reason":{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}}]},"status":500}
          		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:260)
          		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:238)
          		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212)
          		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1433)
          		... 80 common frames omitted
          19:11:14.533 [ThreadPoolTaskScheduler-1] ERROR o.c.o.s.i.e.m.ProcessDefinitionXmlEngineImportMediator - Was not able to import next page, retrying after sleeping for 3375ms.
          org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
          	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
          	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1706)
          	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1683)
          	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1446)
          	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403)
          	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373)
          	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:915)
          	at org.camunda.optimize.service.es.OptimizeElasticsearchClient.search(OptimizeElasticsearchClient.java:181)
          	at org.camunda.optimize.service.importing.engine.handler.ProcessDefinitionXmlImportIndexHandler.performInitialSearchQuery(ProcessDefinitionXmlImportIndexHandler.java:115)
          	at org.camunda.optimize.service.importing.ScrollBasedImportIndexHandler.fetchNextPageOfIds(ScrollBasedImportIndexHandler.java:91)
          	at org.camunda.optimize.service.importing.ScrollBasedImportIndexHandler.getNextPage(ScrollBasedImportIndexHandler.java:56)
          	at org.camunda.optimize.service.importing.ScrollBasedImportMediator.importNextPage(ScrollBasedImportMediator.java:22)
          	at org.camunda.optimize.service.importing.BackoffImportMediator.importNextPageRetryOnError(BackoffImportMediator.java:97)
          	at org.camunda.optimize.service.importing.BackoffImportMediator.runImport(BackoffImportMediator.java:44)
          	at org.camunda.optimize.service.importing.engine.EngineImportScheduler.lambda$executeImportRound$2(EngineImportScheduler.java:118)
          	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
          	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
          	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
          	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
          	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
          	at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
          	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
          	at org.camunda.optimize.service.importing.engine.EngineImportScheduler.executeImportRound(EngineImportScheduler.java:124)
          	at org.camunda.optimize.service.importing.engine.EngineImportScheduler.runImportRound(EngineImportScheduler.java:100)
          	at org.camunda.optimize.service.importing.engine.EngineImportScheduler.runImportRound(EngineImportScheduler.java:80)
          	at org.camunda.optimize.service.importing.engine.EngineImportScheduler.run(EngineImportScheduler.java:44)
          	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
          	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          
      • expected:
        • Optimize does not throw any errors even if it is important data and has many users interacting with it

      Hints:

      • We are sometimes setting the scroll time for Elasticsearch with the method TimeValue.timeValueSeconds(configurationService.getElasticsearchScrollTimeout()). However, the scroll time in the config is given in milliseconds.
      • suggestion for a solution:
        • the configuration should be in seconds.
        • The configuration is renamed to elasticsearchScrollTimeoutInSeconds
        • Everywhere we're setting the scroll timeout, we use TimeValue.timeValueSeconds

        mgm-controller-panel

        This is the controller panel for Smart Panels app

          Attachments

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              johannes.heinemann Johannes Heinemann
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Salesforce