Uploaded image for project: 'Camunda Optimize'
  1. Camunda Optimize
  2. OPT-3565

Resolve query performance test instability with ES 7.0.0-7.2.x

XMLWordPrintable

      Context:
      During the work on OPT-3524 we encountered a weird exception on the elastic side related to the the startup of Optimize and the initial load of alarms:

      05:12:17  	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/optimize-process-instance/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 429 Too Many Requests]
      05:12:17  {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [8094251846/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8094247352/7.5gb], new bytes reserved: [4494/4.3kb]","bytes_wanted":8094251846,"bytes_limit":8094194073,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [8094251846/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8094247352/7.5gb], new bytes reserved: [4494/4.3kb]","bytes_wanted":8094251846,"bytes_limit":8094194073,"durability":"PERMANENT"},"status":429}
      05:12:17  		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:260)
      05:12:17  		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:238)
      05:12:17  		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212)
      05:12:17  		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1433)
      05:12:17  		... 79 common frames omitted
      

      no matter how big the heap was sized it always hit the limit and given there are even no alarms stored this appeared to be a bug of the memory circuit breaker.

      Testing revealed this affects Elasticsearch 7.0.0-7.2.1, but from 7.3.0 it never occured.

      AT:

      • track down cause of the issue
      • reset default es version used in the query performance test

        This is the controller panel for Smart Panels app

          1. 7.6.0.png
            67 kB
            Sebastian Bathke

              Unassigned Unassigned
              sebastian.bathke Sebastian Bathke
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Created:
                Updated:
                Resolved: