Uploaded image for project: 'Camunda Optimize'
  1. Camunda Optimize
  2. OPT-3565

Resolve query performance test instability with ES 7.0.0-7.2.x

    XMLWordPrintable

    Details

      Description

      Context:
      During the work on OPT-3524 we encountered a weird exception on the elastic side related to the the startup of Optimize and the initial load of alarms:

      05:12:17  	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/optimize-process-instance/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 429 Too Many Requests]
      05:12:17  {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [8094251846/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8094247352/7.5gb], new bytes reserved: [4494/4.3kb]","bytes_wanted":8094251846,"bytes_limit":8094194073,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [8094251846/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8094247352/7.5gb], new bytes reserved: [4494/4.3kb]","bytes_wanted":8094251846,"bytes_limit":8094194073,"durability":"PERMANENT"},"status":429}
      05:12:17  		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:260)
      05:12:17  		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:238)
      05:12:17  		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212)
      05:12:17  		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1433)
      05:12:17  		... 79 common frames omitted
      

      no matter how big the heap was sized it always hit the limit and given there are even no alarms stored this appeared to be a bug of the memory circuit breaker.

      Testing revealed this affects Elasticsearch 7.0.0-7.2.1, but from 7.3.0 it never occured.

      AT:

      • track down cause of the issue
      • reset default es version used in the query performance test

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              sebastian.bathke Sebastian Bathke
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: