Uploaded image for project: 'Camunda Optimize'
  1. Camunda Optimize
  2. OPT-5303

Optimize import can miss data if there are more entities with the same timstamp than the maxPage value

      Problem:
      If the engine history contains more entities (activityInstances, variable updates etc.) with the exact same timestamp (same milliseconds) than the currently configured maxPage size for that particular entity (see *.maxPageSize properties https://docs.camunda.org/optimize/latest/technical-guide/setup/configuration/#engine-common-settings ). The Optimize import will miss some of these entities as the "same timestamp as last entity" step of the import will not catch more than the maxPageSize entities.

      Steps to reproduce:
      1. Make sure that more than the maxPageSize of a particular entity import entities exist in the cambpm history. (e.g. variable updates).
      2. run the Optimize import and let it finish
      3. compare the expected and actual counts of the particular entity

      Actual result :
      Optimize contains less entities than expected

      Expected result:
      All data is imported as expected, regardless of timestamp distribution

      Hint:
      Addressing the root cause will likely require adjustments to the Camunda BPM Optimize rest api.

        This is the controller panel for Smart Panels app

            [OPT-5303] Optimize import can miss data if there are more entities with the same timstamp than the maxPage value

            Sebastian Bathke created issue -
            Sebastian Bathke made changes -
            Status Original: Open [ 1 ] New: Scheduled [ 10912 ]
            Sebastian Bathke made changes -
            Description Original: *Problem:*
            If the engine history contains more entities (activityInstances, variable updates etc.) with the exact same timestamp (same milliseconds) than the currently configured maxPage size for that particular entity (see {{*.maxPageSize}} properties https://docs.camunda.org/optimize/latest/technical-guide/setup/configuration/#engine-common-settings ). The Optimize import will miss some of these entities as the "same timestamp as last entity" step of the import will not catch more than the maxPageSize entities.


            *Steps to reproduce:*
            1. Make sure that more than the maxPageSize of a particular entity import entities exist in the cambpm history. (e.g. variable updates).
            2. run the Optimize import and let it finish
            3. compare the expected and actual counts of the particular entity


            *Actual result :*
            Optimize contains less entities than expected

            *Expected result:*
            All data is imported as expected, regardless of timestamp distribution


            New: *Problem:*
            If the engine history contains more entities (activityInstances, variable updates etc.) with the exact same timestamp (same milliseconds) than the currently configured maxPage size for that particular entity (see {{*.maxPageSize}} properties https://docs.camunda.org/optimize/latest/technical-guide/setup/configuration/#engine-common-settings ). The Optimize import will miss some of these entities as the "same timestamp as last entity" step of the import will not catch more than the maxPageSize entities.


            *Steps to reproduce:*
            1. Make sure that more than the maxPageSize of a particular entity import entities exist in the cambpm history. (e.g. variable updates).
            2. run the Optimize import and let it finish
            3. compare the expected and actual counts of the particular entity


            *Actual result :*
            Optimize contains less entities than expected

            *Expected result:*
            All data is imported as expected, regardless of timestamp distribution


            *Hint:*
            Addressing the root cause will likely require adjustments to the Camunda BPM Optimize rest api.
            Sebastian Bathke made changes -
            Mentioned Roles
            Sebastian Bathke made changes -
            Mentioned Groups
            Sebastian Bathke made changes -
            Fix Version/s New: 3.6.0 [ 16894 ]
            Sebastian Bathke made changes -
            DRI New: Sebastian Bathke [ sebastian.bathke ]
            Sebastian Bathke made changes -
            Fix Version/s Original: 3.6.0 [ 16894 ]
            Sebastian Bathke made changes -
            Status Original: Scheduled [ 10912 ] New: Open [ 1 ]
            Eric Lundberg made changes -
            PM Priority New: 2

              Unassigned Unassigned
              sebastian.bathke Sebastian Bathke
              Andromachi Rozaki Andromachi Rozaki
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: