Uploaded image for project: 'camunda BPM'
  1. camunda BPM
  2. CAM-13604

Listener for History Cleanup Removal Time

      User Story (Required on creation):

      I can configure a listener which is notified when the removal time of one of the following entities is due:

      • Historic Process Instance
      • Historic Decision Instance
      • Historic Case Instance
      • Historic Batch

      Functional Requirements (Required before implementation):

      • In the BPMN/DMN/CMMN Model, I can define a Java class via the property camunda:historyRemovalTimeListener; for batches, I can register the listener in the process engine configuration
      • The user-provided Java class must implement the interface HistoryRemovalTimeListener
      • When the removal time is due, HistoryRemovalTimeListener#notify is called and the to be removed entity type (process, decision, case, batch) and id is passed
      • Changing the removal time via batch operation is supported

      Technical Requirements (Required before implementation):

      • When the removal time is set (instance start or end time), a job is created with removal time as the due date
        • This is to avoid adding more logic to the cleanup job execution which could potentially degrade the performance
          • Performance is an important non-functional requirement of the removal time based history cleanup strategy
      • All data that should be passed into HistoryRemovalTimeListener#notify is stored in the job since the listener could be called after the cleanup job run and the related historical data has already been cleaned up (historic entity cannot be queried anymore)

      Limitations of Scope (Optional):

      Start small by implementing this feature only for processes.

      Hints (Optional):

      Original description

      After reaching the process' Time To Live, the process is cleared from Camunda's database. But the business data that I needed to get out (and which also doesn't interest me anymore), I'd like to delete it along with TTL. So I thought a good way to do this would be to create the possibility to configure Listeners tied to TTL.

        This is the controller panel for Smart Panels app

            [CAM-13604] Listener for History Cleanup Removal Time

            Hi rodrigocarlstrom,

            Thank you for your feature request.

            Could you please reformat your ticket description that it follows our feature request template?

            h4. User Story (Required on creation): 
            h4. Functional Requirements (Required before implementation): 
            h4. Technical Requirements (Required before implementation): 
            h4. Limitations of Scope (Optional): 
            h4. Hints (Optional):
            

            Best,
            Tassilo

            Tassilo Weidner added a comment - Hi rodrigocarlstrom , Thank you for your feature request. Could you please reformat your ticket description that it follows our feature request template? h4. User Story (Required on creation): h4. Functional Requirements (Required before implementation): h4. Technical Requirements (Required before implementation): h4. Limitations of Scope (Optional): h4. Hints (Optional): Best, Tassilo

            Hi rodrigocarlstrom,

            While I see this feature generally in the scope of the product and understand the use case for it, I doubt if this can be realized concerning our implementation.

            The removal time based history cleanup strategy [1] is designed in a way to be very efficient with regards to the actual deletion of data from the relational database:

            • The removal time is calculated based on the start or end time of the process + TTL and set when a historical event is produced (start time) or when the root process instance is completed (end time)
            • When the history cleanup is running, the delete queries remove a batch of a maximum of 500 rows WHERE REMOVAL_TIME <= $NOW
            • When the removal time of rows is due, we don't know at which point in time the respective historic process instance has been or will be removed

            What do you think about the following alternative solution:
            Create a non-interrupting event subprocess with a timer start event set to the removal time for the process instance and call your business logic to clean up your business data.

            For a product feature, we could create a job that is due on the removal time. When changing the removal time of the process, we would need to maintain the due date of the job as well. Also, when the job is due, it might be executed and trigger the listener, although the actual data hasn't been cleaned up since the overall amount of data to be cleaned up is high, and it takes a while to remove everything.

            Best,
            Tassilo

            [1] https://docs.camunda.org/manual/7.15/user-guide/process-engine/history/#removal-time-based-strategy

            Tassilo Weidner added a comment - Hi rodrigocarlstrom , While I see this feature generally in the scope of the product and understand the use case for it, I doubt if this can be realized concerning our implementation. The removal time based history cleanup strategy [1] is designed in a way to be very efficient with regards to the actual deletion of data from the relational database: The removal time is calculated based on the start or end time of the process + TTL and set when a historical event is produced (start time) or when the root process instance is completed (end time) When the history cleanup is running, the delete queries remove a batch of a maximum of 500 rows WHERE REMOVAL_TIME <= $NOW When the removal time of rows is due, we don't know at which point in time the respective historic process instance has been or will be removed What do you think about the following alternative solution: Create a non-interrupting event subprocess with a timer start event set to the removal time for the process instance and call your business logic to clean up your business data. For a product feature, we could create a job that is due on the removal time. When changing the removal time of the process, we would need to maintain the due date of the job as well. Also, when the job is due, it might be executed and trigger the listener, although the actual data hasn't been cleaned up since the overall amount of data to be cleaned up is high, and it takes a while to remove everything. Best, Tassilo [1] https://docs.camunda.org/manual/7.15/user-guide/process-engine/history/#removal-time-based-strategy

            Hi rodrigocarlstrom,

            I have qualified your feature request and will now forward it to our product management for decision-making.

            Usually, our product engineering team is very busy with other topics. However, we would appreciate it if you contribute this feature to our codebase.

            Best,
            Tassilo

            Tassilo Weidner added a comment - Hi rodrigocarlstrom , I have qualified your feature request and will now forward it to our product management for decision-making. Usually, our product engineering team is very busy with other topics. However, we would appreciate it if you contribute this feature to our codebase. Best, Tassilo

            Hi rodrigocarlstrom,

            thanks for bringing this up. I agree with Tassilo here that this generally makes sense as a feature in the scope of our product.
            However, we will not be able to work on this in the near future.

            Thanks for understanding.

            Best,
            Tobias

            Tobias Metzke-Bernstein added a comment - Hi rodrigocarlstrom , thanks for bringing this up. I agree with Tassilo here that this generally makes sense as a feature in the scope of our product. However, we will not be able to work on this in the near future. Thanks for understanding. Best, Tobias

            We are closing this ticket as part of our backlog grooming. Reasons:

            • It is very unlikely that we will implement this
            • We did not receive sufficient evidence that this ticket is important

            Thorben Lindhauer added a comment - We are closing this ticket as part of our backlog grooming. Reasons: It is very unlikely that we will implement this We did not receive sufficient evidence that this ticket is important

              Unassigned Unassigned
              rodrigocarlstrom Rodrigo Carlstrom
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: