Uploaded image for project: 'camunda BPM'
  1. camunda BPM
  2. CAM-11147

NullPointerException when two nodes tries to reconfigure history clean up

    XMLWordPrintable

    Details

    • Type: Bug Report
    • Status: Closed
    • Priority: L3 - Default
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.13.0, 7.12.1, 7.13.0-alpha1
    • Component/s: engine
    • Labels:
    • Environment:
      7.11.3

      Description

      Given:

      • A cluster exists with multiple nodes pointing to the same database
      • The cluster was already up and running so that the History Cleanup jobs are stored in the jobs table
      • History Cleanup job is configured on all nodes
      • All nodes are down
      • The previous run of History Cleanup job failed with an exception so that the job references the exception stacktrace (stored as bytearray)

      When:

      • The cluster is started so that all nodes are started at the very same time

      Then:

      • All nodes will reconfigure the History Cleanup jobs (i.e., release any locks, ...)
      • Therefore the nodes will also delete the referenced stacktrace in the bytearray table (see [1])
      • Two nodes (node1 and node2) are in progress of reconfiguring the History Cleanup jobs, so that the following happens:
        • node1 and node2 executes the HistoryCleanupCmd [2]
        • both nodes decide to reconfigure the History Cleanup jobs
        • so that both nodes fetched already the existing History Cleanup jobs
        • node1 already executed the code block in [1]
        • node2 hasn't executed the code block in [1]
        • node1 completes the execution of HistoryCleanupCmd and commits the transaction
        • as a result, the stacktrace stored in the bytearray table gets deleted
        • node2 executes the code block in [1] and fails with a NullPointerException because the respective bytearray does not exist anymore
      02-Dec-2019 16:35:29.452 SEVERE [main] org.camunda.commons.logging.BaseLogger.logError ENGINE-16004 Exception while closing command context: null
       java.lang.NullPointerException
      	at org.camunda.bpm.engine.impl.db.entitymanager.cache.DbEntityCache.getCachedEntity(DbEntityCache.java:145)
      	at org.camunda.bpm.engine.impl.db.entitymanager.cache.DbEntityCache.setDeleted(DbEntityCache.java:362)
      	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.delete(DbEntityManager.java:513)
      	at org.camunda.bpm.engine.impl.persistence.entity.EverLivingJobEntity.init(EverLivingJobEntity.java:79)
      	at org.camunda.bpm.engine.impl.persistence.entity.JobManager.reschedule(JobManager.java:98)
      	at org.camunda.bpm.engine.impl.cmd.HistoryCleanupCmd.reconfigureJobs(HistoryCleanupCmd.java:157)
      	at org.camunda.bpm.engine.impl.cmd.HistoryCleanupCmd.execute(HistoryCleanupCmd.java:82)
      	at org.camunda.bpm.engine.impl.cmd.HistoryCleanupCmd.execute(HistoryCleanupCmd.java:45)
      	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28)
      	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110)
      	at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
      	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
      	at org.camunda.bpm.engine.impl.HistoryServiceImpl.cleanUpHistoryAsync(HistoryServiceImpl.java:170)
      	at org.camunda.bpm.engine.impl.HistoryServiceImpl.cleanUpHistoryAsync(HistoryServiceImpl.java:166)
      	at org.camunda.bpm.engine.impl.BootstrapEngineCommand.createHistoryCleanupJob(BootstrapEngineCommand.java:62)
      	at org.camunda.bpm.engine.impl.BootstrapEngineCommand.execute(BootstrapEngineCommand.java:42)
      	at org.camunda.bpm.engine.impl.BootstrapEngineCommand.execute(BootstrapEngineCommand.java:32)
      	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28)
      	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110)
      	at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
      	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
      	at org.camunda.bpm.engine.impl.ProcessEngineImpl.executeSchemaOperations(ProcessEngineImpl.java:124)
      	at org.camunda.bpm.engine.impl.ProcessEngineImpl.<init>(ProcessEngineImpl.java:93)
      	at org.camunda.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl.buildProcessEngine(ProcessEngineConfigurationImpl.java:798)
      	at org.camunda.bpm.container.impl.jmx.services.JmxManagedProcessEngineController.start(JmxManagedProcessEngineController.java:38)
      	at org.camunda.bpm.container.impl.jmx.MBeanServiceContainer.startService(MBeanServiceContainer.java:77)
      	at org.camunda.bpm.container.impl.jmx.MBeanServiceContainer.startService(MBeanServiceContainer.java:63)
      	at org.camunda.bpm.container.impl.deployment.StartProcessEngineStep.performOperationStep(StartProcessEngineStep.java:116)
      	at org.camunda.bpm.container.impl.spi.DeploymentOperation.execute(DeploymentOperation.java:120)
      	at org.camunda.bpm.container.impl.jmx.MBeanServiceContainer.executeDeploymentOperation(MBeanServiceContainer.java:160)
      	at org.camunda.bpm.container.impl.spi.DeploymentOperation$DeploymentOperationBuilder.execute(DeploymentOperation.java:216)
      	at org.camunda.bpm.container.impl.tomcat.TomcatBpmPlatformBootstrap.deployBpmPlatform(TomcatBpmPlatformBootstrap.java:83)
      	at org.camunda.bpm.container.impl.tomcat.TomcatBpmPlatformBootstrap.lifecycleEvent(TomcatBpmPlatformBootstrap.java:61)
      	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
      	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
      	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
      	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:925)
      	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
      	at org.apache.catalina.startup.Catalina.start(Catalina.java:633)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
      	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:350)
      	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
      

      [1]: https://github.com/camunda/camunda-bpm-platform/blob/68dfbd4ecab899ac0061d3ece1dfc0b38e54c698/engine/src/main/java/org/camunda/bpm/engine/impl/persistence/entity/EverLivingJobEntity.java#L77-L80
      [2]: https://github.com/camunda/camunda-bpm-platform/blob/68dfbd4ecab899ac0061d3ece1dfc0b38e54c698/engine/src/main/java/org/camunda/bpm/engine/impl/cmd/HistoryCleanupCmd.java

        Attachments

          Activity

            People

            Assignee:
            nikola.koevski Nikola Koevski
            Reporter:
            garima.yadav Garima Yadav
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: