ENGINE-14006 Exception while executing job b1ae145b-f39e-11eb-bd6d-56ad6cfb5f7c: org.camunda.bpm.engine.exception.NullValueException: Invalid process definition id: processDefinitionId is null

Environment (Required on creation): PRODUCTION

Description (Required on creation; please attach any relevant screenshots, stacktraces, log files, etc. to the ticket):

We use camunda in embedded mode as a Spring boot application. We have one spring boot application per workflow and have deployed 5 workflows/5 spring boot applications. The issue happens in all the workflows and all spring boot applications. The issue is not unfortunately directly reproducible and happens only occasionally. What we observe is a process instance "stuck/frozen" in a task (implemented as JavaDelegate) where we throw an exception inside the task. We would at that point expect an Incident to be created in Camunda for the process instance but instead of Incident, we see the following in the logs:

ENGINE-14006 Exception while executing job b1ae145b-f39e-11eb-bd6d-56ad6cfb5f7c: org.camunda.bpm.engine.exception.NullValueException: Invalid process definition id: processDefinitionId is nullENGINE-14006 Exception while executing job b1ae145b-f39e-11eb-bd6d-56ad6cfb5f7c: org.camunda.bpm.engine.exception.NullValueException: Invalid process definition id: processDefinitionId is null at jdk.internal.reflect.GeneratedConstructorAccessor223.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) at org.camunda.bpm.engine.impl.util.EnsureUtil.generateException(EnsureUtil.java:394) at org.camunda.bpm.engine.impl.util.EnsureUtil.ensureNotNull(EnsureUtil.java:55) at org.camunda.bpm.engine.impl.util.EnsureUtil.ensureNotNull(EnsureUtil.java:50) at org.camunda.bpm.engine.impl.persistence.deploy.cache.ProcessDefinitionCache.checkInvalidDefinitionId(ProcessDefinitionCache.java:42) at org.camunda.bpm.engine.impl.persistence.deploy.cache.ResourceDefinitionCache.findDeployedDefinitionById(ResourceDefinitionCache.java:48) at org.camunda.bpm.engine.impl.persistence.deploy.cache.DeploymentCache.findDeployedProcessDefinitionById(DeploymentCache.java:77) at org.camunda.bpm.engine.impl.cmd.DefaultJobRetryCmd.getCurrentActivity(DefaultJobRetryCmd.java:134) at org.camunda.bpm.engine.impl.cmd.DefaultJobRetryCmd.execute(DefaultJobRetryCmd.java:63) at org.camunda.bpm.engine.impl.jobexecutor.FailedJobListener$FailedJobListenerCmd.execute(FailedJobListener.java:111) at org.camunda.bpm.engine.impl.jobexecutor.FailedJobListener$FailedJobListenerCmd.execute(FailedJobListener.java:92) at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28) at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110) at org.camunda.bpm.engine.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:46) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:44) at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70) at org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:35) at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33) at org.camunda.bpm.engine.impl.jobexecutor.FailedJobListener.execute(FailedJobListener.java:54) at org.camunda.bpm.engine.impl.jobexecutor.FailedJobListener.execute(FailedJobListener.java:32) at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28) at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110) at org.camunda.bpm.engine.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:46) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:44) at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70) at org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:35) at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33) at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.callFailedJobListenerWithRetries(ExecuteJobHelper.java:113) at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.invokeJobListener(ExecuteJobHelper.java:94) at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:75) at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:110) at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:71) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)

As an example (this happens in all our workflows), I attached Fulfillment Management workflow where we observed this issue in "Enrich Statement Information" task (it's set as Async-Before).

After executing query in Camunda database:

select * from ACT_RU_JOB t where ID_ = 'b1ae145b-f39e-11eb-bd6d-56ad6cfb5f7c';

I can see that indeed PROCESS_DEF_ID column is NULL (together with other columns).

Steps to reproduce (Required on creation): 

This can't be directly reproduced in a deterministic way.

Observed Behavior (Required on creation):

Process instances get stuck in a task after exception is thrown inside JavaDelegate and job executor keeps failing with the exception above.

Expected behavior (Required on creation):

Incident to be created in Camunda.

Root Cause (Required on prioritization):

Solution Ideas (Optional):

We found a workaround using Camunda's "Modify" functionality. If we modify the process instance to

1. Cancel the current task
2. Start before the current task

Then the "broken" job gets deleted in the database a new one is created which works fine and Incident gets created in Camunda.

Hints (Optional):

Since this happens only occasionally (i.e. in 99% cases when we throw an exception, an incident will be created in Camunda and in 1% the process instance gets stucked with the exception above), I have a suspicion this could be some kind of concurrency issue in Camunda. Also the affected field is completely out of control in our application so I expect it is an issue in Camunda code rather than our application code.