Engine startup can fail with foreign key constraint violation when re-configuring history cleanup

Steps to reproduce:

1. Start Camunda engine 7.12.4
2. Run history cleanup
3. Let the history cleanup fail so that the history cleanup job references a job exception byte array
4. Restart the engine
5. In parallel, the history cleanup job is currently executed or if another engine does the same in a cluster
6. The update of the history cleanup jobs "fails"

Expected behavior:

• The re-configuration failure is logged, but other than the engine startup should not be affected.

Observed behavior:

• The re-configuration fails with a foreign key constraint violation exception:

  ENGINE-16004 Exception while closing command context: ENGINE-03004 Exception while executing Database Operation 'DELETE ByteArrayEntity[11...]' with message '
  ...
  ### Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.deleteByteArray (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: ORA-02292: integrity constraint (ACT_FK_JOB_EXCEPTION) violated - child record found
  '. Flush summary: 
   [
    UPDATE EverLivingJobEntity[1...]
     ...
    DELETE ByteArrayEntity[11...]
    ...
  ]
  	at org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationException(EnginePersistenceLogger.java:130) ~[camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:354) ~[camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:321) ~[camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:293) ~[camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:207) ~[camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:136) [camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:115) [camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:45) [camunda-engine-spring-7.10.14-ee.jar:7.10.14-ee]
  	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133) [spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  	at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:43) [camunda-engine-spring-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:69) [camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32) [camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.impl.ProcessEngineImpl.executeSchemaOperations(ProcessEngineImpl.java:123) [camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.impl.ProcessEngineImpl.<init>(ProcessEngineImpl.java:92) [camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl.buildProcessEngine(ProcessEngineConfigurationImpl.java:790) [camunda-engine-7.10.14-ee.jar:7.10.14-ee]
  	at org.camunda.bpm.engine.spring.SpringTransactionsProcessEngineConfiguration.buildProcessEngine(SpringTransactionsProcessEngineConfiguration.java:62) [camunda-engine-spring-7.10.14-ee.jar:7.10.14-ee]
  

Hints:

• The update UPDATE EverLivingJobEntity[1...] "fails" in a sense that no rows are updated.
• With CAM-8866, we fixed that this optimistic locking exception is not propagated.
• With CAM-10409, we implemented that whenever an OptimisticLockingException is handled, the engine continues with flushing the remaining DB operations.
• This leads to a situation where the reference from the EverLivingJobEntity[1...] to the ByteArrayEntity[11...] has not been removed so that the deletion of the byte array fails.
• We want to treat this case as an instance of OptimisticLockingException. We already catch foreign key constraint violations on UPDATE statements on Oracle and treat them that way, but it isn't in place for DELETEs yet.