L3 - Default Problem:
When a job is executed within a JTA transaction and an exception is thrown upon commit (i.e. invocation of JtaTransactionInterceptor#doCommit), the job's retries are decremented properly but the job's exception message (and incident message) is not populated.
Reason:
Job retries are implemented via a transaction listener. This listener is handed the exception when it is thrown. However, this is only done by the CommandContextInterceptor, meaning that only exceptions thrown from within a command or on its flush are treated that way. Exceptions thrown from a transacrtion synchronization (transaction listener) cannot be caught this way. A common example is JPA which often implements manged flush of the transaction scoped entity manager as TX-Sync on commit. Exceptions thrown by JPA at this point are not visible in the process engine database. Those kinds of exceptions are then only logged in ExecuteJobsRunnable#run.
Possible solution:
Implement job retry decrements in ExecuteJobsRunnable#run instead of the transaction listener. Then, the catched exception can not only be logged but also set on the job. However, we have to make sure that we do not change the existing behavior other than that. Especially an exception thrown within the job command should not be wrapped in any other exception by a command interceptor. Thus, there has to be some way of handing these "out", for example by setting them on the current command.