When shutting down a Spring Boot application orderly, the job executor is shut down gracefully

User Story (Required on creation):

If process execution can be interrupted at any point during shutdown, this can lead to application states that are hard to repair, especially when interacting with external systems.

Examples:

• Ongoing transactions in the Camunda or an external database will only be cleaned up after a database-side timeout
• Service tasks that have non-transactional side effects can leave the overall system state inconsistent, for example if an external system is called via HTTP and that system state should remain in sync with the process state

The goal of this feature request is to make such states less likely by employing a more ordered shutdown (such as letting executing jobs finish). However, it will not rule out the general possibility of such inconsistent states.

Functional Requirements (Required before implementation):

• Ongoing jobs can complete within a configurable time frame before the application shuts down

Technical Requirements (Required before implementation):

• I can configure the job executor to wait for jobs to complete with a timeout (see org.springframework.scheduling.concurrent.ExecutorConfigurationSupport.setAwaitTerminationSeconds(int) on ThreadPoolTaskExecutor)
• I can configure if executing threads should be interrupted or not (org.springframework.scheduling.concurrent.ExecutorConfigurationSupport.setWaitForTasksToCompleteOnShutdown(boolean) on ThreadPoolExecutor; note that this will also wait for all queued jobs to complete); note that thread interruption can let executing jobs fail
  • Ideal: Jobs that are queued at the time of shutdown are still drained from the queue, but are not executed (e.g. by checking at the beginning of job execution if the job executor or process engine is already in a shutting down phase)
• Beans that are required for job execution are only destroyed after the job executor thread pool shutdown has finished (e.g. the data source, any custom delegate beans, process engine beans, ...); needs investigation to which extent this is possible without user configuration
• It is still possible to keep the current behavior which ensures that the application stops as soon as possible
• to be decided: change the default behavior + migration instructions or keep it?
• The behavior is applied on orderly application shutdowns (e.g. on SIGTERM or System#exit)

Limitations of Scope (Optional):

• Spring Boot Starter (the thread pool integration in shared and embedded engine setups is very diverse, so extending the scope would probably increase the implementation effort significantly)
• This behavior is not applied on unorderly shutdown (e.g. on SIGKILL or Runtime#halt or the machine crashing)

Hints (Optional):

• See https://www.baeldung.com/jvm-shutdown-hooks and https://stackoverflow.com/questions/191215/how-to-stop-java-process-gracefully for differences in JVM shutdown
• The feature should integrate with Spring Boot's own shutdown nicely and as necessary. Refer to https://spring.io/blog/2020/05/15/spring-boot-2-3-0-available-now:

  Graceful shutdown

  Graceful shutdown is supported with all four embedded web servers (Jetty, Reactor Netty, Tomcat, and Undertow) and with both reactive and Servlet-based web applications. When a grace period is configured, upon shutdown, the web server will no longer permit new requests and will wait for up to the grace period for active requests to complete.