Uploaded image for project: 'camunda BPM'
  1. camunda BPM
  2. CAM-9043

Classloading issues when using Spring Boot Starter and Spring Dev Tools

      Spring Boot provides a library called Spring Dev Tools [1] that makes development easier by providing things like auto-restart on source code changes. This is implemented by an additional classloader which can result in problems with object (de-)serialization (see linked support case).

      Let's evaluate if we can improve things on our side to avoid such problems.

      [1] https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#using.devtools

        This is the controller panel for Smart Panels app

            [CAM-9043] Classloading issues when using Spring Boot Starter and Spring Dev Tools

            Thorben Lindhauer added a comment - - edited

            About classloading in Java and the engine:

            • Classloading is hierarchic:
              • Bootstrap classloader: the root of the hierarchy
              • Extension Classloader: Next in line
              • System Classloader: Loads classes from the application's classpath; is the root of any further application-defined classloaders (e.g. the hierarchy an application server would set up)
              • See https://www.baeldung.com/java-classloaders for details
            • Classes have an initiating and a defining loader
              • Initiating classloader: the loader on which the loadClass method is called
              • Defining classloader: the loader that does the actual loading and defines the class (e.g. this can be a parent of the initiating loader)
            • The System Classloader always loads the main class (I assume)
            • Whenever a class A references another class B (e.g. as a method parameter), then the defining class loader of A is used to load B (of course, it may then delegate to another classloader that then becomes the defining classloader of B) (see https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-5.html#jvms-5.3)
            • It is up to the JVM to decide when exactly classes are loaded (e.g. are referenced classes loaded eagerly or lazily?); it also seems to differ a bit in practice (https://stackoverflow.com/questions/34259275/when-is-a-java-class-loaded)
            • Most libraries/APIs that load classes should have a dedicated Classloader parameter so that the calling code can choose. And usually it makes a lot of sense to use the calling class's defining loader (i.e. getClass().getClassLoader()), because that would be similar to the default behavior when classes reference each other directly
            • There are some exceptions:
              • JNDI API does not provide the classloader argument. JNDI classes themselves are loaded by the bootstrap classloader, but the resources are only know to the system classloader or a lower application classloader
              • To get access to that lower classloader, it uses the Thread's context classloader
              • For example, in Camunda, we have CdiResolver to resolve CDI beans/EJBs in expression. It looks up the BeanManager via JNDI (see BeanManagerLookup). That is why it is important that we modify the Thread context classloader when we do a context switch to a process application
              • The same partially applies to the loading of script engines (non-PA script engines are resolved from a ScriptEngineManager with the no-args constructor, which uses the context classloader (see org.camunda.bpm.engine.impl.scripting.engine.ScriptingEngines.ScriptingEngines(ScriptBindingsFactory)); PA script engines use the PA classloader (see org.camunda.bpm.application.impl.ProcessApplicationScriptEnvironment.getScriptEngineForName(String, boolean)))
              • For ObjectInputStreams it is yet a bit different: They search for the latest classloader on the stack (see java.io.ObjectInputStream.resolveClass(ObjectStreamClass) and java.io.ObjectInputStream.latestUserDefinedLoader())
              • Apparently these exceptions are also the only reason the thread context classloader exists
              • See https://stackoverflow.com/questions/1771679/difference-between-threads-context-class-loader-and-normal-classloader for details

            Applying this to the problem here:

            • With Spring Dev Tools, the Spring Beans are loaded with the RestartClassLoader
            • The referenced classes are loaded with the same classloader, in particular the DTO class that we use for the variable is also loaded with the RestartClassLoader
            • After the async continuation and when the variable is deserialized (in case of Java serialization), org.camunda.bpm.engine.impl.variable.serializer.JavaObjectSerializer.ClassloaderAwareObjectInputStream.resolveClass(ObjectStreamClass) loads the class with the Thread context classloader
            • Jobs are always executed with the process engine's defining classloader as the thread context classloader (See org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.switchClassLoader())
            • When Spring Dev tools are used, this is not the RestartClassLoader, because that one by default only takes care of classes located in directories on the classpath (and the engine comes as a jar), see https://docs.spring.io/spring-boot/docs/current/reference/html/using-spring-boot.html#using-boot-devtools-restart
              • I was able to verify with debugging: The restart classloader is the initiating classloader, but it delegates to the parent which then becomes the defining classloader
              • If I set the engine dependency to a snapshot and resolve it from within Eclipse's workspace, then I can see that the engine is also defined by the RestartClassLoader (=> the application startup fails however, because I have not all engine dependencies in the workspace and other classes that reference the engine classes will still be loaded by the root classloader, potentially then loading the engine classes twice)

            Solutions:

            • Users can tell the engine to use the RestartClassLoader for class loading (via an engine plugin)
            • The Spring Boot Starter can contain a corresponding process engine plugin that is activated if
              • with @ConditionalOnClass that changes the engine classloader when dev tools are used
              • with a configuration property

            Example plugin:

            import org.camunda.bpm.engine.ProcessEngine;
            import org.camunda.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl;
            import org.camunda.bpm.engine.impl.cfg.ProcessEnginePlugin;
            import org.springframework.beans.BeansException;
            import org.springframework.context.ApplicationContext;
            import org.springframework.context.ApplicationContextAware;
            import org.springframework.stereotype.Service;
            
            @Service
            public class MyEnginePlugin implements ProcessEnginePlugin, ApplicationContextAware {
            
              private ClassLoader applicationContextClassloader;
            
              @Override
              public void preInit(ProcessEngineConfigurationImpl processEngineConfiguration) {
                processEngineConfiguration.setClassLoader(applicationContextClassloader);
              }
            
              @Override
              public void postInit(ProcessEngineConfigurationImpl processEngineConfiguration) {
              }
            
              @Override
              public void postProcessEngineBuild(ProcessEngine processEngine) {
              }
            
              @Override
              public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
                applicationContextClassloader = applicationContext.getClassLoader();
              }
            }
            

            Thorben Lindhauer added a comment - - edited About classloading in Java and the engine: Classloading is hierarchic: Bootstrap classloader: the root of the hierarchy Extension Classloader: Next in line System Classloader: Loads classes from the application's classpath; is the root of any further application-defined classloaders (e.g. the hierarchy an application server would set up) See https://www.baeldung.com/java-classloaders for details Classes have an initiating and a defining loader Initiating classloader: the loader on which the loadClass method is called Defining classloader: the loader that does the actual loading and defines the class (e.g. this can be a parent of the initiating loader) The System Classloader always loads the main class (I assume) Whenever a class A references another class B (e.g. as a method parameter), then the defining class loader of A is used to load B (of course, it may then delegate to another classloader that then becomes the defining classloader of B) (see https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-5.html#jvms-5.3 ) It is up to the JVM to decide when exactly classes are loaded (e.g. are referenced classes loaded eagerly or lazily?); it also seems to differ a bit in practice ( https://stackoverflow.com/questions/34259275/when-is-a-java-class-loaded ) Most libraries/APIs that load classes should have a dedicated Classloader parameter so that the calling code can choose. And usually it makes a lot of sense to use the calling class's defining loader (i.e. getClass().getClassLoader()), because that would be similar to the default behavior when classes reference each other directly There are some exceptions: JNDI API does not provide the classloader argument. JNDI classes themselves are loaded by the bootstrap classloader, but the resources are only know to the system classloader or a lower application classloader To get access to that lower classloader, it uses the Thread's context classloader For example, in Camunda, we have CdiResolver to resolve CDI beans/EJBs in expression. It looks up the BeanManager via JNDI (see BeanManagerLookup ). That is why it is important that we modify the Thread context classloader when we do a context switch to a process application The same partially applies to the loading of script engines (non-PA script engines are resolved from a ScriptEngineManager with the no-args constructor, which uses the context classloader (see org.camunda.bpm.engine.impl.scripting.engine.ScriptingEngines.ScriptingEngines(ScriptBindingsFactory)); PA script engines use the PA classloader (see org.camunda.bpm.application.impl.ProcessApplicationScriptEnvironment.getScriptEngineForName(String, boolean))) For ObjectInputStreams it is yet a bit different: They search for the latest classloader on the stack (see java.io.ObjectInputStream.resolveClass(ObjectStreamClass) and java.io.ObjectInputStream.latestUserDefinedLoader()) "if there is a method on the current thread's stack whose declaring class was defined by a user-defined class loader (and was not a generated to implement reflective invocations), then loader is class loader corresponding to the closest such method to the currently executing frame" (via https://docs.oracle.com/javase/7/docs/api/java/io/ObjectInputStream.html#resolveClass(java.io.ObjectStreamClass )) Apparently these exceptions are also the only reason the thread context classloader exists See https://stackoverflow.com/questions/1771679/difference-between-threads-context-class-loader-and-normal-classloader for details Applying this to the problem here: With Spring Dev Tools, the Spring Beans are loaded with the RestartClassLoader The referenced classes are loaded with the same classloader, in particular the DTO class that we use for the variable is also loaded with the RestartClassLoader After the async continuation and when the variable is deserialized (in case of Java serialization), org.camunda.bpm.engine.impl.variable.serializer.JavaObjectSerializer.ClassloaderAwareObjectInputStream.resolveClass(ObjectStreamClass) loads the class with the Thread context classloader Jobs are always executed with the process engine's defining classloader as the thread context classloader (See org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.switchClassLoader()) When Spring Dev tools are used, this is not the RestartClassLoader, because that one by default only takes care of classes located in directories on the classpath (and the engine comes as a jar), see https://docs.spring.io/spring-boot/docs/current/reference/html/using-spring-boot.html#using-boot-devtools-restart I was able to verify with debugging: The restart classloader is the initiating classloader, but it delegates to the parent which then becomes the defining classloader If I set the engine dependency to a snapshot and resolve it from within Eclipse's workspace, then I can see that the engine is also defined by the RestartClassLoader (=> the application startup fails however, because I have not all engine dependencies in the workspace and other classes that reference the engine classes will still be loaded by the root classloader, potentially then loading the engine classes twice) Solutions: Users can tell the engine to use the RestartClassLoader for class loading (via an engine plugin) The Spring Boot Starter can contain a corresponding process engine plugin that is activated if with @ConditionalOnClass that changes the engine classloader when dev tools are used with a configuration property Example plugin: import org.camunda.bpm.engine.ProcessEngine; import org.camunda.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl; import org.camunda.bpm.engine.impl.cfg.ProcessEnginePlugin; import org.springframework.beans.BeansException; import org.springframework.context.ApplicationContext; import org.springframework.context.ApplicationContextAware; import org.springframework.stereotype.Service; @Service public class MyEnginePlugin implements ProcessEnginePlugin, ApplicationContextAware { private ClassLoader applicationContextClassloader; @Override public void preInit(ProcessEngineConfigurationImpl processEngineConfiguration) { processEngineConfiguration.setClassLoader(applicationContextClassloader); } @Override public void postInit(ProcessEngineConfigurationImpl processEngineConfiguration) { } @Override public void postProcessEngineBuild(ProcessEngine processEngine) { } @Override public void setApplicationContext(ApplicationContext applicationContext) throws BeansException { applicationContextClassloader = applicationContext.getClassLoader(); } }

              hariharan.parasuraman Hariharan Parasuraman
              thorben.lindhauer Thorben Lindhauer
              Yana Vasileva Yana Vasileva
              Tassilo Weidner Tassilo Weidner
              Hariharan Parasuraman Hariharan Parasuraman
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: