L3 - Default Environment (Required on creation):
- Application server environments with multiple deployments and parallel deployment threads, e.g. WildFly distribution with multiple process application deployments
Description (Required on creation; please attach any relevant screenshots, stacktraces, log files, etc. to the ticket):
Application deployment fails irregularly for any of the deployments with a ConcurrentModificationException
2021-12-21 08:35:18,114 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service jboss.deployment.unit."PROD_DEPLOYMENT-war-1.125.67.war".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.unit."PROD_DEPLOYMENT-war-1.125.67.war".POST_MODULE: WFLYSRV0153: Failed to process phase POST_MODULE of deployment "PROD_DEPLOYMENT-war-1.125.67.war" at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:183) [wildfly-server-10.1.23.Final-redhat-00001-jbeap-22415.jar:10.1.23.Final-redhat-00001-jbeap-22415] at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001] at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001] at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_301] Caused by: org.camunda.bpm.engine.ProcessEngineException: ENGINE-09003 Could not parse 'vfs:/content/PROD_DEPLOYMENT-war-1.125.67.war/WEB-INF/classes/META-INF/processes.xml'. null at org.camunda.bpm.engine.impl.util.EngineUtilLogger.parsingFailureException(EngineUtilLogger.java:50) at org.camunda.bpm.engine.impl.util.xml.Parse.execute(Parse.java:164) at org.camunda.bpm.container.impl.metadata.DeploymentMetadataParse.execute(DeploymentMetadataParse.java:50) at org.camunda.bpm.application.impl.metadata.ProcessesXmlParse.execute(ProcessesXmlParse.java:60) at org.camunda.bpm.container.impl.jboss.deployment.processor.ProcessesXmlProcessor.parseProcessesXml(ProcessesXmlProcessor.java:176) at org.camunda.bpm.container.impl.jboss.deployment.processor.ProcessesXmlProcessor.deploy(ProcessesXmlProcessor.java:85) at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:176) [wildfly-server-10.1.23.Final-redhat-00001-jbeap-22415.jar:10.1.23.Final-redhat-00001-jbeap-22415] ... 8 more Caused by: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445) [rt.jar:1.8.0_301] at java.util.HashMap$EntryIterator.next(HashMap.java:1479) [rt.jar:1.8.0_301] at java.util.HashMap$EntryIterator.next(HashMap.java:1477) [rt.jar:1.8.0_301] at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.setFeatures(SAXParserImpl.java:247) [rt.jar:1.8.0_301] at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:182) [rt.jar:1.8.0_301] at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:123) [rt.jar:1.8.0_301] at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.newSAXParserImpl(SAXParserFactoryImpl.java:96) [rt.jar:1.8.0_301] at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.setFeature(SAXParserFactoryImpl.java:134) [rt.jar:1.8.0_301] at __redirected.__SAXParserFactory.setFeature(__SAXParserFactory.java:96) [jboss-modules.jar:1.11.0.Final-redhat-00001] at org.camunda.bpm.engine.impl.util.xml.Parse.execute(Parse.java:140) ... 13 more
Steps to reproduce (Required on creation):
This is not easy to reproduce as this is based on a concurrent access pattern to a class instance that is not thread-safe. The general scheme here is:
- Create multiple process application deployments with a processes.xml each
- Place those deployments into the application server, e.g., a WildFly instance
- Start the application server
Observed Behavior (Required on creation):
Occasionally, one or more deployments fail due to a ConcurrentModificationException.
Expected behavior (Required on creation):
All deployments are started successfully.
Root Cause (Required on prioritization):
- Upon startup of a process application deployment, the processes.xml file is parsed by the engine
- Parsing this file is done by a SAXParser that is initialized with certain parser features - see https://github.com/camunda/camunda-bpm-platform/blob/0b6a8f89243c69dd46fd8d5ccb5451f031b305ca/engine/src/main/java/org/camunda/bpm/engine/impl/util/xml/Parse.java#L141-L146
- The deployments are started by concurrent threads in the application server, accessing the linked code concurrently as well
- The SAXParserFactory we use is NOT THREAD-SAFE, we reuse the same factory instance created once in the Parser for all threads - created over here and fetched over here
- Setting features on the same factory instance concurrently can lead to modifying the underlying shared parser features HashMap in one thread while iterating over it in another one, leading to the exception over at java.util.HashMap.HashIterator#nextNode (can be found at HashMap.java line 1445 in JDK8_301).
Solution Ideas (Optional):
There are at least two ways of preventing this:
- Avoid concurrent access to the code in question, e.g., by synchronizing the initialization code block of the parser factory between threads using Java synchronization
- Avoid using the same parser factory instance in multiple threads, e.g., by keeping a parser factory instance per thread using ThreadLocal
Hints (optional):
- The issue occurs in all supported JDKs, as far as I can tell