-
Bug Report
-
Resolution: Fixed
-
L3 - Default
-
7.3.0
-
None
-
JBoss AS7
Deploying an EAR file containing Skinny-WARs (multiple WARs with shared libs in the EAR) to JBoss (AS7), leads sometimes to a NullPointerException.
The problem is, that the class org.camunda.bpm.container.impl.jboss.deployment.processor.ModuleDependencyProcessor.java adds system dependencies to OTHER sibling subdeployment-units. JBoss starts the subdeployment for each WAR file inside the EAR as subdeployment in parallel. This causes concurrency problems, because each WAR subdeployment adds the camunda libs as system dependencies to all other WARs.
The access to ModuleSpecification.addSystemDependency(), which adds the camunda libraries, is not synchronized and finally leads to an inconsistent system-dependency-list containing null values. As a result JBoss throws some times later a NPE:
08:07:58,214 ERROR [MSC service thread 1-47] org.jboss.msc.service.ServiceControllerImpl$StartTask.startFailed:1987 - MSC000001: Failed to start service jboss.deployment.subunit."GUIWF.ear"."CCP_ConfCalc.jar".CONFIGURE_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.subunit."GUIWF.ear"."CCP_ConfCalc.jar".CONFIGURE_MODULE: JBAS018733: Verarbeitung von Phase CONFIGURE_MODULE von subdeployment "CCP_ConfCalc.jar" of deployment "GUIWF.ear" fehlgeschlagen
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:166)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.jboss.as.server.deployment.module.ModuleSpecProcessor.createDependencies(ModuleSpecProcessor.java:237)
at org.jboss.as.server.deployment.module.ModuleSpecProcessor.createModuleService(ModuleSpecProcessor.java:182)
at org.jboss.as.server.deployment.module.ModuleSpecProcessor.deployModuleSpec(ModuleSpecProcessor.java:116)
at org.jboss.as.server.deployment.module.ModuleSpecProcessor.deploy(ModuleSpecProcessor.java:79)
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:159)
... 5 more
The same problem was reported and fixed for the JBoss JSFDependencyProcessor: https://issues.jboss.org/browse/WFLY-2496
We are currently working with camunda release 7.3, but as I looked in the sources, it should still be open.
Solution?
In general, it seems not to be expected, that one Subdeployment unit adds system dependencies to another Subdeployment unit. So I can not present a clean solution, but adding a "synchronized" to the method ModuleDependencyProcessor.addSystemDependency() should decrease the concurrency possibility.
This is the controller panel for Smart Panels app
[CAM-6322] Deployment of EAR with multiple WARs fails with NPE in Camunda JBoss Subsystem
Hi Thorben,
thanks for your help!
Finally I created a pull request with the changes above:
https://github.com/camunda/camunda-bpm-platform/pull/231
Hi Ferenc,
would you be able to provide a sample ear for testing? or pom descriptors.
It is not easy to reproduce the error, because the concurrency depends on memory, cpu speed, number of deployed application, and much more ...
But you can check, whether there are unallowed concurrent accesses.
1. Start JBoss in debug mode
2. Create a method Breakpoint in org.camunda.bpm.container.impl.jboss.deployment.processor.ModuleDependencyProcessor.deploy(DeploymentPhaseContext phaseContext)
3. Deploy WFGUI.ear
There are four concurrent calls to deploy() for each subdeployment.
And each of this calls adds the system dependencies to the same instances of module specification (parent depoyment and all sibling deployments). This produces the error.
Hi Ferenc,
christopher.zell and I looked into the issue today and we can see what is going wrong and understand your pull request. Christopher is going to simplify the code a bit and put the fix on master.
Best Regards,
Daniel
Hi Daniel,
Hi Christopher,
that are good News.
For your Information: We have installed the patched version for 7.3.0 at our development server and it runs stable for two weeks now. 
Best regards,
feri
I see you found a simpler solution, which adds all system depenencies from the root deployment (EAR or WAR) and such avoids concurrency between the subdeployments.
I have tested your changes in the master and it worked well.
Best regards,
feri
Hey Ferenc,
thanks for your feedback and also for your pull request.
It helps us to identify the problem and to code a simpler solution.
I'm glad that our solution works well for you.
Best regards,
Chris
Fork the repository, create a branch in that repository where you apply the changes, then create a pull request from that branch against the Camunda master branch.