Uploaded image for project: 'camunda BPM'
  1. camunda BPM
  2. CAM-9811

Engine fails on WebSphere 8.5/9 with certain log levels

XMLWordPrintable

    • Icon: Bug Report Bug Report
    • Resolution: Fixed
    • Icon: L3 - Default L3 - Default
    • 7.12.0, 7.11.5, 7.12.0-alpha5
    • 7.10.3, 7.12.0-alpha4
    • None
    • None

      What are the steps to reproduce your problem?

      1. Perform a manual installation of the engine on WebSphere 8.5. Testing environment was WebSphere 8.5, JDK7 and Oracle 12.1 (ojdbc7).
      2. Confirm that the engine starts successfully
      3. Set the log levels to the following:
        *=info:WAS.j2c=all
      4. Restart WebSphere
      5. Attempt to start the engine application, you should get an exception
      [2/21/19 17:38:29:375 CET] 00000065 RALifeCycleMa E   J2CA0052E: The lookup of the Activation Specification with JNDI Name  eis/PlatformJobExecutorActivation failed due to the following exception: javax.nam
ing.NameNotFoundException: Context: localhostNode01Cell/nodes/localhostNode01/servers/server1, name: eis/PlatformJobExecutorActivation: First component in name PlatformJobExecutorActivation not found. [Root
 exception is org.omg.CosNaming.NamingContextPackage.NotFound: IDL:omg.org/CosNaming/NamingContext/NotFound:1.0]
        at com.ibm.ws.naming.jndicos.CNContextImpl.mapNotFoundException(CNContextImpl.java:4564)
        at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1822)
        at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1777)
        at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1434)
        at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:616)
        at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:165)
        at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:179)
        at org.apache.aries.jndi.DelegateContext.lookup(DelegateContext.java:161)
        at javax.naming.InitialContext.lookup(InitialContext.java:423)
        at com.ibm.ejs.j2c.RALifeCycleManagerImpl.activateEndpoint(RALifeCycleManagerImpl.java:1770)
        at com.ibm.ejs.container.MessageEndpointFactoryImpl.activateEndpoint(MessageEndpointFactoryImpl.java:168)
        at com.ibm.ws.runtime.component.EJBContainerImpl.startMDBs(EJBContainerImpl.java:4726)
        at com.ibm.ws.runtime.component.WASEJBRuntimeImpl.startMDBs(WASEJBRuntimeImpl.java:686)
        at com.ibm.ws.ejbcontainer.runtime.AbstractEJBRuntime.startModule(AbstractEJBRuntime.java:701)
        at com.ibm.ws.ejbcontainer.runtime.SharedEJBRuntimeImpl.startModule(SharedEJBRuntimeImpl.java:338)
        at com.ibm.ws.runtime.component.EJBContainerImpl.start(EJBContainerImpl.java:3588)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1211)
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1390)
        at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:979)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:795)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplicationDynamically(ApplicationMgrImpl.java:1413)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2273)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436)
        at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$1.run(CompositionUnitMgrImpl.java:654)
        at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5464)
        at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5680)
        at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:668)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:612)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:1303)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
        at java.lang.reflect.Method.invoke(Method.java:620)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:88)
        at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
        at java.lang.reflect.Method.invoke(Method.java:620)

      What is the problem?
      Engine is crashing.

      What would be the expected behavior:
      Engine should start successfully.

      Additional info:

        This is the controller panel for Smart Panels app

          1. SystemOut -extract.log
            116 kB
          2. SystemOut-7.10.3-extract.log
            130 kB
          3. SystemOut-7.10.3.log
            345 kB

            [CAM-9811] Engine fails on WebSphere 8.5/9 with certain log levels

            Thorben Lindhauer added a comment - - edited

            Analysis result: This is a bug in Websphere, but we can implement a workaround.

            Bug in Websphere:

            • The job execution resource adapter fails to be activated. The reason is a NullPointerException in the #toString method of the class com.ibm.ejs.j2c.ActivationSpecBindingInfo. ActivationSpecBindingInfo seems to be a Websphere internal wrapper class around activation spec classes, in our case org.camunda.bpm.container.impl.threading.ra.inflow.JobExecutionHandlerActivationSpec.
            • The #toString method is only called if trace logging is activated. I found this by debugging the corresponding websphere jar
            • The problem occurs when an activation spec is defined that defines no properties. ActivationSpecBindingInfo then wrongfully does not initialize a properties hashmap, but in #toString tries to iterate it

            Stacktrace of the NPE (cannot be found in the logs as is, maybe it can be by activating another logger; I obtained it from the debugger as a variable in the stack):

            [com.ibm.ejs.j2c.ActivationSpecBindingInfo.toString(ActivationSpecBindingInfo.java:202), java.lang.String.valueOf(String.java:1862), java.lang.StringBuilder.append(StringBuilder.java:218), com.ibm.ejs.j2c.ActivationSpecBindingInfo.initialize(ActivationSpecBindingInfo.java:188), com.ibm.ejs.j2c.RALifeCycleManagerImpl.startRA(RALifeCycleManagerImpl.java:1309), com.ibm.ejs.j2c.RALifeCycleManagerImpl.resourceProviderEvent(RALifeCycleManagerImpl.java:923), com.ibm.ws.runtime.component.ResourceMgrImpl$ResourceEventListenerFilterProxy.resourceProviderEvent(ResourceMgrImpl.java:2208), com.ibm.ws.runtime.component.ResourceMgrImpl.notifyResourceEventListeners(ResourceMgrImpl.java:1622), com.ibm.ws.runtime.component.ResourceMgrImpl.installResourceProvider(ResourceMgrImpl.java:1579), com.ibm.ws.runtime.component.ResourceMgrImpl.installResourceProvider(ResourceMgrImpl.java:1462), com.ibm.ejs.j2c.RALifeCycleManagerImpl.start(RALifeCycleManagerImpl.java:720), com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1246), com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1514), com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:704), com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:1096), com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:798), com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2314), com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5398), com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5614), com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255), com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2319), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436), com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127), com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:985), com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502), com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)]

            Workaround:

            • declare a dummy POJO property in JobExecutionHandlerActivationSpec, e.g. as follows:
              private String foo;
  
  public void setFoo(String foo) {
    this.foo = foo;
  }
  
  public String getFoo() {
    return foo;
  }

            Thorben Lindhauer added a comment - - edited Analysis result: This is a bug in Websphere, but we can implement a workaround. Bug in Websphere: The job execution resource adapter fails to be activated. The reason is a NullPointerException in the #toString method of the class com.ibm.ejs.j2c.ActivationSpecBindingInfo . ActivationSpecBindingInfo seems to be a Websphere internal wrapper class around activation spec classes, in our case org.camunda.bpm.container.impl.threading.ra.inflow.JobExecutionHandlerActivationSpec . The #toString method is only called if trace logging is activated. I found this by debugging the corresponding websphere jar The problem occurs when an activation spec is defined that defines no properties. ActivationSpecBindingInfo then wrongfully does not initialize a properties hashmap, but in #toString tries to iterate it Stacktrace of the NPE (cannot be found in the logs as is, maybe it can be by activating another logger; I obtained it from the debugger as a variable in the stack): [com.ibm.ejs.j2c.ActivationSpecBindingInfo.toString(ActivationSpecBindingInfo.java:202), java.lang. String .valueOf( String .java:1862), java.lang.StringBuilder.append(StringBuilder.java:218), com.ibm.ejs.j2c.ActivationSpecBindingInfo.initialize(ActivationSpecBindingInfo.java:188), com.ibm.ejs.j2c.RALifeCycleManagerImpl.startRA(RALifeCycleManagerImpl.java:1309), com.ibm.ejs.j2c.RALifeCycleManagerImpl.resourceProviderEvent(RALifeCycleManagerImpl.java:923), com.ibm.ws.runtime.component.ResourceMgrImpl$ResourceEventListenerFilterProxy.resourceProviderEvent(ResourceMgrImpl.java:2208), com.ibm.ws.runtime.component.ResourceMgrImpl.notifyResourceEventListeners(ResourceMgrImpl.java:1622), com.ibm.ws.runtime.component.ResourceMgrImpl.installResourceProvider(ResourceMgrImpl.java:1579), com.ibm.ws.runtime.component.ResourceMgrImpl.installResourceProvider(ResourceMgrImpl.java:1462), com.ibm.ejs.j2c.RALifeCycleManagerImpl.start(RALifeCycleManagerImpl.java:720), com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1246), com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1514), com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:704), com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:1096), com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:798), com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2314), com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5398), com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5614), com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255), com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2319), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436), com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127), com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:985), com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502), com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)] Workaround: declare a dummy POJO property in JobExecutionHandlerActivationSpec , e.g. as follows: private String foo; public void setFoo( String foo) { this .foo = foo; } public String getFoo() { return foo; }

              nikola.koevski Nikola Koevski
              nikola.koevski Nikola Koevski
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: