[CAM-14283] Concurrent access to SAX parser factory fails application deployment

Type: Bug Report
Resolution: Fixed
Priority: L3 - Default
Fix Version/s: 7.11.21, 7.17.0, 7.16.6, 7.17.0-alpha4
Affects Version/s: None
Component/s: engine
Labels:
- SUPPORT

Environment (Required on creation):

Application server environments with multiple deployments and parallel deployment threads, e.g. WildFly distribution with multiple process application deployments

Description (Required on creation; please attach any relevant screenshots, stacktraces, log files, etc. to the ticket):

Application deployment fails irregularly for any of the deployments with a ConcurrentModificationException

2021-12-21 08:35:18,114 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service jboss.deployment.unit."PROD_DEPLOYMENT-war-1.125.67.war".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.unit."PROD_DEPLOYMENT-war-1.125.67.war".POST_MODULE: WFLYSRV0153: Failed to process phase POST_MODULE of deployment "PROD_DEPLOYMENT-war-1.125.67.war"
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:183) [wildfly-server-10.1.23.Final-redhat-00001-jbeap-22415.jar:10.1.23.Final-redhat-00001-jbeap-22415]
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001]
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001]
    at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001]
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_301]
Caused by: org.camunda.bpm.engine.ProcessEngineException: ENGINE-09003 Could not parse 'vfs:/content/PROD_DEPLOYMENT-war-1.125.67.war/WEB-INF/classes/META-INF/processes.xml'. null
    at org.camunda.bpm.engine.impl.util.EngineUtilLogger.parsingFailureException(EngineUtilLogger.java:50)
    at org.camunda.bpm.engine.impl.util.xml.Parse.execute(Parse.java:164)
    at org.camunda.bpm.container.impl.metadata.DeploymentMetadataParse.execute(DeploymentMetadataParse.java:50)
    at org.camunda.bpm.application.impl.metadata.ProcessesXmlParse.execute(ProcessesXmlParse.java:60)
    at org.camunda.bpm.container.impl.jboss.deployment.processor.ProcessesXmlProcessor.parseProcessesXml(ProcessesXmlProcessor.java:176)
    at org.camunda.bpm.container.impl.jboss.deployment.processor.ProcessesXmlProcessor.deploy(ProcessesXmlProcessor.java:85)
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:176) [wildfly-server-10.1.23.Final-redhat-00001-jbeap-22415.jar:10.1.23.Final-redhat-00001-jbeap-22415]
    ... 8 more
Caused by: java.util.ConcurrentModificationException
    at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445) [rt.jar:1.8.0_301]
    at java.util.HashMap$EntryIterator.next(HashMap.java:1479) [rt.jar:1.8.0_301]
    at java.util.HashMap$EntryIterator.next(HashMap.java:1477) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.setFeatures(SAXParserImpl.java:247) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:182) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:123) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.newSAXParserImpl(SAXParserFactoryImpl.java:96) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.setFeature(SAXParserFactoryImpl.java:134) [rt.jar:1.8.0_301]
    at __redirected.__SAXParserFactory.setFeature(__SAXParserFactory.java:96) [jboss-modules.jar:1.11.0.Final-redhat-00001]
    at org.camunda.bpm.engine.impl.util.xml.Parse.execute(Parse.java:140)
    ... 13 more

Steps to reproduce (Required on creation):

This is not easy to reproduce as this is based on a concurrent access pattern to a class instance that is not thread-safe. The general scheme here is:

Create multiple process application deployments with a processes.xml each
Place those deployments into the application server, e.g., a WildFly instance
Start the application server

Observed Behavior (Required on creation):

Occasionally, one or more deployments fail due to a ConcurrentModificationException.

Expected behavior (Required on creation):

All deployments are started successfully.

Root Cause (Required on prioritization):

Upon startup of a process application deployment, the processes.xml file is parsed by the engine
Parsing this file is done by a SAXParser that is initialized with certain parser features - see https://github.com/camunda/camunda-bpm-platform/blob/0b6a8f89243c69dd46fd8d5ccb5451f031b305ca/engine/src/main/java/org/camunda/bpm/engine/impl/util/xml/Parse.java#L141-L146
The deployments are started by concurrent threads in the application server, accessing the linked code concurrently as well
The SAXParserFactory we use is NOT THREAD-SAFE, we reuse the same factory instance created once in the Parser for all threads - created over here and fetched over here
Setting features on the same factory instance concurrently can lead to modifying the underlying shared parser features HashMap in one thread while iterating over it in another one, leading to the exception over at java.util.HashMap.HashIterator#nextNode (can be found at HashMap.java line 1445 in JDK8_301).

Solution Ideas (Optional):

There are at least two ways of preventing this:

Avoid concurrent access to the code in question, e.g., by synchronizing the initialization code block of the parser factory between threads using Java synchronization
Avoid using the same parser factory instance in multiple threads, e.g., by keeping a parser factory instance per thread using ThreadLocal

Hints (optional):

The issue occurs in all supported JDKs, as far as I can tell

This is the controller panel for Smart Panels app

Tobias Metzke-Bernstein created issue - 13/Jan/22 8:49 AM

Tobias Metzke-Bernstein made changes - 13/Jan/22 8:49 AM

Description

Original: h3. Environment (Required on creation):

* Application server environments with multiple deployments and parallel deployment threads, e.g. WildFly distribution with multiple process application deployments

h3. Description (Required on creation; please attach any relevant screenshots, stacktraces, log files, etc. to the ticket):

Application deployment fails irregularly for any of the deployments with a {{ConcurrentModificationException}}

{code}
2021-12-21 08:35:18,114 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service jboss.deployment.unit."PROD_DEPLOYMENT-war-1.125.67.war".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.unit."PROD_DEPLOYMENT-war-1.125.67.war".POST_MODULE: WFLYSRV0153: Failed to process phase POST_MODULE of deployment "PROD_DEPLOYMENT-war-1.125.67.war"
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:183) [wildfly-server-10.1.23.Final-redhat-00001-jbeap-22415.jar:10.1.23.Final-redhat-00001-jbeap-22415]
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001]
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001]
    at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001]
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_301]
Caused by: org.camunda.bpm.engine.ProcessEngineException: ENGINE-09003 Could not parse 'vfs:/content/PROD_DEPLOYMENT-war-1.125.67.war/WEB-INF/classes/META-INF/processes.xml'. null
    at org.camunda.bpm.engine.impl.util.EngineUtilLogger.parsingFailureException(EngineUtilLogger.java:50)
    at org.camunda.bpm.engine.impl.util.xml.Parse.execute(Parse.java:164)
    at org.camunda.bpm.container.impl.metadata.DeploymentMetadataParse.execute(DeploymentMetadataParse.java:50)
    at org.camunda.bpm.application.impl.metadata.ProcessesXmlParse.execute(ProcessesXmlParse.java:60)
    at org.camunda.bpm.container.impl.jboss.deployment.processor.ProcessesXmlProcessor.parseProcessesXml(ProcessesXmlProcessor.java:176)
    at org.camunda.bpm.container.impl.jboss.deployment.processor.ProcessesXmlProcessor.deploy(ProcessesXmlProcessor.java:85)
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:176) [wildfly-server-10.1.23.Final-redhat-00001-jbeap-22415.jar:10.1.23.Final-redhat-00001-jbeap-22415]
    ... 8 more
Caused by: java.util.ConcurrentModificationException
    at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445) [rt.jar:1.8.0_301]
    at java.util.HashMap$EntryIterator.next(HashMap.java:1479) [rt.jar:1.8.0_301]
    at java.util.HashMap$EntryIterator.next(HashMap.java:1477) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.setFeatures(SAXParserImpl.java:247) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:182) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:123) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.newSAXParserImpl(SAXParserFactoryImpl.java:96) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.setFeature(SAXParserFactoryImpl.java:134) [rt.jar:1.8.0_301]
    at __redirected.__SAXParserFactory.setFeature(__SAXParserFactory.java:96) [jboss-modules.jar:1.11.0.Final-redhat-00001]
    at org.camunda.bpm.engine.impl.util.xml.Parse.execute(Parse.java:140)
    ... 13 more
{code}

h3. Steps to reproduce (Required on creation):

* This is not easy to reproduce as this is based on a concurrent access pattern to a class instance that is not thread-safe. The general scheme here is:
# Create multiple process application deployments with a {{processes.xml}} each
# Place those deployments into the application server, e.g., a WildFly instance
# Start the application server

h3. Observed Behavior (Required on creation):

Occasionally, one or more deployments fail due to a {{ConcurrentModificationException}}.

h3. Expected behavior (Required on creation):

All deployments are started successfully.

h3. Root Cause (Required on prioritization):

* Upon startup of a process application deployment, the {{processes.xml}} file is parsed by the engine
* Parsing this file is done by a SAXParser that is initialized with certain parser features - see https://github.com/camunda/camunda-bpm-platform/blob/0b6a8f89243c69dd46fd8d5ccb5451f031b305ca/engine/src/main/java/org/camunda/bpm/engine/impl/util/xml/Parse.java#L141-L146
* The deployments are started by concurrent threads in the application server, accessing the linked code concurrently as well
* The {{SAXParserFactory}} we use is NOT THREAD-SAFE, we reuse the same factory instance created once in the Parser for all threads - created [over here|https://github.com/camunda/camunda-bpm-platform/blob/0b6a8f89243c69dd46fd8d5ccb5451f031b305ca/engine/src/main/java/org/camunda/bpm/engine/impl/util/xml/Parser.java#L28] and fetched [over here|https://github.com/camunda/camunda-bpm-platform/blob/0b6a8f89243c69dd46fd8d5ccb5451f031b305ca/engine/src/main/java/org/camunda/bpm/engine/impl/util/xml/Parse.java#L140]
* Setting features on the same factory instance concurrently can lead to modifying the underlying shared parser features HashMap in one thread while iterating over it in another one, leading to the exception over at {{java.util.HashMap.HashIterator#nextNode}} (can be found at HashMap.java line 1445 in JDK8_301).

h3. Solution Ideas (Optional):

There are at least two ways of preventing this:
# Avoid concurrent access to the code in question, e.g., by synchronizing the initialization code block of the parser factory between threads using Java synchronization
# Avoid using the same parser factory instance in multiple threads, e.g., by keeping a parser factory instance per thread using ThreadLocal

h3. Hints (optional):

* The issue occurs in all supported JDKs, as far as I can tell

New: h3. Environment (Required on creation):

* Application server environments with multiple deployments and parallel deployment threads, e.g. WildFly distribution with multiple process application deployments

h3. Description (Required on creation; please attach any relevant screenshots, stacktraces, log files, etc. to the ticket):

Application deployment fails irregularly for any of the deployments with a {{ConcurrentModificationException}}

{code}
2021-12-21 08:35:18,114 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service jboss.deployment.unit."PROD_DEPLOYMENT-war-1.125.67.war".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.unit."PROD_DEPLOYMENT-war-1.125.67.war".POST_MODULE: WFLYSRV0153: Failed to process phase POST_MODULE of deployment "PROD_DEPLOYMENT-war-1.125.67.war"
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:183) [wildfly-server-10.1.23.Final-redhat-00001-jbeap-22415.jar:10.1.23.Final-redhat-00001-jbeap-22415]
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001]
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001]
    at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559) [jboss-msc-1.4.11.Final-redhat-00001.jar:1.4.11.Final-redhat-00001]
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363) [jboss-threads-2.3.3.Final-redhat-00001.jar:2.3.3.Final-redhat-00001]
    at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_301]
Caused by: org.camunda.bpm.engine.ProcessEngineException: ENGINE-09003 Could not parse 'vfs:/content/PROD_DEPLOYMENT-war-1.125.67.war/WEB-INF/classes/META-INF/processes.xml'. null
    at org.camunda.bpm.engine.impl.util.EngineUtilLogger.parsingFailureException(EngineUtilLogger.java:50)
    at org.camunda.bpm.engine.impl.util.xml.Parse.execute(Parse.java:164)
    at org.camunda.bpm.container.impl.metadata.DeploymentMetadataParse.execute(DeploymentMetadataParse.java:50)
    at org.camunda.bpm.application.impl.metadata.ProcessesXmlParse.execute(ProcessesXmlParse.java:60)
    at org.camunda.bpm.container.impl.jboss.deployment.processor.ProcessesXmlProcessor.parseProcessesXml(ProcessesXmlProcessor.java:176)
    at org.camunda.bpm.container.impl.jboss.deployment.processor.ProcessesXmlProcessor.deploy(ProcessesXmlProcessor.java:85)
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:176) [wildfly-server-10.1.23.Final-redhat-00001-jbeap-22415.jar:10.1.23.Final-redhat-00001-jbeap-22415]
    ... 8 more
Caused by: java.util.ConcurrentModificationException
    at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445) [rt.jar:1.8.0_301]
    at java.util.HashMap$EntryIterator.next(HashMap.java:1479) [rt.jar:1.8.0_301]
    at java.util.HashMap$EntryIterator.next(HashMap.java:1477) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.setFeatures(SAXParserImpl.java:247) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:182) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:123) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.newSAXParserImpl(SAXParserFactoryImpl.java:96) [rt.jar:1.8.0_301]
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.setFeature(SAXParserFactoryImpl.java:134) [rt.jar:1.8.0_301]
    at __redirected.__SAXParserFactory.setFeature(__SAXParserFactory.java:96) [jboss-modules.jar:1.11.0.Final-redhat-00001]
    at org.camunda.bpm.engine.impl.util.xml.Parse.execute(Parse.java:140)
    ... 13 more
{code}

h3. Steps to reproduce (Required on creation):

This is not easy to reproduce as this is based on a concurrent access pattern to a class instance that is not thread-safe. The general scheme here is:
# Create multiple process application deployments with a {{processes.xml}} each
# Place those deployments into the application server, e.g., a WildFly instance
# Start the application server

h3. Observed Behavior (Required on creation):

Occasionally, one or more deployments fail due to a {{ConcurrentModificationException}}.

h3. Expected behavior (Required on creation):

All deployments are started successfully.

h3. Root Cause (Required on prioritization):

* Upon startup of a process application deployment, the {{processes.xml}} file is parsed by the engine
* Parsing this file is done by a SAXParser that is initialized with certain parser features - see https://github.com/camunda/camunda-bpm-platform/blob/0b6a8f89243c69dd46fd8d5ccb5451f031b305ca/engine/src/main/java/org/camunda/bpm/engine/impl/util/xml/Parse.java#L141-L146
* The deployments are started by concurrent threads in the application server, accessing the linked code concurrently as well
* The {{SAXParserFactory}} we use is NOT THREAD-SAFE, we reuse the same factory instance created once in the Parser for all threads - created [over here|https://github.com/camunda/camunda-bpm-platform/blob/0b6a8f89243c69dd46fd8d5ccb5451f031b305ca/engine/src/main/java/org/camunda/bpm/engine/impl/util/xml/Parser.java#L28] and fetched [over here|https://github.com/camunda/camunda-bpm-platform/blob/0b6a8f89243c69dd46fd8d5ccb5451f031b305ca/engine/src/main/java/org/camunda/bpm/engine/impl/util/xml/Parse.java#L140]
* Setting features on the same factory instance concurrently can lead to modifying the underlying shared parser features HashMap in one thread while iterating over it in another one, leading to the exception over at {{java.util.HashMap.HashIterator#nextNode}} (can be found at HashMap.java line 1445 in JDK8_301).

h3. Solution Ideas (Optional):

There are at least two ways of preventing this:
# Avoid concurrent access to the code in question, e.g., by synchronizing the initialization code block of the parser factory between threads using Java synchronization
# Avoid using the same parser factory instance in multiple threads, e.g., by keeping a parser factory instance per thread using ThreadLocal

h3. Hints (optional):

* The issue occurs in all supported JDKs, as far as I can tell

Tobias Metzke-Bernstein made changes - 13/Jan/22 8:49 AM

Mentioned Roles

Tobias Metzke-Bernstein made changes - 13/Jan/22 8:49 AM

Mentioned Groups

Tobias Metzke-Bernstein made changes - 13/Jan/22 8:50 AM

Labels

New: SUPPORT

Tobias Metzke-Bernstein made changes - 13/Jan/22 8:50 AM

Link

New: This issue is related to SUPPORT-12554 [ SUPPORT-12554 ]

Tobias Metzke-Bernstein made changes - 13/Jan/22 8:54 AM

Status

Original: Open [ 1 ]

New: Ready [ 10005 ]

Thorben Lindhauer made changes - 14/Jan/22 3:57 PM

Fix Version/s		New: 7.11.x [ 15373 ]
Fix Version/s		New: 7.16.x [ 16897 ]
Fix Version/s		New: 7.17.0 [ 16007 ]

Thorben Lindhauer made changes - 14/Jan/22 3:58 PM

Mentioned Roles

Thorben Lindhauer made changes - 14/Jan/22 3:58 PM

Mentioned Groups

Assignee:: Tassilo Weidner

Reporter:: Tobias Metzke-Bernstein

DRI:: Tassilo Weidner

Reviewer:: Nikola Koevski

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Created:: 13/Jan/22 8:49 AM

Updated:: 07/Feb/22 8:51 AM

Resolved:: 26/Jan/22 1:13 PM

Details

Description

Environment (Required on creation):

Description (Required on creation; please attach any relevant screenshots, stacktraces, log files, etc. to the ticket):

Steps to reproduce (Required on creation):

Observed Behavior (Required on creation):

Expected behavior (Required on creation):

Root Cause (Required on prioritization):

Solution Ideas (Optional):

Hints (optional):

mgm-controller-panel

This is the controller panel for Smart Panels app

Attachments

Activity

People

Dates

Salesforce