[CAM-3810] Concurrent process application deployment causes deadlock

Type: Bug Report
Resolution: Fixed
Priority: L3 - Default
Fix Version/s: 7.4.0, 7.2.6, 7.1.11, 7.3.3, 7.4.0-alpha1, 7.5.0-alpha3
Affects Version/s: 7.2.4
Component/s: engine
Labels:
- SUPPORT
Environment:
GNU/Linux, JBoss 7, java version "1.6.0_31", mysql

How to reproduce:

start the engine with min. 4 parallel deployments

Problem:
During deployment process, jboss throw many server errors and doesn't deploy the application module

Log:

SEVERE [org.camunda.bpm.engine.impl.interceptor.CommandContext] (ServerService Thread Pool -- 50) Error while closing command context: org.apache.ibatis.exceptions.PersistenceException: 
### Error updating database.  Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: SELECT * FROM ACT_GE_PROPERTY WHERE NAME_ = 'deployment.lock' for update
### Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:154)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:145)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.lock(DbSqlSession.java:110) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.lock(DbEntityManager.java:242) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.persistence.entity.PropertyManager.acquireExclusiveLock(PropertyManager.java:31) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.acquireExclusiveLock(DeployCmd.java:112) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:75) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:61) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:97) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:59) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.RepositoryServiceImpl.deploy(RepositoryServiceImpl.java:82) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.repository.DeploymentBuilderImpl.deploy(DeploymentBuilderImpl.java:125) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.repository.ProcessApplicationDeploymentBuilderImpl.deploy(ProcessApplicationDeploymentBuilderImpl.java:51) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:167)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:164)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runWithTccl(Tccl.java:53)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runUnderClassloader(Tccl.java:45)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService.performDeployment(ProcessApplicationDeploymentService.java:164)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$1.run(ProcessApplicationDeploymentService.java:87)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [rt.jar:1.6.0_31]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_31]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_31]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
	at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1362)
	at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:404)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:44)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:69)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
	... 27 more

13:38:53,675 SEVERE [org.camunda.bpm.engine.impl.interceptor.CommandContext] (ServerService Thread Pool -- 51) Error while closing command context: org.apache.ibatis.exceptions.PersistenceException: 
### Error updating database.  Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: SELECT * FROM ACT_GE_PROPERTY WHERE NAME_ = 'deployment.lock' for update
### Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:154)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:145)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.lock(DbSqlSession.java:110) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.lock(DbEntityManager.java:242) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.persistence.entity.PropertyManager.acquireExclusiveLock(PropertyManager.java:31) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.acquireExclusiveLock(DeployCmd.java:112) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:75) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:61) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:97) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:59) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.RepositoryServiceImpl.deploy(RepositoryServiceImpl.java:82) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.repository.DeploymentBuilderImpl.deploy(DeploymentBuilderImpl.java:125) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.repository.ProcessApplicationDeploymentBuilderImpl.deploy(ProcessApplicationDeploymentBuilderImpl.java:51) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:167)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:164)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runWithTccl(Tccl.java:53)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runUnderClassloader(Tccl.java:45)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService.performDeployment(ProcessApplicationDeploymentService.java:164)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$1.run(ProcessApplicationDeploymentService.java:87)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [rt.jar:1.6.0_31]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_31]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_31]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
	at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1362)
	at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:404)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:44)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:69)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
	... 27 more

13:38:53,673 SEVERE [org.camunda.bpm.engine.impl.interceptor.CommandContext] (ServerService Thread Pool -- 60) Error while closing command context: org.apache.ibatis.exceptions.PersistenceException: 
### Error updating database.  Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: SELECT * FROM ACT_GE_PROPERTY WHERE NAME_ = 'deployment.lock' for update
### Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:154)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:145)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.lock(DbSqlSession.java:110) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.lock(DbEntityManager.java:242) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.persistence.entity.PropertyManager.acquireExclusiveLock(PropertyManager.java:31) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.acquireExclusiveLock(DeployCmd.java:112) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:75) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:61) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:97) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:59) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.RepositoryServiceImpl.deploy(RepositoryServiceImpl.java:82) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.repository.DeploymentBuilderImpl.deploy(DeploymentBuilderImpl.java:125) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.engine.impl.repository.ProcessApplicationDeploymentBuilderImpl.deploy(ProcessApplicationDeploymentBuilderImpl.java:51) [camunda-engine-7.2.4-ee.jar:7.2.4-ee]
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:167)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:164)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runWithTccl(Tccl.java:53)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runUnderClassloader(Tccl.java:45)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService.performDeployment(ProcessApplicationDeploymentService.java:164)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$1.run(ProcessApplicationDeploymentService.java:87)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [rt.jar:1.6.0_31]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_31]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_31]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
	at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1362)
	at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:404)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:44)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:69)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
	... 27 more

13:38:53,693 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 60) MSC00001: Failed to start service org.camunda.bpm.platform.process-application-module."delivery-manager.war".DEPLOY.delivery-manager: org.jboss.msc.service.StartException in service org.camunda.bpm.platform.process-application-module."delivery-manager.war".DEPLOY.delivery-manager: Could not register process application with shared process engine 
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService.performDeployment(ProcessApplicationDeploymentService.java:178)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$1.run(ProcessApplicationDeploymentService.java:87)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [rt.jar:1.6.0_31]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_31]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_31]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: org.camunda.bpm.engine.ProcessEngineException: Process engine persistence exception
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.rethrow(CommandInvocationContext.java:60)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:229)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:106)
	at org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:59)
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32)
	at org.camunda.bpm.engine.impl.RepositoryServiceImpl.deploy(RepositoryServiceImpl.java:82)
	at org.camunda.bpm.engine.impl.repository.DeploymentBuilderImpl.deploy(DeploymentBuilderImpl.java:125)
	at org.camunda.bpm.engine.impl.repository.ProcessApplicationDeploymentBuilderImpl.deploy(ProcessApplicationDeploymentBuilderImpl.java:51)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:167)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:164)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runWithTccl(Tccl.java:53)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runUnderClassloader(Tccl.java:45)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService.performDeployment(ProcessApplicationDeploymentService.java:164)
	... 8 more
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error updating database.  Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: SELECT * FROM ACT_GE_PROPERTY WHERE NAME_ = 'deployment.lock' for update
### Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:154)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:145)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.lock(DbSqlSession.java:110)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.lock(DbEntityManager.java:242)
	at org.camunda.bpm.engine.impl.persistence.entity.PropertyManager.acquireExclusiveLock(PropertyManager.java:31)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.acquireExclusiveLock(DeployCmd.java:112)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:75)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:61)
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:97)
	... 18 more
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
	at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1362)
	at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:404)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:44)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:69)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
	... 27 more

13:38:53,695 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 51) MSC00001: Failed to start service org.camunda.bpm.platform.process-application-module."network-1.0-SNAPSHOT.war".DEPLOY.network-meeting-process: org.jboss.msc.service.StartException in service org.camunda.bpm.platform.process-application-module."network-1.0-SNAPSHOT.war".DEPLOY.network-meeting-process: Could not register process application with shared process engine 
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService.performDeployment(ProcessApplicationDeploymentService.java:178)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$1.run(ProcessApplicationDeploymentService.java:87)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [rt.jar:1.6.0_31]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_31]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_31]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final.jar:2.1.0.Final]
Caused by: org.camunda.bpm.engine.ProcessEngineException: Process engine persistence exception
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.rethrow(CommandInvocationContext.java:60)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:229)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:106)
	at org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:59)
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32)
	at org.camunda.bpm.engine.impl.RepositoryServiceImpl.deploy(RepositoryServiceImpl.java:82)
	at org.camunda.bpm.engine.impl.repository.DeploymentBuilderImpl.deploy(DeploymentBuilderImpl.java:125)
	at org.camunda.bpm.engine.impl.repository.ProcessApplicationDeploymentBuilderImpl.deploy(ProcessApplicationDeploymentBuilderImpl.java:51)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:167)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:164)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runWithTccl(Tccl.java:53)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runUnderClassloader(Tccl.java:45)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService.performDeployment(ProcessApplicationDeploymentService.java:164)
	... 8 more
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error updating database.  Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: SELECT * FROM ACT_GE_PROPERTY WHERE NAME_ = 'deployment.lock' for update
### Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:154)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:145)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.lock(DbSqlSession.java:110)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.lock(DbEntityManager.java:242)
	at org.camunda.bpm.engine.impl.persistence.entity.PropertyManager.acquireExclusiveLock(PropertyManager.java:31)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.acquireExclusiveLock(DeployCmd.java:112)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:75)
	at org.camunda.bpm.engine.impl.cmd.DeployCmd.execute(DeployCmd.java:61)
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:97)
	... 18 more
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
	at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1362)
	at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:404)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:44)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:69)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
	... 27 more

This is the controller panel for Smart Panels app

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

server.log.2015-04-24
1.04 MB
27/Apr/15 2:27 PM

Daniel Meyer added a comment - 27/Apr/15 6:11 PM

Thanks for reporting

Daniel Meyer added a comment - 27/Apr/15 6:11 PM Thanks for reporting

Daniel Meyer added a comment - 17/Jun/15 1:49 PM

Experienced deadlock with parallel deployment on Wildfly today:

Found one Java-level deadlock:
=============================
"H2 Log Writer PROCESS-ENGINE":
  waiting to lock monitor 0x00007f9d6c227e78 (object 0x00000000f8355e30, a org.h2.engine.Database),
  which is held by "ServerService Thread Pool -- 55"
"ServerService Thread Pool -- 55":
  waiting to lock monitor 0x00007f9d6c227dc8 (object 0x00000000f83ba3f8, a org.h2.engine.Session),
  which is held by "ServerService Thread Pool -- 54"
"ServerService Thread Pool -- 54":
  waiting to lock monitor 0x00007f9d6c227e78 (object 0x00000000f8355e30, a org.h2.engine.Database),
  which is held by "ServerService Thread Pool -- 55"

Java stack information for the threads listed above:
===================================================

"H2 Log Writer PROCESS-ENGINE":
	at org.h2.engine.Database.flush(Database.java:1850)
	- waiting to lock <0x00000000f8355e30> (a org.h2.engine.Database)
	at org.h2.store.WriterThread.run(WriterThread.java:87)
	at java.lang.Thread.run(Thread.java:744)

"ServerService Thread Pool -- 55":
	at org.h2.store.LobStorageBackend$LobInputStream.<init>(LobStorageBackend.java:614)
	- waiting to lock <0x00000000f83ba3f8> (a org.h2.engine.Session)
	at org.h2.store.LobStorageBackend.getInputStream(LobStorageBackend.java:331)
	at org.h2.value.ValueLobDb.getInputStream(ValueLobDb.java:326)
	at org.h2.value.ValueLobDb.getBytesNoCopy(ValueLobDb.java:260)
	at org.h2.value.Value.convertTo(Value.java:733)
	at org.h2.value.ValueLobDb.convertTo(ValueLobDb.java:137)
	at org.h2.table.Column.convert(Column.java:145)
	at org.h2.command.dml.Insert.insertRows(Insert.java:115)
	at org.h2.command.dml.Insert.update(Insert.java:86)
	at org.h2.command.CommandContainer.update(CommandContainer.java:79)
	at org.h2.command.Command.executeUpdate(Command.java:235)
	- locked <0x00000000f8355e30> (a org.h2.engine.Database)
	at org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:193)
	- locked <0x00000000f83b89f8> (a org.h2.engine.Session)
	at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:404)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:44)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:69)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:141)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.executeInsertEntity(DbSqlSession.java:136)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.insertEntity(DbSqlSession.java:126)
	at org.camunda.bpm.engine.impl.db.AbstractPersistenceSession.executeDbOperation(AbstractPersistenceSession.java:35)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:276)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:265)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:263)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:192)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:106)
	at org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:59)
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32)
	at org.camunda.bpm.engine.impl.RepositoryServiceImpl.deploy(RepositoryServiceImpl.java:82)
	at org.camunda.bpm.engine.impl.repository.DeploymentBuilderImpl.deploy(DeploymentBuilderImpl.java:125)
	at org.camunda.bpm.engine.impl.repository.ProcessApplicationDeploymentBuilderImpl.deploy(ProcessApplicationDeploymentBuilderImpl.java:57)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:168)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:165)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runWithTccl(Tccl.java:53)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runUnderClassloader(Tccl.java:45)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService.performDeployment(ProcessApplicationDeploymentService.java:165)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$1.run(ProcessApplicationDeploymentService.java:88)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

"ServerService Thread Pool -- 54":
	at org.h2.store.LobStorageBackend.addLob(LobStorageBackend.java:369)
	- waiting to lock <0x00000000f8355e30> (a org.h2.engine.Database)
	- locked <0x00000000f83ba3f8> (a org.h2.engine.Session)
	at org.h2.store.LobStorageBackend.createBlob(LobStorageBackend.java:540)
	at org.h2.jdbc.JdbcConnection.createBlob(JdbcConnection.java:1744)
	at org.h2.jdbc.JdbcPreparedStatement.setBinaryStream(JdbcPreparedStatement.java:901)
	at org.h2.jdbc.JdbcPreparedStatement.setBinaryStream(JdbcPreparedStatement.java:923)
	at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.setBinaryStream(WrappedPreparedStatement.java:871)
	at org.apache.ibatis.type.BlobTypeHandler.setNonNullParameter(BlobTypeHandler.java:34)
	at org.apache.ibatis.type.BlobTypeHandler.setNonNullParameter(BlobTypeHandler.java:28)
	at org.apache.ibatis.type.BaseTypeHandler.setParameter(BaseTypeHandler.java:50)
	at org.apache.ibatis.scripting.defaults.DefaultParameterHandler.setParameters(DefaultParameterHandler.java:81)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.parameterize(PreparedStatementHandler.java:80)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.parameterize(RoutingStatementHandler.java:61)
	at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:74)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:47)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:141)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.executeInsertEntity(DbSqlSession.java:136)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.insertEntity(DbSqlSession.java:126)
	at org.camunda.bpm.engine.impl.db.AbstractPersistenceSession.executeDbOperation(AbstractPersistenceSession.java:35)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:276)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:265)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:263)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:192)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:106)
	at org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:59)
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32)
	at org.camunda.bpm.engine.impl.RepositoryServiceImpl.deploy(RepositoryServiceImpl.java:82)
	at org.camunda.bpm.engine.impl.repository.DeploymentBuilderImpl.deploy(DeploymentBuilderImpl.java:125)
	at org.camunda.bpm.engine.impl.repository.ProcessApplicationDeploymentBuilderImpl.deploy(ProcessApplicationDeploymentBuilderImpl.java:57)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:168)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:165)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runWithTccl(Tccl.java:53)
	at org.camunda.bpm.container.impl.jboss.util.Tccl.runUnderClassloader(Tccl.java:45)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService.performDeployment(ProcessApplicationDeploymentService.java:165)
	at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$1.run(ProcessApplicationDeploymentService.java:88)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Found 1 deadlock.

Daniel Meyer added a comment - 17/Jun/15 1:49 PM Experienced deadlock with parallel deployment on Wildfly today: Found one Java-level deadlock: ============================= "H2 Log Writer PROCESS-ENGINE" : waiting to lock monitor 0x00007f9d6c227e78 (object 0x00000000f8355e30, a org.h2.engine.Database), which is held by "ServerService Thread Pool -- 55" "ServerService Thread Pool -- 55" : waiting to lock monitor 0x00007f9d6c227dc8 (object 0x00000000f83ba3f8, a org.h2.engine.Session), which is held by "ServerService Thread Pool -- 54" "ServerService Thread Pool -- 54" : waiting to lock monitor 0x00007f9d6c227e78 (object 0x00000000f8355e30, a org.h2.engine.Database), which is held by "ServerService Thread Pool -- 55" Java stack information for the threads listed above: =================================================== "H2 Log Writer PROCESS-ENGINE" : at org.h2.engine.Database.flush(Database.java:1850) - waiting to lock <0x00000000f8355e30> (a org.h2.engine.Database) at org.h2.store.WriterThread.run(WriterThread.java:87) at java.lang. Thread .run( Thread .java:744) "ServerService Thread Pool -- 55" : at org.h2.store.LobStorageBackend$LobInputStream.<init>(LobStorageBackend.java:614) - waiting to lock <0x00000000f83ba3f8> (a org.h2.engine.Session) at org.h2.store.LobStorageBackend.getInputStream(LobStorageBackend.java:331) at org.h2.value.ValueLobDb.getInputStream(ValueLobDb.java:326) at org.h2.value.ValueLobDb.getBytesNoCopy(ValueLobDb.java:260) at org.h2.value.Value.convertTo(Value.java:733) at org.h2.value.ValueLobDb.convertTo(ValueLobDb.java:137) at org.h2.table.Column.convert(Column.java:145) at org.h2.command.dml.Insert.insertRows(Insert.java:115) at org.h2.command.dml.Insert.update(Insert.java:86) at org.h2.command.CommandContainer.update(CommandContainer.java:79) at org.h2.command.Command.executeUpdate(Command.java:235) - locked <0x00000000f8355e30> (a org.h2.engine.Database) at org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:193) - locked <0x00000000f83b89f8> (a org.h2.engine.Session) at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:404) at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:44) at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:69) at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:141) at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.executeInsertEntity(DbSqlSession.java:136) at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.insertEntity(DbSqlSession.java:126) at org.camunda.bpm.engine.impl.db.AbstractPersistenceSession.executeDbOperation(AbstractPersistenceSession.java:35) at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:276) at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:265) at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:263) at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:192) at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:106) at org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:59) at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32) at org.camunda.bpm.engine.impl.RepositoryServiceImpl.deploy(RepositoryServiceImpl.java:82) at org.camunda.bpm.engine.impl.repository.DeploymentBuilderImpl.deploy(DeploymentBuilderImpl.java:125) at org.camunda.bpm.engine.impl.repository.ProcessApplicationDeploymentBuilderImpl.deploy(ProcessApplicationDeploymentBuilderImpl.java:57) at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:168) at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:165) at org.camunda.bpm.container.impl.jboss.util.Tccl.runWithTccl(Tccl.java:53) at org.camunda.bpm.container.impl.jboss.util.Tccl.runUnderClassloader(Tccl.java:45) at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService.performDeployment(ProcessApplicationDeploymentService.java:165) at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$1.run(ProcessApplicationDeploymentService.java:88) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:744) at org.jboss.threads.JBossThread.run(JBossThread.java:122) "ServerService Thread Pool -- 54" : at org.h2.store.LobStorageBackend.addLob(LobStorageBackend.java:369) - waiting to lock <0x00000000f8355e30> (a org.h2.engine.Database) - locked <0x00000000f83ba3f8> (a org.h2.engine.Session) at org.h2.store.LobStorageBackend.createBlob(LobStorageBackend.java:540) at org.h2.jdbc.JdbcConnection.createBlob(JdbcConnection.java:1744) at org.h2.jdbc.JdbcPreparedStatement.setBinaryStream(JdbcPreparedStatement.java:901) at org.h2.jdbc.JdbcPreparedStatement.setBinaryStream(JdbcPreparedStatement.java:923) at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.setBinaryStream(WrappedPreparedStatement.java:871) at org.apache.ibatis.type.BlobTypeHandler.setNonNullParameter(BlobTypeHandler.java:34) at org.apache.ibatis.type.BlobTypeHandler.setNonNullParameter(BlobTypeHandler.java:28) at org.apache.ibatis.type.BaseTypeHandler.setParameter(BaseTypeHandler.java:50) at org.apache.ibatis.scripting.defaults.DefaultParameterHandler.setParameters(DefaultParameterHandler.java:81) at org.apache.ibatis.executor.statement.PreparedStatementHandler.parameterize(PreparedStatementHandler.java:80) at org.apache.ibatis.executor.statement.RoutingStatementHandler.parameterize(RoutingStatementHandler.java:61) at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:74) at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:47) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:141) at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.executeInsertEntity(DbSqlSession.java:136) at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.insertEntity(DbSqlSession.java:126) at org.camunda.bpm.engine.impl.db.AbstractPersistenceSession.executeDbOperation(AbstractPersistenceSession.java:35) at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:276) at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:265) at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:263) at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:192) at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:106) at org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:59) at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32) at org.camunda.bpm.engine.impl.RepositoryServiceImpl.deploy(RepositoryServiceImpl.java:82) at org.camunda.bpm.engine.impl.repository.DeploymentBuilderImpl.deploy(DeploymentBuilderImpl.java:125) at org.camunda.bpm.engine.impl.repository.ProcessApplicationDeploymentBuilderImpl.deploy(ProcessApplicationDeploymentBuilderImpl.java:57) at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:168) at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$3.run(ProcessApplicationDeploymentService.java:165) at org.camunda.bpm.container.impl.jboss.util.Tccl.runWithTccl(Tccl.java:53) at org.camunda.bpm.container.impl.jboss.util.Tccl.runUnderClassloader(Tccl.java:45) at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService.performDeployment(ProcessApplicationDeploymentService.java:165) at org.camunda.bpm.container.impl.jboss.service.ProcessApplicationDeploymentService$1.run(ProcessApplicationDeploymentService.java:88) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:744) at org.jboss.threads.JBossThread.run(JBossThread.java:122) Found 1 deadlock.

Daniel Meyer added a comment - 06/Jul/15 5:13 PM

Review zum 2.

Daniel Meyer added a comment - 06/Jul/15 5:13 PM Review zum 2.

Thorben Lindhauer added a comment - 08/Jul/15 10:52 AM

The code is ok. I accept that there is no test case to reproduce the issue reliably. For future development, we really have to watch out for deadlocks when using heavy locks like

synchronized (ProcessEngine.class)

From going through the current code base, I could not identify deadlock potential.

Thorben Lindhauer added a comment - 08/Jul/15 10:52 AM The code is ok. I accept that there is no test case to reproduce the issue reliably. For future development, we really have to watch out for deadlocks when using heavy locks like synchronized (ProcessEngine.class) From going through the current code base, I could not identify deadlock potential.

Michael Schoettes added a comment - 04/Aug/15 10:52 AM

Due to problems in high load scenarios we have to investigate again

Michael Schoettes added a comment - 04/Aug/15 10:52 AM Due to problems in high load scenarios we have to investigate again

Daniel Meyer added a comment - 18/Aug/15 10:15 AM

I just had a chance to reproduce some aspects of this together with jirauser10636 in our datacenter.
Environment: JBoss 7.2.0, Mysql 5.5

What we observed is that the deployment takes very long (many queries are performed to resume the latest versions etc...)
If the configuration property "deploymentLockUsed" is set to true, then a lock timeout occurs, not due to a deadlock but becase deployment takes more than a minute and after waiting on the lock for a minute, the waiting transactions are cancelled and rolled back.

With the lates patch (the fix to this issue), the above behavior cannot be observed since the delpoyment is already serialized using the java lock before the db locks are acquired. It would still be observable in a cluster.

Daniel Meyer added a comment - 18/Aug/15 10:15 AM I just had a chance to reproduce some aspects of this together with jirauser10636 in our datacenter. Environment: JBoss 7.2.0, Mysql 5.5 What we observed is that the deployment takes very long (many queries are performed to resume the latest versions etc...) If the configuration property "deploymentLockUsed" is set to true, then a lock timeout occurs, not due to a deadlock but becase deployment takes more than a minute and after waiting on the lock for a minute, the waiting transactions are cancelled and rolled back. With the lates patch (the fix to this issue), the above behavior cannot be observed since the delpoyment is already serialized using the java lock before the db locks are acquired. It would still be observable in a cluster.

Sreeni Kaparthi added a comment - 25/Feb/18 2:24 AM

We are using Camunda7.8, Wildfly10.1 with MariaDB10.1 database and we have been seeing “deployment.lock” errors in our wildfly server logs. This error happens during code install or during server bounce. Any advice on how to resolve these errors?

2018-02-24 04:34:20,797 ERROR
[org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - a

ddress: ([("deployment" =>
"bpmn-all-os-based-1802.48.1-83.war")]) - failure description: {

"WFLYCTL0080: Failed services"
=>

{"jboss.undertow.deployment.default-server.default-host./mso" => "org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./mso: java.lang.RuntimeException: org.camunda.bpm.engine.ProcessEngineException: ENGINE-08043 Exception while performing 'Deployment of Process Application MSO BPMN OS Based Application' => 'Deployment of process archive 'b pmn-all-os-based': Process engine persistence exception Caused by: java.lang.RuntimeException: org.camunda.bpm.engine.ProcessEngineException: ENGINE-08043 Exception while performing 'Deployment o f Process Application MSO BPMN OS Based Application' => 'Deployment of process archive 'bpmn-all-os-based': Process engine persistence exceptio n Caused by: org.camunda.bpm.engine.ProcessEngineException: ENGINE-08043 Exception while performing 'Deployment of Process Application MSO BP MN OS Based Application' => 'Deployment of process archive 'bpmn-all-os-based': Process engine persistence exception Caused by: org.camunda.bpm.engine.ProcessEngineException: Process engine persistence exception Caused by: org.apache.ibatis.exceptions.PersistenceException: ### Error querying database. Cause: java.sql.SQLNonTransientConnectionException: Connection timed out ### The error may exist in org/camunda/bpm/engine/impl/mapping/entity/Property.xml ### The error may involve defaultParameterMap ### The error occurred while setting parameters ### SQL: SELECT VALUE_ FROM ACT_GE_PROPERTY WHERE NAME_ = 'deployment.lock' for update ### Cause: java.sql.SQLNonTransientConnectionException: Connection timed out Caused by: java.sql.SQLNonTransientConnectionException: Connection timed out Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Connection timed out Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could not read resultset: Read timed out Query is: SELECT VALUE_ FROM ACT_GE_PROPERTY WHERE NAME_ = 'deployment.lock' for update Caused by: java.net.SocketTimeoutException: Read timed out"}

"WFLYCTL0412: Required services that
are not installed:" =>
["jboss.undertow.deployment.default-server.default-host./mso"],

"WFLYCTL0180: Services with
missing/unavailable dependencies" => undefined

}

Sreeni Kaparthi added a comment - 25/Feb/18 2:24 AM We are using Camunda7.8, Wildfly10.1 with MariaDB10.1 database and we have been seeing “deployment.lock” errors in our wildfly server logs. This error happens during code install or during server bounce. Any advice on how to resolve these errors? 2018-02-24 04:34:20,797 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - a ddress: ([("deployment" => "bpmn-all-os-based-1802.48.1-83.war")]) - failure description: { "WFLYCTL0080: Failed services" => {"jboss.undertow.deployment.default-server.default-host./mso" => "org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./mso: java.lang.RuntimeException: org.camunda.bpm.engine.ProcessEngineException: ENGINE-08043 Exception while performing 'Deployment of Process Application MSO BPMN OS Based Application' => 'Deployment of process archive 'b pmn-all-os-based': Process engine persistence exception Caused by: java.lang.RuntimeException: org.camunda.bpm.engine.ProcessEngineException: ENGINE-08043 Exception while performing 'Deployment o f Process Application MSO BPMN OS Based Application' => 'Deployment of process archive 'bpmn-all-os-based': Process engine persistence exceptio n Caused by: org.camunda.bpm.engine.ProcessEngineException: ENGINE-08043 Exception while performing 'Deployment of Process Application MSO BP MN OS Based Application' => 'Deployment of process archive 'bpmn-all-os-based': Process engine persistence exception Caused by: org.camunda.bpm.engine.ProcessEngineException: Process engine persistence exception Caused by: org.apache.ibatis.exceptions.PersistenceException: ### Error querying database. Cause: java.sql.SQLNonTransientConnectionException: Connection timed out ### The error may exist in org/camunda/bpm/engine/impl/mapping/entity/Property.xml ### The error may involve defaultParameterMap ### The error occurred while setting parameters ### SQL: SELECT VALUE_ FROM ACT_GE_PROPERTY WHERE NAME_ = 'deployment.lock' for update ### Cause: java.sql.SQLNonTransientConnectionException: Connection timed out Caused by: java.sql.SQLNonTransientConnectionException: Connection timed out Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Connection timed out Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could not read resultset: Read timed out Query is: SELECT VALUE_ FROM ACT_GE_PROPERTY WHERE NAME_ = 'deployment.lock' for update Caused by: java.net.SocketTimeoutException: Read timed out"} , "WFLYCTL0412: Required services that are not installed:" => ["jboss.undertow.deployment.default-server.default-host./mso"] , "WFLYCTL0180: Services with missing/unavailable dependencies" => undefined }

Assignee:: Thorben Lindhauer

Reporter:: user-dfc32 (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Created:: 27/Apr/15 2:27 PM

Updated:: 25/Feb/18 2:24 AM

Resolved:: 13/Aug/15 9:29 AM

camunda BPM

Details

Description

mgm-controller-panel

This is the controller panel for Smart Panels app

Attachments

Attachments

Activity

Collapse comment: Daniel Meyer added a comment - 27/Apr/15 6:11 PM

Expand comment: Daniel Meyer added a comment - 27/Apr/15 6:11 PM

Collapse comment: Daniel Meyer added a comment - 17/Jun/15 1:49 PM

Expand comment: Daniel Meyer added a comment - 17/Jun/15 1:49 PM

Collapse comment: Daniel Meyer added a comment - 06/Jul/15 5:13 PM

Expand comment: Daniel Meyer added a comment - 06/Jul/15 5:13 PM

Collapse comment: Thorben Lindhauer added a comment - 08/Jul/15 10:52 AM

Expand comment: Thorben Lindhauer added a comment - 08/Jul/15 10:52 AM

Collapse comment: Michael Schoettes added a comment - 04/Aug/15 10:52 AM

Expand comment: Michael Schoettes added a comment - 04/Aug/15 10:52 AM

Collapse comment: Daniel Meyer added a comment - 18/Aug/15 10:15 AM

Expand comment: Daniel Meyer added a comment - 18/Aug/15 10:15 AM

Collapse comment: Sreeni Kaparthi added a comment - 25/Feb/18 2:24 AM

Expand comment: Sreeni Kaparthi added a comment - 25/Feb/18 2:24 AM

People

Dates

Salesforce