Details
-
Bug Report
-
Resolution: Cannot Reproduce
-
L3 - Default
-
None
-
None
Description
Environment (Required on creation):
All enterprise version and community can reproduce this bug.
Tested over 50 times on Camunda Community V7.16.0/ V7.15.0
MySQL 8.0 version
Description (Required on creation; please attach any relevant screenshots, stacktraces, log files, etc. to the ticket):
I am calling external task API including (1)fetch and lock and (2)complete api
(1)https://docs.camunda.org/manual/7.16/reference/rest/external-task/fetch/
(2)https://docs.camunda.org/manual/7.16/reference/rest/external-task/post-complete/
I created a docker environment in Linus and created 8000 process instances and then fetch and lock them and complete them one by one. After all instance do all the tasks. It returned 500 server status on API and on camunda .
Here is my docker .yml file:
version: "3" services: camunda-db-master: container_name: camunda-db-master image: docker.io/mysql:8.0 command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci restart: always ports: - 13306:3306 volumes: - /usr/local/camunda/mysql-data/:/var/lib/mysql/ environment: - MYSQL_REPLICATION_MODE=master - MYSQL_REPLICATION_USER=repl_user - MYSQL_ROOT_PASSWORD=Camunda@support - MYSQL_USER=camunda - MYSQL_PASSWORD=camunda - MYSQL_DATABASE=camunda - TZ=Asia/HongKong networks: - CamundaNetwork camunda-platform: container_name: camunda-platform depends_on: - camunda-db-master image: docker.io/camunda/camunda-bpm-platform:7.15.0 ports: - 19090:8080 restart: always environment: - DB_DRIVER=com.mysql.cj.jdbc.Driver - DB_URL=jdbc:mysql://camunda-db-master:3306/camunda?sendFractionalSeconds=false - DB_USERNAME=root - DB_PASSWORD=Camunda@support - WAIT_FOR=camunda-db-master:3306 - WAIT_FOR_TIMEOUT=60 - TZ=Asia/HongKong networks: - CamundaNetwork
Here are the testing bpmn file:
I check the log and here are some example:
Sample log (1)
2021-11-19T10:08:07.019879271+08:00 stderr F 19-Nov-2021 02:08:06.828 SEVERE [http-nio-8080-exec-8] org.camunda.commons.logging.BaseLogger.logError ENGINE-16004 Exception while closing command context: An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace. 2021-11-19T10:08:07.019879271+08:00 stderr F org.camunda.bpm.engine.ProcessEngineException: An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace. 2021-11-19T10:08:07.019879271+08:00 stderr F at org.camunda.bpm.engine.impl.util.ExceptionUtil.wrapPersistenceException(ExceptionUtil.java:263)
Sample log (2)
2021-11-19T10:07:36.837902423+08:00 stderr F 19-Nov-2021 02:07:36.823 SEVERE [http-nio-8080-exec-4] org.camunda.commons.logging.BaseLogger.logError ENGINE-16004 Exception while closing command context: An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace. 2021-11-19T10:07:36.837902423+08:00 stderr F org.camunda.bpm.engine.ProcessEngineException: An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace. 2021-11-19T10:07:36.837902423+08:00 stderr F at org.camunda.bpm.engine.impl.util.ExceptionUtil.wrapPersistenceException(ExceptionUtil.java:263)
Sample log (3)
2021-11-19T10:07:36.837902423+08:00 stderr F 19-Nov-2021 02:07:36.823 SEVERE [http-nio-8080-exec-4] org.camunda.commons.logging.BaseLogger.logError ENGINE-16004 Exception while closing command context: An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace. 2021-11-19T10:07:36.837902423+08:00 stderr F org.camunda.bpm.engine.ProcessEngineException: An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace. 2021-11-19T10:07:36.837902423+08:00 stderr F at org.camunda.bpm.engine.impl.util.ExceptionUtil.wrapPersistenceException(ExceptionUtil.java:263)
Steps to reproduce (Required on creation):
I am using C# language calling Camunda API:
Create 8000 process instances
Write a program to auto fetch and lock and then complete the task.
Create 16000 process instances
Auto fetch and lock and then complete the tasks.
Create 30000 process instances
Auto fetch and lock and then complete the tasks.
Wait unto another day and it will die.
We have also create some environment for empty process instance and it will return error even we do not do any thing .
Observed Behavior (Required on creation):
It can successfully done all the task.
But status 500 after a day.
Expected behavior (Required on creation):
Should not have server error and can continue to work.
Root Cause (Required on prioritization):
Engine error that relate on database
Solution Ideas (Optional):
Which database is best to work with camunda?
We can change camunda version and the database.
Hints (optional):
More details on https://forum.camunda.org/t/camunda-stability-issue/31309/5
