-
Bug Report
-
Resolution: None
-
L3 - Default
-
None
-
7.16.0
Environment (Required on creation):
Docker on Ubuntu
Description (Required on creation; please attach any relevant screenshots, stacktraces, log files, etc. to the ticket):
Camunda lost connection to MySQL after idle for 8 hours (default wait_timeout of MySQL). Once the connection is closed by MySQL, Camunda cannot restore it and eventually run into dead.
Steps to reproduce (Required on creation):
- docker-compose up -d;
- login to http://localhost:50100/camunda/app/cockpit/default/#/processes
- enter "Invoice Receipt"
- enter arbitrary ID
- you will find that everything is smooth (successful scenario)
- close current tab and wait for 6 minutes
- repeat from step 2 again and you will find error message prompted (An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace.)
- go to home and you will find 500 error
Observed Behavior (Required on creation):
Camunda prompt out error message on page and then throw 500 error on main page.
Expected behavior (Required on creation):
Camunda can still run without error.
Root Cause (Required on prioritization): MySQL closes the connection after idle for 8 hours and Camunda does not have mechanism to handle it.
Solution Ideas (Optional):
Hints (optional):
In order to make developer to test it more efficient, the --wait_timeout=300 command of MySQL service inside attached docker-compose.yml is set to 300 (5 minutes) (default is 28800 (8 hours)) so that you may let it idle for 5 minutes after startup and replay the error.
This is the controller panel for Smart Panels app
[CAM-14132] Camunda Platform is not compatible with MySQL (internal error after idle period)
Yes, I saw this in server log and have attached it in this comment - camunda-server-log.txt
We didn't use autoReconnect=true at first and have just tried with autoReconnect=true. Seems the problem is gone. However, another problem shows up. After the --wait_timeout period and we try to visit certain process instance page again, the same error shows up - camunda-server-log-with-autoreconnect.txt Fortunately, the error gone after we refresh the page once, and seems the application runs fine afterwards.
Although a refresh seems making the error gone, we doubt if it is normally to see this error and if there is any side effect or data loss/integrity after this error shows up?
Thanks
Hi mannok - In your test case you reduce the wait_timeout to a value shorter than the user session timeout. I think under normal circumstance this new error message would not occur because the user session would time out before the connection timeout is reached. So the user would be redirected to the login in screen and not get the error message resulting from a call to the server. Please check again with your "overnight timeout" scenario and I expect you to be redirected to login.
Anyway, while the connection is not successfully recreated the user won't be able to perform any operations. The error would be on UI layer only and wont have harmful effects.
The error message says: "(An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace.)".
Do you see an error similar to this in the server log?
“com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 54,607,614 milliseconds ago.The last packet sent successfully to the server was 54,607,614 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.”
How have you configured the DB url? Try:
mysql://db_user:db_user@localhost/mydb?autoReconnect=true
See: https://dev.mysql.com/doc/c-api/8.0/en/c-api-auto-reconnect.html