-
Task
-
Resolution: Unresolved
-
L3 - Default
-
None
-
None
-
Not defined
Context:
In the past, we experienced the problem that a user wasn't able to perform the upgrade successfully as there wasn't enough disk space available. The error message also used to be very cryptic:
{"@timestamp":"2020-09-13T20:13:49.208+02:00","@version":"1","message":"Error while executing upgrade from 3.0.0 to 3.1.0","logger_name":"org.camunda.optimize.upgrade.main.UpgradeProcedure","thread_name":"main","level":"ERROR","level_value":40000,"stack_trace":"org.camunda.optimize.upgrade.exception.UpgradeRuntimeException: Error while trying to read reindex progress!\n\tat org.camunda.optimize.upgrade.es.ESIndexAdjuster.waitUntilReindexingTaskIsFinished(ESIndexAdjuster.java:460)\n\tat org.camunda.optimize.upgrade.es.ESIndexAdjuster.reindex(ESIndexAdjuster.java:145)\n\tat org.camunda.optimize.upgrade.steps.schema.UpdateIndexStep.execute(UpdateIndexStep.java:80)\n\tat org.camunda.optimize.upgrade.plan.UpgradeExecutionPlan.execute(UpgradeExecutionPlan.java:77)\n\tat org.camunda.optimize.upgrade.main.UpgradeProcedure.performUpgrade(UpgradeProcedure.java:43)\n\tat org.camunda.optimize.upgrade.main.UpgradeMain.main(UpgradeMain.java:62)\nCaused by: org.elasticsearch.client.ResponseException: method [GET], host http://localhost:8098, URI [/_tasks/pQE1nR1ySrmoqee4HNADnA:1700177], status line [HTTP/1.1 404 Not Found]\n{\"error\":{\"root_cause\":[
{\"type\":\"resource_not_found_exception\",\"reason\":\"task [pQE1nR1ySrmoqee4HNADnA:1700177] isn't running and hasn't stored its results\"}
],\"type\":\"resource_not_found_exception\",\"reason\":\"task [pQE1nR1ySrmoqee4HNADnA:1700177] isn't running and hasn't stored its results\"},\"status\":404}\n\tat org.elasticsearch.client.RestClient.convertResponse(RestClient.java:260)\n\tat org.elasticsearch.client.RestClient.performRequest(RestClient.java:238)\n\tat org.elasticsearch.client.RestClient.performRequest(RestClient.java:212)\n\tat org.camunda.optimize.upgrade.es.ESIndexAdjuster.waitUntilReindexingTaskIsFinished(ESIndexAdjuster.java:410)\n\t... 5 common frames omitted\n"}
AT:
- it's more prominent in the upgrade guide that the user needs to ensure that there's enough disk space
- the warning message on the start up of the upgrade script emphasizes the disk space issue
- Optimize fails the upgrade right from the start if Elasticsearch does not have enough disk space available
Hint:
To check the disk space of Elasticsearch one can use:
curl -XGET 'localhost:9200/_cat/allocation?v&pretty'
This ticket was migrated to github: https://github.com/camunda/camunda-optimize/issues/10482. Please use this link for any future references and continue any discussion there.