Spike feasibility of supporting ES8

In order to assess the feasibility of supporting ES8, we should conduct a timeboxed spike.

Suggested time: 2-4 hours

Intended outcome:

• Understand and document the differences between 7 and 8
• Understand if it is possible to support them both in parallel. Is there a client for both?
• Document the changes required to start supporting ES8, in terms of the new client, removing deprecated method usage etc.

Result of the spike:

• The RestHighLevelClient is deprecated since 7.15 (in favor of the Java API client), but it still works currently.
• The deprecated RestHighLevelClient offers the possibility to send requests in "compatibility mode" to ElasticSearch 8 (for details see this)
• I did a proof of concept with a locally running ES8 instance using the compatibility mode, and Optimize worked without any issues
• I ran all the automated tests against this setup (ES8 + compatibility mode), and nearly all succeeded. From our 9830 integration tests, only one fails. Additionally, some migration and upgrade tests also fail, but the overhead to fix them is manageable
• These settings are additionally necessary for ES8, so that it behaves like our configured ES7:

- name: xpack.security.enabled
  value: false
- name: action.destructive_requires_name
  value: false 

• Elasticsearch does not officially guarantee that everything works as intended with ES8 + compatibility mode
• To provide "real support" to ES8, we need to rewrite our code away from the RestHighLevelClient and use the Java API Client instead 
• Tip from Elastic search itself: 

   

  Transition strategies
There are different ways you can start transitioning away from the HLRC in your application code.
For example:

keep the existing code as-is and use the new Java API Client for new features in your application, then later migrate the existing code, rewrite the parts of your application where the new Java API Client is much easier to use than that of the HLRC, like everything related to search, rewrite those parts where you need to map application objects to/from JSON, by leveraging the tight integration of the new Java API Client with JSON object mappers

• Transitioning from the HLRC to the Java API client requires a considerable effort for refactoring our Optimize codebase
• Conclusion: It is possible to claim we support ES8 by using our existing code base and running the client on compatibility mode. Although not guaranteed from ElasticSearch that this will always work, I have not yet found a situation where this fails. A proof of concept has already been implemented and the effort for that is very small. If we do choose to go with this solution, we need to do some performance tests to assess the impact of the compatibility mode on performance. Long term we need to steer away from the HLRC and go to the Java Client (as described in OPT-6635), as it is deprecated. So sooner or later we will have to take on the big effort of refactoring our code