What/Where is the issue ?

This affects authentication on engine connections. When both authentication is enabled in the config and an engineRestPlugin exists, we would expect Optimize to first apply the basic auth filter and then run the plugin on every engine request.
On some runs however, the Basic Authentication Filter is applied after the plugin instead of before. 
This appears to be some sort of race condition during startup/initialisation of the client since it is consistent for each Optimize run: sometimes you start Optimize and its configured the right way around, sometimes you start Optimize and its configured the wrong way around. The order then appears to be consistent (either always basic auth first or always plugin first).
This becomes an issue when the user's plugin implementation depends on the presence of the expected basic auth Authentication header.

Environment: C7

Optimize version : 3.11 - but I think technically all versions with engine rest filter. Will backport to 3.10 as well
ES version : 8.7 but probably irrelevant
OS + Browser version : Windows + chrome but probably irrelevant

Steps to reproduce:

• Preconditions:
  • I used the 3.11 maintenance branch with some log additions but could also just use latest demo distribution
  • Configure basic auth in Optimzie config by enabling authentication on the default engine: 
    ```
    engines:
      # An alias of the engine, which will be used for internal purposes like
      # logging and displaying which data belong to which engine.
      'camunda-bpm':
        name: default
        defaultTenant:
          id: null
          name: null
        rest: 'http://localhost:8080/engine-rest'
        importEnabled: true
        eventImportEnabled: false
        authentication:
          enabled: true
          password: 'testpw'
          user: 'testuser'
    ```
  • Add the test plugin package:
    ```
    plugin:
      engineRestFilter:
        basePackages: ["org.example"]
    ```
  • Add the test plugin jar (attached) to the plugin folder
• Steps:
  • Start platform
  • Start Optimize
  • Wait until started, stop Optimize again
  • Observe logs
  • Repeat above steps if bug behaviour doesnt occur

Observed behavior:

On engine requests, Plugin is applied first and Basic auth filter is applied second. See example log of bug behaviour:

WARN  org.example.AddCustomTokenFilter - engine default NO BASIC AUTH applied on request http://localhost:8080/engine-rest/engine/default/user/demo/profile

Expected behavior:

On engine requsts, basic auth filter is applied first and plugin second. See example log of bug behaviour:

WARN  org.example.AddCustomTokenFilter - engine default has basic auth on request http://localhost:8080/engine-rest/engine/default/user/demo/profile

Additional Information:

Note that when I reproduced this I also added some additional logging in Optimize to confirm that the basic auth filter was indeed run and not just skipped (see attached logs). Not including that in the testing notes for easier testing.

We may need to update docs on execution order, though I'm unsure if there are proper use cases where execution order matters.

Hint: from jakarta ClientRequestFilter docs:

"Filters in the filter chain are ordered according to their jakarta.annotation.Priority class-level annotation value. "

the same however is true for the javax ClientRequestFilters we used before migrating spring: https://docs.oracle.com/javaee/7/api/javax/ws/rs/client/ClientRequestFilter.html#filter-javax.ws.rs.client.ClientRequestContext-