Steps to reproduce:

• Configure queryMaxResultsLimit in the process engine configuration
• Navigate to Batch operation page in Cockpit
• Scenarios:
  • Try to perform Delete running process instances by filtering Unfinished Only and do not select a specific process instance but rather apply the batch for all listed instances
  • Try to perform Set retries of Jobs belonging to the process instances by filtering Unfinished Only and do not select a specific process instance but rather apply the batch for all listed instances
    (The important part is to not select instances. Apply the batch operation for all listed process instances.)

Observed:
Scenario 1:

Request:
POST http://portainer.camunda.loc:30194/camunda/api/engine/engine/default/process-instance/delete-historic-query-based
{"historicProcessInstanceQuery":{"unfinished":true},"skipCustomListeners":true}
Response:
400 Bad request
{"type":"BadUserRequestException","message":"An unbound number of results is forbidden!"}

Scenario 2:

Request:
POST http://portainer.camunda.loc:30194/camunda/api/engine/engine/default/process-instance/job-retries-historic-query-based
{"historicProcessInstanceQuery":{"unfinished":true},"retries":"2"}
Response:
400 Bad request
{"type":"BadUserRequestException","message":"An unbound number of results is forbidden!"}

Expected:
The operation completes successfully.

Hint:
In the Rest API a separate query is performed to fetch the ids fulfilling the historicProcessInstanceQuery. That query does not have a limit applied.
Scenario 1: https://github.com/camunda/camunda-bpm-platform/blob/6d61e9fb6ba961f4231f9aff84baa82d48f6922c/engine-rest/engine-rest/src/main/java/org/camunda/bpm/engine/rest/impl/ProcessInstanceRestServiceImpl.java#L170-L171
Scenario 2: https://github.com/camunda/camunda-bpm-platform/blob/6d61e9fb6ba961f4231f9aff84baa82d48f6922c/engine-rest/engine-rest/src/main/java/org/camunda/bpm/engine/rest/impl/ProcessInstanceRestServiceImpl.java#L228-L229

Workarounds:

• For those two batch operations select explicitly the process instances for which the batch operation should be performed. That why the historicProcessInstanceQuery is skipped in the request.
• When it's not possible to select all of the desired process instances in case they are too many, please use the Rest API endpoints to perform the batch operation.