Hi unlimitechcloud.

Thank you for raising this feature request.

Can you expand upon your use case? In general, an external service task is handled by some custom logic in an external worker. Fetching & locking all external service tasks to be handled by a generic worker is not a common use case.

Furthermore, it is currently possible to query for all the available external service tasks by calling:

List<ExternalTask> allAvailableExternalTasks = processEngine.getExternalTaskService().createExternalTaskQuery().notLocked().list();

Are you aware of this API?

If no, would it work for you?

Best,
Nikola

Hi Nikola,

Indeed my use case may not be very common, I will try to explain. The usual deployment is to deploy a camunda engine with external tasks that are somehow coupled to the engine itself. This coupling is represented by the fact that the service task implementation needs to know where to query for external tasks, making the deployment as a "pull tasks". What I wanted to achieve was to make the deployment a "push tasks" style, so it is actually a separate process who pulls the tasks from the camunda engine and then published these tasks as events expecting "someone" resolves them. So the events are the pushed tasks, and the process don't care who will resolve it, and the workers who process these tasks don't need to know about camunda, so a full decoupling is achieved for the workers. Because of this, the process who pulls the tasks will not know what topics there exists, it just wants to aquire tasks for any topic so it can emit the event for the task to be processed.

The code you suggest may work for querying tasks, but the problem is that they don't become locked. I do need to lock the tasks while querying them because I need to ensure the event for the task is emitted once. This is why i suggest the queryAndLock "*" feature, it was the only way I could solve it.

Hi unlimitechcloud,

The fetch & lock mechanism was implemented so that workers can concurrently obtain external tasks without conflicts. If you're implementing a push mechanism, would you have a concurrency problem?

Since you already want to obtain all tasks at once, having multiple workers that lock all tasks would be inefficient. They would compete in locking all of the available tasks, and only one worker will win no matter what.

And if you have just a single worker, you can query for all the available tasks with the ExternalTaskService and then lock them without fear of concurrency conflicts.

Does this make sense, or am I missing somethign with your use case?

Best,
Nikola

Hi Nikola,

I understand what you say but I don't think this is the case. In my implementation I can have multiple workers pulling tasks so they can push them later, but at each run I fetch a limited number of task, lets say 10 or 20, so when scaling up in using the engine, all workers will be bussy fetching batchs of 10 or 20 tasks a at time. So they will compete but still each will get their batch of task, because fetchAndLock is already handling concurrency.

Hi unlimitechcloud,

That makes sense! I think that the feature that you're describing can be phrased as allowing wildcard characters/expressions when querying external service tasks by topic. This would basically add functionality similar to MQTT and Kafka provide, so users could query for *, or topicFirstLevel/*/topicThirdLevel and have more dynamic control over what tasks are locked.

Am I correct?

Best,
Nikola

Hi Nikola,

I agree, that description will fit my needs and will make the feature generic enough.

Thanks for the response unlimitechcloud. I'll adjust this ticket to better reflect the topics we discussed, and I will forward it for a decision.

Best,
Nikola

Hi unlimitechcloud,

thanks for bringing this up and providing more details along the way.

We will consider this ticket for the next prioritization rounds.
This does not yet mean we will work on this in the near future, it does however mean we see a certain value in it.
We will update here accordingly when we made a decision.

As a side note: several users have already taken on a Kafka integration for Camunda Platform, also specifically with regards to external tasks. You can see a discussion around that on our forum as well. Such an integration provides several benefits, for example not having to fetch and lock the tasks in order to re-publish them via Kafka. On the other hand, you would have to provide a Java-based plugin for the process engine. This might not be suitable in your scenario.

Best,
Tobias

We are closing this ticket as part of our backlog grooming. Reasons:

• We did not receive sufficient evidence that this ticket is important