Add ability to filter orchestrations at worker

[halspang]

This change allows a user to set a lambda that acts a filter for orchestrations. If the lambda returns false, the orchestration will be abandoned, leaving it to be attempted again later.

In this initial commit, the filter can work off of the orchestration name and the orchestration tags.

[cgillum]

@halspang can you add a bit more context on how this is intended to be used in real-world situations?

Normally a filtering feature would be built on the server-side so that you don't waste cycles delivering and abandoning messages repeatedly (or infinitely in some cases). Is there a plan to add such functionality to DTS? And would the design change at all if/when we go down that path?

[halspang]

@halspang can you add a bit more context on how this is intended to be used in real-world situations?

Normally a filtering feature would be built on the server-side so that you don't waste cycles delivering and abandoning messages repeatedly (or infinitely in some cases). Is there a plan to add such functionality to DTS? And would the design change at all if/when we go down that path?

@cgillum - The general goal of this is to give users the ability to react to events that require they temporarily pause or block work. It's being done this way as it allows for users to hook in their own custom logic for how that filtering is achieved. The main scenario that I was presented to make this change was the ability to add a dynamic kill switch to a specific worker.

I agree that it would be good to do this at a server level, and I've long wanted to add more worker knowledge to the backend. That's the more correct path, but also the much longer path. Especially so when it comes to integrating this kind of filtering as it would require persisting data between the worker and scheduler. I have ideas for this but, again, it'll just take a bit longer.

Let me know what you think in terms of, should we do this to get the functionality out there and then follow up with backend support, or if we should just aim for the server-side solution from the start.

[cgillum]

The main scenario that I was presented to make this change was the ability to add a dynamic kill switch to a specific worker.

I think the scenario is interesting, but I'm curious to understand how one would operationally implement a kill switch that only targets a specific worker. Can we clarify what "specific worker" means? For example, in an AKS context, is this a specific deployment, a specific pod, or perhaps a specific pod replica within the deployment?

[halspang]

The main scenario that I was presented to make this change was the ability to add a dynamic kill switch to a specific worker.

I think the scenario is interesting, but I'm curious to understand how one would operationally implement a kill switch that only targets a specific worker. Can we clarify what "specific worker" means? For example, in an AKS context, is this a specific deployment, a specific pod, or perhaps a specific pod replica within the deployment?

That all depends on how they implement the interface. Deployment is easiest as you could just be deploying a different image with specific logic about how to approve/deny orchestrations. You could also pass in the pod/replica ID as an env var and use that in your service. We left it very open so the user can make all the choices here.