DeploymentResolver selects deployments with status Stopped

[Raphael-Kleindienst]

Describe the Problem/Bug

For our Orchestrationclient configuration we followed the documentation here to dynamically set the resource group for our OrchestrationClient.

This works, but if multiple orchestration deployments exist within the resource group then they are selected randomly or round-robin at run-time. In our case, one of the orchestration deployments had "status": "STOPPED", leading to requests failing at random and with a misleading log-message ("Response from AI Core: Request failed with status 404 (Not Found): The response Content-Type is not JSON")

Please consider changing the way deployments are selected, so only active/running deployments are selected.

Steps to Reproduce

1. Deploy multiple orchestration deployments to same resource group
2. Stop one of the deployments
3. Set up code as described

Expected Behavior

Only running deployments are selected at runtime

Screenshots

No response

Used Versions

    <dependency>
        <groupId>com.sap.ai.sdk</groupId>
        <artifactId>core</artifactId>
        <version>1.17.0</version>
    </dependency>

Code Examples

@Configuration
open class OrchestrationClientConfig(
    @Value("\${aicore.resource-group}") private val resourceGroup: String,
) {

    @Bean
    open fun orchestrationClient(): OrchestrationClient {
        val destination: HttpDestination  = AiCoreService().getInferenceDestination(resourceGroup).forScenario("orchestration")
        return OrchestrationClient(destination)
    }
}

Stack Trace

No response

Log File

Log file

...

Affected Development Phase

Production

Impact

Inconvenience

Timeline

No response

[rpanackal]

Thanks for pointing out the issue.

We do have some caching of deployments involved, which is currently unaware of the change of deployment status.

The original expectation was that the deployment will be maintained throughout runtime and there shouldn't be any behavioural difference the between choice of deployments. I can see that deployments stopped or deleted can be an issue.

I will try to reproduce the issue first and will get back to you soon.

Could you tell us a bit more about your use case? I am assuming you would have multiple deployments (atleast 2), when you stop one and then expect another to be picked up. Since deployments are effectively the same, why stop/delete in the first place?

[Raphael-Kleindienst]

Hi @rpanackal,

the issue isn't only limited to when the deployment changes during runtime - it also occurs when a stopped deployment already exists along a running deployment during application start-up.

We don't really have a usecase for this - our fix was to clean up the stopped deployments which is why this isn't a breaking issue. But as application teams perhaps don't always control the full production landscape it would be good if this deployment selection is as robust as possible.