feat: bound how long a routed request may take

[gburachas]

Summary

A routed request has no upper bound on how long it can take. There are two places a bound is needed, and this issue tracks both. Supersedes #346.

1. A total timeout on the upstream request

crates/switchyard-server/src/config.rs has no per-client timeout setting. An upstream that accepts a request and then responds slowly holds the connection for as long as it likes.

Relationship to #271. That PR adds a connection timeout and an idle-read timeout to TranslatingLlmClient. Both are needed and neither bounds total duration: an idle-read timeout resets on every successful read, so an upstream that emits one token every 100 seconds never trips it while keeping the request open indefinitely.

The proposal here is a total per-request bound, configurable per client:

[llm_clients.weak]
timeout_secs = 600

If #271 lands first this should be added to its HttpTransportConfig rather than as a separate mechanism. I have commented on #271 to that effect.

One consequence to note: since #316 moved token counting to an inherent method that still goes through send_encoded, a per-client timeout would also bound count_tokens.

2. A deadline on the judge consultation

For llm_classifier routes, the judge call runs before the routed call. JudgeClassifier::verdict in crates/libsy/src/algorithms/util/llm_judge.rs has no time limit, so a slow judge delays a request that the user is waiting on for content the judge does not produce.

Proposal: judge_deadline_ms on the route. On expiry the judge is treated as unavailable and the route falls back through the existing path used for any other judge failure, so no new behaviour is introduced. Unset by default.

Two details:

• The bound must cover the whole consultation, not only the HTTP call. A judge that returns headers promptly and then stops mid-stream would otherwise hold the turn for the same length of time.
• Expiry should be reported through the existing fail-open reporting (report_fail_open in the same file), so it appears in the switchyard.classifier_fail_open metric alongside the other reasons rather than as an untracked path.

judge_deadline_ms = 0 should be rejected at load time rather than treated as "no deadline".

Sequencing

These are separate changes and will be submitted as separate pull requests. The judge deadline has no external dependency. The client timeout waits on #271.

[gburachas]

Refreshing this against what has landed since, so the scope is unambiguous.

This is not covered by #271. That PR adds HttpTransportConfig { connect_timeout, read_timeout } at the reqwest-builder level: a connect timeout, and an idle-read timeout that resets after every successful read. Both are necessary and neither bounds a request's total duration — an upstream that streams a token every 100 s keeps a turn open indefinitely without ever tripping an idle-read timeout.

What this issue asks for is the total per-request bound, exposed in the deployment TOML so an operator can set it per client:

[llm_clients.weak]
timeout_secs = 600

The two compose rather than compete: connect/read timeouts catch a dead or stalled socket, a total timeout caps the turn. If #271 lands first I will wire this to its HttpTransportConfig rather than adding a parallel mechanism.

One consequence worth stating up front: since #316 moved token counting to an inherent method that still routes through send_encoded, a per-client timeout also bounds count_tokens. That is almost certainly desirable, but it is new behaviour rather than an accident.

I have this rebased onto current main and passing the full gate set; happy to open the PR whenever it is useful, or to hold until #271 resolves.