Report signal_reason when failing jobs via the Stacks API

[walkerdb]

Problem

When the controller fails a Buildkite job for infrastructure reasons — pod pending timeout, init container failure, podSpec rejection, image-pull failure — the job lands in Buildkite as a bare failed / exit_status: -1 with no signal_reason.

Every failure site already constructs FailureInfo{Reason: agent.SignalReasonStackError}, but failForK8sObject only forwards failureInfo.Message, and stacksapi.FinishJobRequest has no field to carry a reason — so FailureInfo.Reason is currently dead code.

This appears to be a regression from the temporary-agent era (a leftover comment in fail_job.go still referenced connecting "the temporary agent"): an agent finishing a job could report signal_reason: stack_error, which is why the retry docs document matching on it and the v0.29.0 release notes advertise it for stack-level visibility. The Stacks API finish path lost it.

Impact

• retry: automatic: [{signal_reason: stack_error}] — the documented way to auto-retry infra failures — can't match jobs failed by this controller.
• Observability: stack-failed jobs are indistinguishable from ordinary agent-lost jobs (exit -1) in the REST API, webhooks, EventBridge, and OpenTelemetry notification payloads, except by parsing free-text detail. (We found this while trying to build an infra-failure metric for our CI: jobs killed by pod-scheduling failures had no machine-readable marker at all.)

Change

• Plumb FailureInfo.Reason through failForK8sObject → AgentClient.FailJob → FinishJobRequest.SignalReason.
• Set SignalReasonStackError on the image-pull-failure path, which previously sent no reason (it carried a // Do we have a better status code to report here? comment).
• Remove the stale temporary-agent comment.

Dependencies

Depends on buildkite/stacksapi#7 adding SignalReason to FinishJobRequest — and on the /stacks/.../finish endpoint accepting the parameter server-side. If the server already supports it, only these client changes are needed; if not, please treat the pair of PRs as the feature request. The go.mod bump is intentionally omitted until stacksapi tags a release containing #7.

Testing

go build ./... and go test ./internal/controller/scheduler/... pass with a local replace pointing at the stacksapi branch.

🤖 Generated with Claude Code

[isaacsu]

Thanks for submitting this @walkerdb. I'll be reviewing and hope to respond in a couple of days.