[API] staleCommandReaper uses a fixed 300s deadline instead of the script's own timeoutSeconds

[ToddHebebrand]

Description

Found while fixing #3162 (PR #3186), left out of scope there.

staleCommandReaper uses a fixed 300s deadline for every command, regardless of the script's own timeoutSeconds. A legitimately long-running script is therefore reaped and reported as stale while it is still executing correctly, and a script with a short timeout is left pending far longer than its own contract allows.

Proposed Fix

Have the reaper honour the originating script's timeoutSeconds (with a sane floor/ceiling and a margin for agent round-trip) instead of the fixed 300s.

Notes

Related family: #3097 (reaper claiming "no response from agent" when the agent replied). Worth checking whether the two want a single change to the reaper's deadline logic.

[bdunncompany]

Answering the "worth checking whether the two want a single change" note directly: they want separate changes, and the reason is that they act on different axes of the same function.

They do both live in reapStaleScriptExecutions, so the diffs will touch adjacent lines — but the logic doesn't overlap:

• This issue ([API] staleCommandReaper uses a fixed 300s deadline instead of the script's own timeoutSeconds #3190) changes which rows are selected as stale: defaultTimeoutMs (line ~291) feeds both the lt(scriptExecutions.createdAt, conservativeCutoff) predicate and the per-row if (now - referenceTime < defaultTimeoutMs) continue re-check.
• [API] Script results submitted over HTTP never update script_executions — 37% of executions mislabeled 'timeout', including 89 that completed with output #3097 / fix(api): stop the reaper claiming "no response from agent" when the agent replied (#3097) #3172 changes what the reaper concludes about a row it has already selected: whether stamping timeout + "no response from agent" is honest, given a terminal device_commands row for the same execution.

The important part is that #3190 cannot subsume #3097, so please don't let a per-script deadline close it. Widening the cutoff only helps an execution that is genuinely still running. In the #3097 case the agent already answered — the result reached device_commands and never reached script_executions — so the row is legitimately old and pending no matter how the deadline is computed. It will still be selected, and still mislabelled, under any cutoff.

If anything the dependency runs the other way: a tighter per-script deadline (a 30s script reaped at ~35s instead of 600s) selects these rows sooner and in greater number, so the terminal-command guard gets more load, not less.

Sequencing suggestion, purely to avoid a textual conflict: #3172 is already open against this function and green apart from the current Actions incident. If it lands first, #3190 becomes a contained change to the two defaultTimeoutMs uses with nothing else moving. The reverse order means rebasing the guard onto a rewritten cutoff.

One thing worth deciding explicitly when #3190 is built: the per-row re-check at line ~320 currently reuses the same constant as the query cutoff. If the deadline becomes per-script, that re-check needs the script's value too, or it silently keeps enforcing the old floor and the fix is inert for exactly the short-timeout case this issue names.