fix(api): stop the reaper claiming "no response from agent" when the agent replied (#3097) - #3172
Open
bdunncompany wants to merge 1 commit into
Open
Conversation
…agent replied (LanternOps#3097) Script results submitted over the HTTP path never reach `script_executions` — only `agentWs` registers `script: handleScriptResult` — so the row stays pending, ages past the cutoff, and lands in `reapStaleScriptExecutions`, which stamped it `timeout` with "Server-side timeout: no response from agent". That claim is false whenever a terminal `device_commands` row exists: the agent did respond, the result simply was never mirrored onto the execution. On one live instance 89 executions read `timeout` while their command had completed successfully with output captured in `device_commands.result`. The reaper now reads the related command BEFORE deciding. When that row is terminal it records the outcome the command actually reached — mapped the same way `handleScriptResult` maps it, so a reaped row agrees with what the WS path would have written — with an error message saying the result was delivered but never recorded. When the command is non-terminal, or there is no command row at all, the original wording stands, because there the claim is true. This does NOT persist stdout/stderr. Mirroring the result belongs with the shared handler-registry work; this only stops the reaper asserting something it cannot know, which is where the false `timeout` labels come from. The command lookup already existed a few lines further down, used only to find `batchId`. It moved above the update and now serves both purposes, so this adds no extra round-trip. Two consequences worth flagging in review: - The batch counter followed the same false assumption: it incremented `devicesFailed` unconditionally. A recovered success now increments `devicesCompleted` instead. Slightly beyond a literal reading of "add a guard", but leaving it would have kept the same wrong claim one level up. - `reapStaleScriptExecutions` is exported so it can be tested directly. Four tests: command completed (recovers as completed, message no longer claims silence), command failed (records failed, not timeout), command non-terminal (genuine silence still reported as timeout), and no command row (same). The last two matter most — a guard that swallowed real agent silence would be worse than the bug. Confirmed the two recovery cases FAIL against the previous behaviour. Local gate, green on node 22.23.2: `tsc --noEmit` clean; `vitest run` in apps/api 1271 files / 20146 tests passed, 0 failed; eslint clean on both changed files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 1 of the sequencing you set on #3097 — the reaper guard, on its own, ahead of the extraction. Leaves #3097 open for the extraction PR.
What it fixes
Results submitted over the HTTP path never reach
script_executions, so the row stays pending, ages past the cutoff, and lands inreapStaleScriptExecutions— which stamped ittimeoutwith "Server-side timeout: no response from agent".That's false whenever a terminal
device_commandsrow exists. The agent did respond; the result was simply never mirrored. 89 executions on one live instance readtimeoutwhile their command had completed successfully with output.The reaper now reads the related command before deciding, and when that row is terminal it records the outcome the command actually reached — mapped the same way
handleScriptResultmaps it, so a reaped row agrees with what the WS path would have written. Non-terminal command, or no command row at all: original wording stands, because there it's true.It does not persist stdout/stderr. That's the extraction's job. This only stops the reaper asserting something it cannot know.
The command lookup already existed a few lines below, used only for
batchId. It moved above the update and serves both purposes now, so no extra round-trip.Two things to look at in review
The batch counter carried the same false assumption — it incremented
devicesFailedunconditionally, so a recovered success was still counted as a batch failure. It now incrementsdevicesCompletedwhen the command completed. That's slightly beyond a literal "add a guard", and I'd rather you saw it flagged than buried: leaving it would have kept the same wrong claim one level up.reapStaleScriptExecutionsis now exported so it can be tested directly.Tests
Four cases, and the two I care most about are the negative ones — a guard that swallowed real agent silence would be worse than the bug it fixes:
completed, message no longer claims silencefailed, nottimeoutsent) → stilltimeout, still "no response from agent"Confirmed the two recovery cases fail against the previous behaviour before the guard went in.
Local gate
Node 22.23.2, all green:
tsc --noEmitclean;vitest runinapps/api1271 files / 20146 tests passed, 0 failed; eslint clean on both changed files.Extraction next, once this lands —
services/home for the handler block and registry, canonical byte-lengthcommandResultSchemaexported fromschemas.tsand imported by both transports, the two dynamic imports left as-is, and a body that can honestly say the handlers are byte-identical with the schema unification as the one intentional behaviour change.