Skip to content

fix(orders): record why an exec order failed, and honour informational exit codes - #9

Draft
benw5483 wants to merge 2 commits into
mainfrom
order-failure-diagnostics
Draft

fix(orders): record why an exec order failed, and honour informational exit codes#9
benw5483 wants to merge 2 commits into
mainfrom
order-failure-diagnostics

Conversation

@benw5483

@benw5483 benw5483 commented Aug 5, 2026

Copy link
Copy Markdown

Summary

  • An exec order that failed recorded that it failed and discarded why: order.failed carried the bare error (exit status 1) and an empty payload, and the run's tracking bead carried an outcome label and no text. Both now carry the resolved exec string, the exit status, and the last 2 KiB of the command's combined stdout/stderr, redacted through the same env redactor the dispatch logs use.
  • Exec orders can declare success_exit_codes = [1]. Plenty of useful scripts reserve a low non-zero code for a finding rather than an error (a drift detector that exits 1 having already reported the drift), and recording every such run as a failure makes an order with a 100% failure rate indistinguishable from a healthy one.
  • gc doctor's order-firing-current check read only order.fired, so an order firing exactly on schedule and failing every time reported green. It now reads order.completed and order.failed too, and reports last-succeeded beside last-fired.

What the record looks like now

bd show on a failed run's tracking bead:

exec: scripts/preflight.sh
exit status: 127
error: exit status 127
output (48 bytes):
scripts/preflight.sh: gc: unknown command "doctor"

gc doctor on an order that fires but doesn't succeed:

branch-protection: last fired 2h ago, expected every 24h
branch-protection: has never succeeded (last failed 2h ago)

Design notes for the reviewer

The doctor signal is advisory, not blocking. If it were blocking, a city that's been running a failing order for weeks would go red-gated the instant this check ships. Advisory still shows up in full in gc doctor output, and it won't wedge dispatch gates that read BlockingFailed. I left the existing fired-side severities alone.

An order with no recorded outcome stays silent. Its success state is unknown, not bad. So a city whose event history predates outcome recording won't light up red, and TestOrderFiringCurrent_NoOutcomeHistoryStaysSilent pins that.

A joined error refuses to yield an exit code. ExitCodeFromError returns (0, false) for an errors.Join, because a joined error carries a second failure (process-group cleanup, usually) alongside the exit status. Reducing it to the code alone would let a declared-informational code swallow that failure whole. Context cancellation isn't resolvable either, so a canceled run can't ever be read as a declared success.

Successful runs write nothing. A busy city carries tens of thousands of order-tracking beads, and a routine exit-0 run hasn't anything to diagnose, so that path skips the description write entirely.

Scope held deliberately. success_exit_codes lands on the order file only, not on [[orders.overrides]] in city.toml, which would pull the generated city schema and docs/reference/config.md into the diff. The order.failed payload stays NoPayload{} for the same reason: the detail rides in the event message rather than forcing a regenerated API client. Both look like reasonable follow-ups, but neither is needed to make a failed order diagnosable.

Test plan

  • go test ./internal/orders/... ./internal/doctor/... ./test/docsync/... is green.
  • go test ./cmd/gc/ gives 7827 pass, 37 fail. All 37 also fail on the base commit 89c9622 with an unmodified tree. They're a macOS /private/var vs /var symlink-resolution mismatch in the bd/dolt store-target tests, unrelated to this diff. One gotcha: the package needs -timeout 25m on a laptop. It runs about 23 minutes, and the default 10 looks like a hang.
  • Each new behaviour was verified by injecting the defect it guards against and confirming the intended test goes red:
    • success guard accepts any non-zero exit → TestOrderDispatchExecUndeclaredExitStillFails fails
    • success guard refuses every declared code → TestOrderDispatchExecDeclaredInformationalExitCompletes fails
    • dispatch stops persisting the detail → all three detail tests fail
    • doctor's success classification disabled → TestOrderFiringCurrent_FiringButNeverSucceeding and TestClassifyOrderSucceeding fail

Generated by the operator's software factory.
• City: factory-main · Agent: local-core.builder-1
• On behalf of: @benw5483

…ational exits failures

An exec order that failed recorded THAT it failed and discarded WHY. The
order.failed event carried the bare error ("exit status 1") and an empty
payload; the run's tracking bead carried an outcome label and nothing else.
Across one city's store, 264 beads carry exec-failed and not one contains an
error string, because they contain no text at all. Diagnosing a failed
scheduled order meant waiting to catch the next failure live.

Three changes, one theme: make the record of a run say what happened.

1. Capture the reason. dispatchExec now builds an ExecRunDetail (the resolved
   exec string, the exit status, and the last 2 KiB of the command's combined
   stdout and stderr), redacts it through the same env redactor the dispatch
   logs already use, and writes it to the tracking bead's description and to
   the order.failed message. The short reason stays on the message's first
   line, where readers and greps have always found it. A routine exit-0 run
   writes no description, since a busy city carries tens of thousands of
   tracking beads and a successful run hasn't anything to diagnose.

2. Let an order declare informational exit codes. Plenty of useful scripts
   reserve a low non-zero code for a finding rather than an error: a drift
   detector exits 1 when it found drift, having already reported it. gc logged
   every such run as a failure, which makes an order with a 100% failure rate
   indistinguishable from a healthy one, and that's precisely where a real
   breakage then hides. success_exit_codes = [1] records those runs as
   completed. Exec-only, entries between 1 and 255, listing 0 rejected as
   redundant, anything unlisted still fails. `gc order run` honours it too, so
   the manual and scheduled paths won't disagree about whether a command
   succeeded.

3. Teach doctor the difference between fired and succeeded. order-firing-current
   read only order.fired, so an order firing exactly on schedule and failing
   every time reported green. It now reads order.completed and order.failed as
   well, and reports last-succeeded beside last-fired. The new signal is
   advisory rather than blocking: a city whose orders have been failing quietly
   for weeks should be informed, not gated, the moment this ships. An order with
   no recorded outcome either way stays silent, because its success state is
   unknown rather than bad, so a city whose history predates outcome recording
   won't light up red.

Tests cover each part, and I verified each one by injection: an over-permissive
success guard, a guard that refuses every declared code, a dispatch that drops
the detail, and a doctor that goes back to reading fired-only. Each turns the
intended test red.

Generated by the operator's software factory.
City: `factory-main` · Agent: `local-core.builder-1`
On behalf of: @benw5483
Co-Authored-By: <operator-factory-bot> <factory-bot@operator-domain.invalid>
@benw5483
benw5483 marked this pull request as draft August 5, 2026 20:30
…ratchet

Two CI failures from the first commit, both mine.

The misspell linter reads "Honour" as a misspelling of "Honor" in a cmd_order.go
comment. Reworded.

The resource census ratchets subprocess call sites and my ExitCodeFromError test
added one, spawning `sh -c 'exit N'` to manufacture a real *exec.ExitError. The
invariant reads "untagged subprocess call/file totals cannot grow", so raising
the baseline is not the fix here. The unit test now covers the non-ExitError
contract only: nil, a context cancellation, a plain error, a wrapped error, and
a joined one. Genuine ExitError resolution stays covered where it is more honest
anyway, in the cmd/gc dispatch tests, which run real commands through
shellExecRunner and assert on exit 12 failing, exit 1 honored as declared
informational, and exit 11 still failing.

Generated by the operator's software factory.
City: `factory-main` · Agent: `local-core.builder-1`
On behalf of: @benw5483
Co-Authored-By: <operator-factory-bot> <factory-bot@operator-domain.invalid>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant