fix(orders): record why an exec order failed, and honour informational exit codes - #9
Draft
benw5483 wants to merge 2 commits into
Draft
fix(orders): record why an exec order failed, and honour informational exit codes#9benw5483 wants to merge 2 commits into
benw5483 wants to merge 2 commits into
Conversation
…ational exits failures
An exec order that failed recorded THAT it failed and discarded WHY. The
order.failed event carried the bare error ("exit status 1") and an empty
payload; the run's tracking bead carried an outcome label and nothing else.
Across one city's store, 264 beads carry exec-failed and not one contains an
error string, because they contain no text at all. Diagnosing a failed
scheduled order meant waiting to catch the next failure live.
Three changes, one theme: make the record of a run say what happened.
1. Capture the reason. dispatchExec now builds an ExecRunDetail (the resolved
exec string, the exit status, and the last 2 KiB of the command's combined
stdout and stderr), redacts it through the same env redactor the dispatch
logs already use, and writes it to the tracking bead's description and to
the order.failed message. The short reason stays on the message's first
line, where readers and greps have always found it. A routine exit-0 run
writes no description, since a busy city carries tens of thousands of
tracking beads and a successful run hasn't anything to diagnose.
2. Let an order declare informational exit codes. Plenty of useful scripts
reserve a low non-zero code for a finding rather than an error: a drift
detector exits 1 when it found drift, having already reported it. gc logged
every such run as a failure, which makes an order with a 100% failure rate
indistinguishable from a healthy one, and that's precisely where a real
breakage then hides. success_exit_codes = [1] records those runs as
completed. Exec-only, entries between 1 and 255, listing 0 rejected as
redundant, anything unlisted still fails. `gc order run` honours it too, so
the manual and scheduled paths won't disagree about whether a command
succeeded.
3. Teach doctor the difference between fired and succeeded. order-firing-current
read only order.fired, so an order firing exactly on schedule and failing
every time reported green. It now reads order.completed and order.failed as
well, and reports last-succeeded beside last-fired. The new signal is
advisory rather than blocking: a city whose orders have been failing quietly
for weeks should be informed, not gated, the moment this ships. An order with
no recorded outcome either way stays silent, because its success state is
unknown rather than bad, so a city whose history predates outcome recording
won't light up red.
Tests cover each part, and I verified each one by injection: an over-permissive
success guard, a guard that refuses every declared code, a dispatch that drops
the detail, and a doctor that goes back to reading fired-only. Each turns the
intended test red.
Generated by the operator's software factory.
City: `factory-main` · Agent: `local-core.builder-1`
On behalf of: @benw5483
Co-Authored-By: <operator-factory-bot> <factory-bot@operator-domain.invalid>
benw5483
marked this pull request as draft
August 5, 2026 20:30
…ratchet Two CI failures from the first commit, both mine. The misspell linter reads "Honour" as a misspelling of "Honor" in a cmd_order.go comment. Reworded. The resource census ratchets subprocess call sites and my ExitCodeFromError test added one, spawning `sh -c 'exit N'` to manufacture a real *exec.ExitError. The invariant reads "untagged subprocess call/file totals cannot grow", so raising the baseline is not the fix here. The unit test now covers the non-ExitError contract only: nil, a context cancellation, a plain error, a wrapped error, and a joined one. Genuine ExitError resolution stays covered where it is more honest anyway, in the cmd/gc dispatch tests, which run real commands through shellExecRunner and assert on exit 12 failing, exit 1 honored as declared informational, and exit 11 still failing. Generated by the operator's software factory. City: `factory-main` · Agent: `local-core.builder-1` On behalf of: @benw5483 Co-Authored-By: <operator-factory-bot> <factory-bot@operator-domain.invalid>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
order.failedcarried the bare error (exit status 1) and an empty payload, and the run's tracking bead carried an outcome label and no text. Both now carry the resolved exec string, the exit status, and the last 2 KiB of the command's combined stdout/stderr, redacted through the same env redactor the dispatch logs use.success_exit_codes = [1]. Plenty of useful scripts reserve a low non-zero code for a finding rather than an error (a drift detector that exits 1 having already reported the drift), and recording every such run as a failure makes an order with a 100% failure rate indistinguishable from a healthy one.gc doctor'sorder-firing-currentcheck read onlyorder.fired, so an order firing exactly on schedule and failing every time reported green. It now readsorder.completedandorder.failedtoo, and reports last-succeeded beside last-fired.What the record looks like now
bd showon a failed run's tracking bead:gc doctoron an order that fires but doesn't succeed:Design notes for the reviewer
The doctor signal is advisory, not blocking. If it were blocking, a city that's been running a failing order for weeks would go red-gated the instant this check ships. Advisory still shows up in full in
gc doctoroutput, and it won't wedge dispatch gates that readBlockingFailed. I left the existing fired-side severities alone.An order with no recorded outcome stays silent. Its success state is unknown, not bad. So a city whose event history predates outcome recording won't light up red, and
TestOrderFiringCurrent_NoOutcomeHistoryStaysSilentpins that.A joined error refuses to yield an exit code.
ExitCodeFromErrorreturns(0, false)for anerrors.Join, because a joined error carries a second failure (process-group cleanup, usually) alongside the exit status. Reducing it to the code alone would let a declared-informational code swallow that failure whole. Context cancellation isn't resolvable either, so a canceled run can't ever be read as a declared success.Successful runs write nothing. A busy city carries tens of thousands of order-tracking beads, and a routine exit-0 run hasn't anything to diagnose, so that path skips the description write entirely.
Scope held deliberately.
success_exit_codeslands on the order file only, not on[[orders.overrides]]incity.toml, which would pull the generated city schema anddocs/reference/config.mdinto the diff. Theorder.failedpayload staysNoPayload{}for the same reason: the detail rides in the event message rather than forcing a regenerated API client. Both look like reasonable follow-ups, but neither is needed to make a failed order diagnosable.Test plan
go test ./internal/orders/... ./internal/doctor/... ./test/docsync/...is green.go test ./cmd/gc/gives 7827 pass, 37 fail. All 37 also fail on the base commit89c9622with an unmodified tree. They're a macOS/private/varvs/varsymlink-resolution mismatch in the bd/dolt store-target tests, unrelated to this diff. One gotcha: the package needs-timeout 25mon a laptop. It runs about 23 minutes, and the default 10 looks like a hang.TestOrderDispatchExecUndeclaredExitStillFailsfailsTestOrderDispatchExecDeclaredInformationalExitCompletesfailsTestOrderFiringCurrent_FiringButNeverSucceedingandTestClassifyOrderSucceedingfail