fix(investigate): make a submitted finding account for its own verdict - #550
Merged
Conversation
Smana
force-pushed
the
fix/findings-account-for-verdict
branch
from
August 24, 2026 13:23
ac0a912 to
3e6a006
Compare
Two live cards shipped with the header promising something the body did not contain, and both payloads were legal: 2026-08-24 02:18 — "Action suggested", no next-steps section at all. suggested_action and actions are both optional, and the verdict badge is drawn from the verdict alone. 2026-08-22 22:53 — "High confidence · 85%" over a Why paragraph with no evidence bullets. evidence was optional too, so the badge was backed by the model's say-so and the verify pass had nothing to trace. Require evidence (minItems 1) on a root cause and say what it is for. Say that suggested_action is required when the verdict claims an action. Add actionWithoutRemedy and unevidencedConclusion next to unaccountedInconclusive — pure predicates, logged at the source, not errors: the analysis is still worth delivering and re-asking would spend a model call to re-answer a question the model just answered badly. The renderer stops hiding the first case: a verdict claiming an action with no remedy now says so on the card instead of dropping the section, so the reader can tell an absent remedy from a failed render. Also close the other half of the inconclusive rule. The schema said a recurrence you DID identify is not inconclusive; it never said the converse, and the 02:18 card reported 85% "high confidence" on a finding whose actual content was that it could not identify the failing resource. Confidence measures support for the cause you stated.
Review of the first revision found the notice rendering off the raw contract predicate, which is the one thing format.go says not to do: "the two must not be the same test". The contract predicate answers "did the MODEL contradict itself" and the loop logs it pre-verify; the renderer has to answer "is there anything the READER can use". The shipped golden fixtures proved it. seen_before quotes a human-reviewed resolution from the KB entry; matched_runbook carries a matched entry and its URL. Both hold an action verdict with no suggested_action, so both printed "the investigation did not supply a remedy" a few blocks below a remedy that was on screen — pointing the on-call away from the answer. - ActionWithoutRemedy and Verdict.ClaimsAction move onto the contract, next to UnaccountedInconclusive, for the reason stated there: two sides consult them and must not drift. - notify.remedyMissingForReader is the reader-side counterpart, and additionally requires that the card show no other remedy. - Format renders the notice too. It feeds Matrix, the webhook text and the CLI, and its doc comment says its claims must never diverge from the card; gating only the Slack block left those channels unfixed. - nextSteps trims, so a whitespace-only suggested_action can no longer suppress the notice while rendering an empty bullet. - unevidencedConclusion tests Verdict.Conclusive() rather than "!= inconclusive", so an unparseable verdict is a parse concern rather than a warning about a badge that is never drawn, and counts non-blank evidence, so minItems:1 cannot be satisfied with "". TestSubmitFindingsRequiresEvidence guards the model-facing half, which has no compiler; mutation-tested against dropping minItems and dropping evidence from required.
Cleanup pass on the same change.
ClaimsAction is defined off Verdict.Conclusive now, for the reason
ValidVerdict states directly above it: the three actionability verdicts
stay enumerated once, so a fifth is one switch to update rather than two
adjacent ones. The predicate's own doc claimed that property while
giving it up.
The Slack gate drops its `len(steps) == 0 &&` conjunct. It was implied —
ActionWithoutRemedy is false as soon as any root cause or action carries
a non-blank remedy, and nextSteps builds its list from exactly those
fields, trimmed the same way — so it was either dead or a divergence
from Format's gate, and a reader could not tell which.
The three contract warnings shared a five-attribute tail written three
times, except the first, which omitted "verdict" for no stated reason.
One closure now, and a fourth predicate is a line rather than a
paragraph.
Comment surgery, since the diff had begun restating one argument in
three files: ActionWithoutRemedy and remedyMissingForReader keep the
part that explains THEM (the live payload, and the fixtures that prove
the reader-side split is load-bearing) and point at
UnaccountedInconclusive for the placement and pre/post-verify reasoning
rather than re-deriving it.
The schema's root-cause properties go one per line, matching their
top-level siblings, so a two-word prompt tweak shows up in blame as one
changed line instead of the whole root-cause contract. The confidence
description also stops restating the verdict rule stated 3 lines below,
which was ~155 bytes re-sent on every turn.
Tests: the card test the table already subsumed is folded into it,
carrying across its one unique assertion, and the empty-bullet guard now
matches the bullets the renderer actually emits ("• ", not "1. ").
Smana
force-pushed
the
fix/findings-account-for-verdict
branch
from
August 24, 2026 16:30
4ca4e26 to
23c6b84
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two live cards shipped with the header promising something the body did not contain, and both payloads were legal against
submit_findings.suggested_actionandactionsare both optional; the verdict badge is drawn from the verdict aloneevidencewas optional tooIn the second case the badge was backed by the model's say-so, and the verify pass had nothing to trace.
The schema now asks for what the verdict implies
evidencebecomes required (minItems: 1) on a root cause, and says what it is for: name the tool and quote the value, error or log line it returned.suggested_actionis documented as required whenever the verdict isaction_suggested/action_required, unless a top-levelactionsentry already covers it.confidencesays it measures support for the cause you stated — not how sure you are of the narrative around it, and not how confident you are that you could not find something.Two predicates, logged — not errors
actionWithoutRemedyandunevidencedConclusionjoinunaccountedInconclusive: pure predicates, warned at the source. Deliberately not errors, for the same reason as the existing one — the analysis is still worth delivering, and re-asking costs a model call to re-answer a question the model just answered badly.The renderer stops hiding it
The first case is no longer silent. A verdict that claims an action with no remedy renders the section anyway, saying so:
The reader can now tell an absent remedy from a failed render.
no_actionandinconclusivemake no such promise and never get the disclaimer.The other half of the inconclusive rule
The schema said a recurrence you did identify is not inconclusive; it never said the converse. The 02:18 card reported 85% "high confidence" on a finding whose actual content was that it could not identify the failing resource. That direction is now stated: if you could not name the failing resource or the cause this time, the verdict is
inconclusivehowever sure you are of everything around it.Fixes F4 and F5.