Skip to content

feat(export): append flagged anomalies to a JSONL file#14

Merged
maxie7 merged 2 commits into
mainfrom
feat/m8-export-jsonl
Jul 24, 2026
Merged

feat(export): append flagged anomalies to a JSONL file#14
maxie7 merged 2 commits into
mainfrom
feat/m8-export-jsonl

Conversation

@maxie7

@maxie7 maxie7 commented Jul 24, 2026

Copy link
Copy Markdown
Owner

What

Anomalies previously existed only as pixels — a TUI card or a --plain line — with nothing another program could consume. --export <path> writes each flagged anomaly as one line of JSONL, so logscry can run in CI or a pipeline and have its output parsed. RDI §2 permits exactly this one form of persistence.

Off by default: with no --export, no file is opened, no goroutine starts, and every code path is the one that runs today. The writer is a nil *export.Writer whose methods are no-ops, so no call site branches on it.

Closes #5.

Schema

One JSON object per line, stable snake_case keys, every key always present:

{
  "kind": "anomaly",
  "template_hash": "a43727ff577d46c2",
  "pattern": "panic writing to db<NUM>.acme.internal (<IP>)",
  "level": "FATAL",
  "source": "stdin",
  "count_at_flag": 1,
  "first_seen": "2026-07-24T14:46:32.849+02:00",
  "last_seen_at_flag": "2026-07-24T14:46:32.849+02:00",
  "score": 1,
  "reasons": ["level FATAL"],
  "explanation": {
    "state": "explained",
    "summary": "", "likely_cause": "", "suggestion": "",
    "truncated": false, "error": "", "at": "2026-07-24T14:46:32.852+02:00"
  }
}
  • kind is "anomaly" (an LLM stage was attached) or "would_escalate" (--explain-dry-run), so a parser reading a mixed file can't confuse them: jq 'select(.kind=="anomaly")'.
  • explanation.state is "explained", "unavailable" (with error set), or "not_requested" (dry-run). "pending" can never appear — streamed partials never produce a line.
  • Fields are named for their semantics: count_at_flag / last_seen_at_flag are the values at the moment the anomaly was flagged, not final totals. A re-occurrence bumps the in-memory count and appends nothing.

Design notes

One record = one flagged event, captured at the flag instant. Every template-derived field is taken at escalation time; the explanation arriving seconds later explains that same event. The TUI path deliberately does not take the fresher numbers its live template state would allow — both renderers capture at the same instant, and a test asserts they emit an identical line for identical input.

A dedicated goroutine owns the file. No disk I/O on the pipeline goroutine (the same rule that keeps the LLM call off it), no locks, no shared mutable state. It's fed from the two places terminal explanations are already consumed for display (Pipeline.attach for the TUI, runPlain for --plain) plus the flag facts from Pipeline.Process, which runs in every mode.

The join is order-independent. The scorer hands the escalation to the pool from inside Evaluate, before the flag is emitted, so a fast backend can resolve first. The writer keeps a flag map and an orphan map and writes when the second half arrives, whichever it is.

Exactly one line per anomaly, guarded in three layers because this is the failure mode that would silently multiply the file under --llm-stream: Resolve ignores ExplainPending, the --plain tap sits after the existing pending-skip, and the writer re-checks.

Dry-run writes the would-be escalations it already prints on screen — a greppable, score-sortable artifact for threshold calibration. The write point keys off !p.explain ("no LLM stage attached, so this flag is terminal on arrival"), not the flag itself — which matters because a nil escalation channel also reports Queued == false for an unrelated reason, and recording that as a dropped escalation would have been a lie.

Real values, never placeholders. Nothing in the record passes through the anonymizer — the file is local, exactly like the terminal. The explanation carries the values llm.anonymizing.restore already put back. No raw trigger line is included, deliberately: pattern is the masked signature, safe by construction, so a file of one signature per line can go into a ticket or a CI artifact without an audit for leaked IPs and tokens.

Durability. Each record is marshalled to one complete buffer with its newline, then written through a loop over short writes — a write(2) to a regular file can return a short count, leaving partial bytes on disk, so "one Write per record" removes interleaving but not single-record atomicity. A failed write is rolled back with Truncate to the offset of the last complete record, so the file always ends on a record boundary; that record is lost and Close reports the error. If the rollback itself fails, the writer stops permanently rather than fusing a partial line onto the next record. SetEscapeHTML(false) keeps <IP> literal instead of \u003cIP\u003e, so the file stays greppable.

Verification

  • gofmt, go vet, golangci-lint (0 issues), go test ./..., go test -race ./..., go build ./... — all green.
  • Integrity tests via an injected sink: 7-bytes-per-call short writes still yield parseable lines; a write dying mid-record leaves no dangling partial line; an unrollable failure appends nothing further; rollback starts from pre-existing content so a first-write error can't truncate a healthy file to nothing.
  • The streaming landmine: flag + three ExplainPending + terminal → still exactly one line.
  • Order independence, default-off (no file created), reopen-appends, dry-run would_escalate records.
  • Manual: --plain --llm-stream --llm-anonymize with --export produced exactly one line carrying the real IP/hostname, while the recorded wire body carried <IP_1> — masked outbound, real on local disk, at the same time.

maxie7 and others added 2 commits July 24, 2026 14:14
--export <path> writes one JSON object per flagged anomaly, which is what turns
a run from something you watch into something a program can consume. RDI §2
allows exactly this much persistence and no more. Default off: with no --export
no file is opened, no goroutine starts, and a nil *export.Writer's methods are
no-ops, so no call site branches on it.

A record is one POINT-IN-TIME snapshot of one flagged event. Everything
template-derived is captured at the instant the anomaly crossed the threshold —
hence count_at_flag / last_seen_at_flag, named so nobody reads them as running
totals. The explanation lands seconds later and explains that same event, not
the state of the world when the model finished; welding the two instants
together would produce a line that looks like one snapshot and is not. Both
renderers capture at the same instant, so the file is a property of the run and
not of what was watching it.

Placement: the file is owned by a single goroutine reached over a buffered
channel with a non-blocking send. The loudest caller is the pipeline goroutine
that owns the template map, and it must never wait on a disk — the same rule
that keeps the LLM call off it (RDI §3). No locks; the only shared state is an
atomic counter for records dropped by a full channel, which is reported rather
than swallowed.

Records complete at two points, because dry-run has no explanations to wait for:
the terminal explanation (attach in the TUI, runPlain in --plain, the two places
one is already consumed for display), and the flag itself when no LLM stage is
attached at all. The latter is what makes --explain-dry-run's calibration output
diffable. Those records are marked kind "would_escalate" with state
"not_requested" so a parser can never mistake one for an explained anomaly.

Streaming was the landmine: since v0.5.0 an answer arrives as several pending
updates before its terminal one. Resolve drops pending outright, mirroring how
--plain already suppresses partials, so a streamed answer writes exactly one
line. The flag is deleted on write, so a duplicate answer adds nothing.

Line integrity is built, not assumed. One Write per record prevents records from
interleaving; it does NOT make one record atomic — write(2) may return short,
and the bytes it did write are already on disk. So each record is marshalled to
one complete buffer with its newline, written through a loop over short writes,
and rolled back with Truncate to the last record boundary if a write fails: the
record is lost, the file stays valid JSONL. If the rollback itself fails the
writer stops for good rather than fusing a partial line to the next record.
O_APPEND is kept for reopen-append semantics, not as an integrity argument.

Values are the real ones. --llm-anonymize masks at the LLM boundary and restores
the answer coming back; this file is local, exactly like the terminal that shows
the same text. There is deliberately no raw trigger line: the masked pattern is
safe by construction, which is what lets the whole file go into a ticket or a CI
artifact unexamined.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… entry

The schema is a contract, so it is written down rather than left to be inferred
from a sample line: a table of every key, what it means, and which of the three
explanation states it can hold.

Two things the README has to be honest about, because the shape of the file
invites the opposite assumption. count_at_flag and last_seen_at_flag are the
values at the instant the anomaly was flagged — a template that has fired two
hundred times since gets no second line and no updated total. And the file keeps
REAL values even under --llm-anonymize: that flag masks what goes to a remote
model, not what logscry writes locally, exactly as the terminal keeps them.

Also documents what --explain-dry-run puts in the file and why (the calibration
artifact that mode was missing), and the durability guarantee.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@maxie7 maxie7 self-assigned this Jul 24, 2026
@maxie7
maxie7 merged commit 5c69018 into main Jul 24, 2026
1 check passed
@maxie7
maxie7 deleted the feat/m8-export-jsonl branch July 24, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Export flagged anomalies to a JSONL file

1 participant