Read Codex transcripts and flag re-asked requests - #2
Merged
Conversation
Three defects surfaced by harvesting 195 transcripts (155 Claude, 55 Codex) after twenty-three unharvested days. Codex was a silent hole. reduce_session read rec["message"]["role"]; Codex rollouts carry the turn under payload. A 1.4MB rollout produced a well-formed reduction saying "human turns: 0" and exited 0 — indistinguishable from a quiet session. Normalising Codex records inside records() fixes every reader at once, and --list now scans ~/.codex/sessions, where dozens of fork snapshots per session collapse by session id (1721 files, 55 sessions). An empty reduction now exits non-zero. Silent zero is worse than unsupported. The correction heuristics are lexical, so they cannot see the most expensive class: a user re-asking for something already reported done. One labelled case from this corpus — 42 PRs added to a project board, verified at 43 items via gh project item-list, while every default view filtered on an unset iteration so the board the user opens rendered empty. That session reduces to "likely corrections: 0". Turns that restate an earlier turn with more specificity or force are now flagged reask and marked ↩. Corpus rate 3.1% of turns, and 58% of them carry no correction keyword at all. find_repeats was mostly false positives: 61 pairs over the corpus, of which ~13 were one- and two-word turns whose word sets overlap by accident and ~14 were the same template prompt reused in two sessions. A six-content-word floor, a near-verbatim skip for long blocks, and treating a leading slash command as an invocation bring it to 39 pairs with template reuse down from 16 to 2. Not fixed, and worth saying: the board case above still does not fire. After stopwords it is three content words, below any threshold that keeps the false-positive rate sane. Catching it needs the assistant's own completion claim as an input, not more lexical overlap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189C5Lc8PeFkh88goiURRMN
The CI wheel and sdist smokes asserted `harness-extractor 1.0.0` as a literal, so every version bump broke CI until someone remembered to edit the workflow. That is a guard that needs a human to stay true, which is the failure mode this release already has enough of. Both smokes now read the installed distribution's version and check it against harness_extractor.__version__ and the CLI's own --version. Same three-way coherence the release workflow already enforces, minus the literal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189C5Lc8PeFkh88goiURRMN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Harvested 195 transcripts (155 Claude + 55 Codex) after twenty-three days in which the
weekly cron logged nothing but
Not logged in · Please run /login. Three defects fell outof the run itself.
Codex reduced to zero turns and exited 0
reduce_sessionreadrec["message"]["role"]; Codex rollouts carry the turn underpayload. On a 1.4MB rollout the tool printed a well-formed reduction reportinghuman turns: 0and returned success — indistinguishable from a quiet session.Normalising Codex records inside
records()fixes every reader at once: reduction, JSON,--only-corrections,--repeats.--listscans~/.codex/sessionsalongside~/.claude/projects(--codex-rootoverrides), and fork snapshots collapse by session id— Codex writes one file per fork, so 1721 files since 2026-08-11 are 55 real sessions.
An empty reduction now exits non-zero and says so on stderr. Silent zero is worse than
unsupported.
Re-ask detection
The correction heuristics are lexical, so the most expensive class is invisible: a user
re-asking for something the assistant already reported done.
The labelled case from this corpus — 42 PRs added to a project board, verified at 43 items
via
gh project item-listwith every field set, while all six default views filtered on aniterationthat was never set, so the board the user actually opens rendered empty. Theuser asked twice more, escalating. That session reduces to
likely corrections: 0.A turn that restates an earlier turn and adds specificity or force is now flagged
reaskand marked
↩. Containment, not Jaccard: escalation adds words rather than sharing them,so an overlap score reads the repeat as a new request. Corpus rate 3.1% of turns, 58% of
them carrying no correction keyword.
This does not catch the board case, and I would rather say so than tune until it does.
After stopwords that turn is three content words — below any threshold that keeps the
false-positive rate sane. Catching it needs the assistant's own completion claim as an
input, not more lexical overlap.
find_repeatswas mostly false positives61 pairs over the corpus. ~13 were one- and two-word turns (
try again,check again)whose word sets overlap by accident; ~14 were the same template prompt reused in two
sessions (
/mission read docs/launch/MISSION.md, the compaction preamble).A six-content-word floor, a near-verbatim skip for long blocks, and treating a leading
slash command as an invocation rather than prose: 61 pairs → 39, template reuse 16 → 2.
Verification
python3 -m unittest discover -s tests— 28 tests, all pass (8 new: Codex reduction,Codex shell/error recovery, Codex goal boilerplate, fork collapse, re-ask positive and
negative, non-zero empty exit, repeat floor).
uv buildanduvx twine check dist/*— both pass at 1.1.0.Version 1.1.0; CHANGELOG and README updated.
🤖 Generated with Claude Code
https://claude.ai/code/session_0189C5Lc8PeFkh88goiURRMN