Skip to content

Read Codex transcripts and flag re-asked requests - #2

Merged
ArkashJ merged 2 commits into
mainfrom
feat/codex-and-reask
Sep 2, 2026
Merged

Read Codex transcripts and flag re-asked requests#2
ArkashJ merged 2 commits into
mainfrom
feat/codex-and-reask

Conversation

@ArkashJ

@ArkashJ ArkashJ commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Harvested 195 transcripts (155 Claude + 55 Codex) after twenty-three days in which the
weekly cron logged nothing but Not logged in · Please run /login. Three defects fell out
of the run itself.

Codex reduced to zero turns and exited 0

reduce_session read rec["message"]["role"]; Codex rollouts carry the turn under
payload. On a 1.4MB rollout the tool printed a well-formed reduction reporting
human turns: 0 and returned success — indistinguishable from a quiet session.

Normalising Codex records inside records() fixes every reader at once: reduction, JSON,
--only-corrections, --repeats. --list scans ~/.codex/sessions alongside
~/.claude/projects (--codex-root overrides), and fork snapshots collapse by session id
— Codex writes one file per fork, so 1721 files since 2026-08-11 are 55 real sessions.

An empty reduction now exits non-zero and says so on stderr. Silent zero is worse than
unsupported.

Re-ask detection

The correction heuristics are lexical, so the most expensive class is invisible: a user
re-asking for something the assistant already reported done.

The labelled case from this corpus — 42 PRs added to a project board, verified at 43 items
via gh project item-list with every field set, while all six default views filtered on an
iteration that was never set, so the board the user actually opens rendered empty. The
user asked twice more, escalating. That session reduces to likely corrections: 0.

A turn that restates an earlier turn and adds specificity or force is now flagged reask
and marked . Containment, not Jaccard: escalation adds words rather than sharing them,
so an overlap score reads the repeat as a new request. Corpus rate 3.1% of turns, 58% of
them carrying no correction keyword
.

This does not catch the board case, and I would rather say so than tune until it does.
After stopwords that turn is three content words — below any threshold that keeps the
false-positive rate sane. Catching it needs the assistant's own completion claim as an
input, not more lexical overlap.

find_repeats was mostly false positives

61 pairs over the corpus. ~13 were one- and two-word turns (try again, check again)
whose word sets overlap by accident; ~14 were the same template prompt reused in two
sessions (/mission read docs/launch/MISSION.md, the compaction preamble).

A six-content-word floor, a near-verbatim skip for long blocks, and treating a leading
slash command as an invocation rather than prose: 61 pairs → 39, template reuse 16 → 2.

Verification

  • python3 -m unittest discover -s tests — 28 tests, all pass (8 new: Codex reduction,
    Codex shell/error recovery, Codex goal boilerplate, fork collapse, re-ask positive and
    negative, non-zero empty exit, repeat floor).
  • uv build and uvx twine check dist/* — both pass at 1.1.0.
  • Every number above is measured over the 195-transcript corpus, not estimated.

Version 1.1.0; CHANGELOG and README updated.

🤖 Generated with Claude Code

https://claude.ai/code/session_0189C5Lc8PeFkh88goiURRMN

ArkashJ and others added 2 commits September 2, 2026 00:39
Three defects surfaced by harvesting 195 transcripts (155 Claude, 55 Codex)
after twenty-three unharvested days.

Codex was a silent hole. reduce_session read rec["message"]["role"]; Codex
rollouts carry the turn under payload. A 1.4MB rollout produced a well-formed
reduction saying "human turns: 0" and exited 0 — indistinguishable from a quiet
session. Normalising Codex records inside records() fixes every reader at once,
and --list now scans ~/.codex/sessions, where dozens of fork snapshots per
session collapse by session id (1721 files, 55 sessions).

An empty reduction now exits non-zero. Silent zero is worse than unsupported.

The correction heuristics are lexical, so they cannot see the most expensive
class: a user re-asking for something already reported done. One labelled case
from this corpus — 42 PRs added to a project board, verified at 43 items via
gh project item-list, while every default view filtered on an unset iteration
so the board the user opens rendered empty. That session reduces to
"likely corrections: 0". Turns that restate an earlier turn with more
specificity or force are now flagged reask and marked ↩. Corpus rate 3.1% of
turns, and 58% of them carry no correction keyword at all.

find_repeats was mostly false positives: 61 pairs over the corpus, of which
~13 were one- and two-word turns whose word sets overlap by accident and ~14
were the same template prompt reused in two sessions. A six-content-word floor,
a near-verbatim skip for long blocks, and treating a leading slash command as an
invocation bring it to 39 pairs with template reuse down from 16 to 2.

Not fixed, and worth saying: the board case above still does not fire. After
stopwords it is three content words, below any threshold that keeps the
false-positive rate sane. Catching it needs the assistant's own completion
claim as an input, not more lexical overlap.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0189C5Lc8PeFkh88goiURRMN
The CI wheel and sdist smokes asserted `harness-extractor 1.0.0` as a literal, so
every version bump broke CI until someone remembered to edit the workflow. That is
a guard that needs a human to stay true, which is the failure mode this release
already has enough of.

Both smokes now read the installed distribution's version and check it against
harness_extractor.__version__ and the CLI's own --version. Same three-way
coherence the release workflow already enforces, minus the literal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0189C5Lc8PeFkh88goiURRMN
@ArkashJ
ArkashJ merged commit 94e836c into main Sep 2, 2026
13 checks passed
@ArkashJ
ArkashJ deleted the feat/codex-and-reask branch September 2, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant