diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 3a965e5..18f2c99 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,237 +1,65 @@ -# Knowledge flush — 13 insight(s) ingested (21 claimed, 2 dropped, 6 released) +# Knowledge flush — 12 insight(s) -Cross-Check: 1× independent adversarial `claude` CLI headless pass over the 5 new pages — it refuted the changed-files gate page's "prettier exits 0 on an empty match set" claim; re-measured against Prettier 3.7.4, confirmed the reviewer was right (unmatched operand exits **2**), and rewrote the page, report rows 3/5 and `log.md`. Other 5 claim groups verdicted sound. Limits: the reviewer's sandbox blocked repo reads, so source-quote-supports-directive and self-contradiction dimensions went unaudited (details in `## Cross-Check`). - -Queue drained under run id `20260827-125731-38371` (this session is the detached -`hooks/auto-flush.sh` run; its step-0 acquire resolved re-entrantly to -`already-owned`, not to a competing holder). 21 rows were claimed; 13 are -ingested below, 2 are retired as out-of-layer, and 6 are released back to -`pending` for a later flush because each needs its own page rather than a row, -and rushing six more pages in one pass would have lowered the bar on all of them. +12 candidates claimed from `~/.dev-loop/queue` (harvested 2026-08-19 … 2026-09-02). Outcome: 5 new pages, 4 amended main pages, 3 candidates folded onto open knowledge PRs (#180 ×2, #181 ×1), 1 dropped as stale. Every quote below was re-fetched with `curl` in this run after three read-only research agents drafted it; local reproductions were re-run in this session. ## Verified best-practice -Every external claim below was live-fetched this session and quoted in the page's -`Sources` block. Field evidence carries the repo, date, and the measured numbers. - -| # | Claim | Sources checked | How verified | Confidence | -|---|-------|-----------------|--------------|------------| -| 1 | `now()` is `transaction_timestamp()` (fixed at transaction start) while `clock_timestamp()` "changes even within a single SQL statement"; `RETURNING` yields computed defaults "without needing a separate database query" | postgresql.org `functions-datetime`, `dml-returning`, `transaction-iso` | Fetched; both key sentences quoted verbatim into the page | verified | -| 2 | A boundary recomputed in a follow-up step is a *second, later* `now`, widening a `<= boundary` set | Field: `rtb-unified` `packages/orpc/src/routers/batch.ts` — codifies "one `now` per decision" and passes `now` into the boundary helper; its result type omits the boundary, which is the shape that invites recomputation | Read the invariant and the signature in the cited file | field-tested | -| 3 | **[CORRECTED BY CROSS-CHECK]** The vacuous-pass shapes for `prettier --check` are: no operands (rc **0**), all operands ignore-filtered (rc **0**), and unsupported extensions with `--ignore-unknown` (rc **0**). A pattern/operand matching nothing exits **2** — it prints the success sentence *and* an unmatched-pattern error | prettier.io CLI + ignore docs; local measurement, Prettier 3.7.4 | The first draft generalised "empty match set ⇒ exit 0" from a field log where both messages appeared together. The independent reviewer flagged it; I then ran all seven cases against a real binary and rewrote the page around the measured table | verified (re-measured) | -| 4 | zsh does not word-split unquoted parameter expansions by default, so `cmd $FILES` arrives as **one** operand | zsh FAQ ch. 3 (`SH_WORD_SPLIT`) | Fetched; quoted ("By default, zsh does not have that behaviour: the variable remains intact") | verified | -| 5 | The zsh word-split operand exits **2**, but its log still carries the success sentence — so the log misleads even though the exit code does not | Field 2026-08-24 (`rtb-unified`, zsh) + local measurement 2026-08-27 | Field log showed both messages together; the local run reproduced it as `rc=2`. The page now says explicitly that this row fails loudly *unless* `--no-error-on-unmatched-pattern` is set. Probe placement re-confirmed: `.claude/tmp/` is `.gitignore`d, so a probe there passes at rc 0 | verified (re-measured) | -| 6 | TypeScript applies excess-property/contextual typing to fresh object literals, so a value of a type can be constructed with the type's name absent from the text | typescriptlang.org handbook, *Object Types* | Fetched; confirmed the check follows from the contextual type, not from a written annotation | verified | -| 7 | `tsc`'s program is `files` ∪ `include` ∪ transitive imports; `exclude` "only changes which files are included as a result of the `include` setting" and does not stop an imported file entering the program | typescriptlang.org TSConfig `#include`, `#exclude` | Fetched; the `exclude` sentence quoted (it sharpens the rule to "in the program", not "in `include`") | verified | -| 8 | Consequence of 6+7 measured | Field 2026-08-24/25 (`rtb-unified`): `grep "DealViewer"` reported 3 construction sites, actual 8 — the missed set included production wiring `routers/deal.ts:38`; `ContractScopeActor` 7→~22. Separately, `packages/orpc/tsconfig.json` `include: ["src/**/*"]` produced 3 production + 13 api-test errors and **zero** for `__tests__/routers/deal.test.ts`, whose 6 sites appeared only as 6 failing tests | Counts recorded from the cited runs | verified | -| 9 | cgroup v2: `memory.peak` is max usage since creation/reset; at `memory.max` "the OOM killer is invoked in the cgroup"; in `memory.events`, `max` counts times usage "was about to go over the max boundary" — **distinct** from `oom_kill` | docs.kernel.org cgroup-v2 admin guide | Fetched; all four quoted. This corrected the candidate, which had read a non-zero `max` as a kill; the page now states the distinction explicitly | verified | -| 10 | An `exec`'d process joins the container's cgroup and is invisible to the application's own semaphore | kubernetes.io `manage-resources-containers`, `assign-memory-resource`, `kubectl exec` reference + field 2026-08-26 (review-bot pod, `limits.memory: 3Gi`): `memory.current` 2.54 GiB, `memory.peak` 3.0 GiB (at the limit), `memory.events: max 5`, while `maxConcurrentAgents: 20` reported free slots | Docs fetched; pod numbers from the cited measurement | verified | -| 11 | Basename-keyed mutation backups collide across directories and restore cross-writes; an untracked file's `git diff` is empty whether restored or destroyed | Field 2026-08-21 (`rtb-unified`, NEWRTB-2936): restore wrote `schemas/deal.ts` into `routers/deal.ts` → `Cannot find module './common.js'`, `grep -c dealRouter` = 0; **both files were 154 lines**, so a line-count check passed; after re-keying, M9/M10 flipped SURVIVED→KILLED. Plus stryker mutant-states / pitest for the verdict vocabulary | Reproduced end to end in the cited run | field-tested | -| 12 | A negative assertion is vacuous when the fixture never supplies the triggering input | Field 2026-08-25 (`rtb-unified`): with `staleQueuedJobIds: []` the code early-returned; the widening the assertion claimed to catch survived 116/116 green | Mutation applied and observed | field-tested | -| 13 | A body-level (non-inline) review finding cites no file, so rebutting from an assumed file rejects real defects | Field 2026-08-19 (PR #327 r16): quote matched `report.py:393/416/425`, not the assumed `fill_plan.py:307` — sibling modules, one already fixed | Grep resolved the quote to the real site | field-tested | -| 14 | Unifying two duplicate allowlists defaults to the union and silently widens each side | Field 2026-08-25 (`rtb-unified` PR #965): folding `DISPLAYABLE_ERROR_CODES` into `USER_FACING_ERROR_CODES` would have added `UNAUTHORIZED` + `VALIDATION_ERROR`, exposing raw server messages as inline UI errors; caught only by computing the difference first | Difference computed before the merge | field-tested | - -Not upgraded: nothing was marked `verified` on field evidence alone. Two pages -carry `confidence: field-tested` (`mutation-harness-file-custody`, -plus the pre-existing `evaluating-review-feedback`), and no candidate was -recorded as `verified` without a fetched primary source. +1. **Fresh-context review of LLM output** (af294fb3f88881a1) — `verified`. https://arxiv.org/abs/2603.12123 (Song, "Cross-Context Review", Mar 2026) re-fetched: abstract reads "CCR reached an F1 of 28.6%, outperforming SR (24.6%, p=0.008, d=0.52), SR2 (21.7%, p<0.001, d=0.72), and SA (23.8%, p=0.004, d=0.57)"; SR2 vs SR p=0.11. Corroborated by this repo's own `agents/test-quality-auditor.md` ("so the session that wrote the code does not grade its own tests") and `agents/integration-reviewer.md` ("from a fresh context the coordinator's own session never reaches"). +2. **Bypass-string construction + coordinator reproduction for allowlist code** (5f91d867cf785a08) — `verified` for the technique (CWE-88 https://cwe.mitre.org/data/definitions/88.html; Cargo docs: `--manifest-path` package selection https://doc.rust-lang.org/cargo/commands/cargo-test.html, build scripts "It will then run the script" https://doc.rust-lang.org/cargo/reference/build-scripts.html, `--config` "should be in TOML syntax" https://doc.rust-lang.org/cargo/reference/config.html); the linkly-crew incident itself is field evidence (2026-08-31). +3. **Copied-plan premise must be read from the target manifest** (412141e74acbc743) — `field-tested` (dev-loop mpa1 run, 2026-08-23); the mechanism is an internal-repo fact (dev-loop's marketplace entry is a url-source self-reference — confirmed by `scripts/check-versions.sh`'s own comment). +4. **Parser vs executor separation in gate scripts** (2548049fc5087ca2) — `verified`. CWE-78 https://cwe.mitre.org/data/definitions/78.html; `timeout(1)` https://man7.org/linux/man-pages/man1/timeout.1.html ("run a command with a time limit", exit 124 "if COMMAND times out"); local read of `skills/loop-implement/scripts/gate-check.sh` (`--status … never executes CHECK`, `GATE_CHECK_TIMEOUT … via a perl alarm`). +5. **Teardown build-cache bloat** (fe6628a56fefc20b) — **dropped**. Candidate observed 2026-08-31; commit `d37fd67` (2026-09-02, "fix(orchestrate): stop archive_scratch from archiving gitignored deps (#170)") switched enumeration to `git ls-files --others --exclude-standard` and `tests/safe-cleanup.bats:746` guards it. The proposed manual recursive delete of `.worktrees/*/target` before teardown would be stale advice against the current tool. +6. **Coordinator under tmux, not a client-bound relay PTY** (c4ab8de2e9c06271) — `field-tested`. tmux(1) https://man7.org/linux/man-pages/man1/tmux.1.html ("Each session is persistent and will survive accidental disconnection", "all sessions are managed by a single server"); POSIX §11.1.10 https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap11.html ("the SIGHUP signal shall be sent to the controlling process for which the terminal is the controlling terminal"); Orca docs https://www.onorca.dev/docs/ssh ("A short grace period (5 minutes by default, configurable per target)") — this **corrects** the candidate's `--grace-time 0`, which is not carried into the page as fact. +7. **False login expiry with idle sibling sessions** (9a9038861c1d93fc) — `field-tested`, causal claim labelled hypothesis. https://code.claude.com/docs/en/authentication ("credentials are stored in `~/.claude/.credentials.json`", a different `CLAUDE_CONFIG_DIR` "reads a different entry"); RFC 9700 §2.2.2 https://www.rfc-editor.org/rfc/rfc9700.txt ("Refresh tokens for public clients MUST be sender-constrained or use refresh token rotation") supports the general pattern only. +8. **Plugin version drift gate** (5b73ca3b8aeee489) — `verified`. https://code.claude.com/docs/en/plugin-marketplaces ("Avoid setting `version` in both `plugin.json` and the marketplace entry. Claude Code always uses the `plugin.json` value without warning"); https://raw.githubusercontent.com/mattpocock/skills/main/scripts/sync-plugin-version.mjs (`--check` → `process.exit(1)`, wired as `check-plugin-version`); this repo's `scripts/check-versions.sh` + `.github/workflows/test.yml` "Version gate (marketplace.json vs plugin.json)" step. +9. **Rust inline tests invisible to test-floor.sh** (b9e2ed778fd6d661) — `verified`. Local read of `skills/orchestrate/scripts/test-floor.sh` `classify()` (patterns `*.bats`, `test_*.py`, `*_test.go`, `*.test.*`, `*/tests/*` — no `.rs`); Rust Book https://doc.rust-lang.org/book/ch11-03-test-organization.html ("put unit tests in the src directory in each file with the code that they're testing … create a module named tests in each file"). +10. **Unix socket `sun_path` limit in deep worktrees** (886a2b6aff8e8a6b) — `verified`. macOS SDK `sys/un.h` line 79 `char sun_path[104];`; Linux https://man7.org/linux/man-pages/man7/unix.7.html `char sun_path[108]`; Node https://nodejs.org/api/net.html ("Typical values are 107 bytes on Linux and 103 bytes on macOS"); **local reproduction this run**: `net.createServer().listen(<131-byte path under ~/.dev-loop/tmp>)` → `listen EINVAL`. +11. **Top-level await for sentinel-driven REPL payloads** (285976623fb9d890) — `verified`. https://nodejs.org/api/repl.html ("Support for the `await` keyword is enabled at the top level"); **local reproduction this run** (Node v26.7.0, `node -i` via piped stdin): unawaited IIFE printed only `Promise { }`, awaited form printed `done-awaited`. +12. **Stamp mock event timestamps at emit, assert monotonicity** (966887be5b42f5fc) — `field-tested`. Research agent read linkly-crew commit `530e7e2` ("stamp MockEventSource ts at delivery, not build time") and its regression test; no external doc names this specific pattern, so it rides on the host pages' verified methodology. ## Existing-layer check -Method: routed via `INDEX.md` → domain `index.md`; then built a full id+title -index of all 265 pre-existing pages and probed it with concept greps -(`clock_timestamp|clock skew`, `changed[- ]files|--ignore-unknown`, `tsconfig`, -`contextual typ|excess property`, `set difference|allowlist`, `cgroup`, -`basename|backup.*restore`, `2>&1`, `delta|baseline`) before deciding new vs merge. - -Pages read: testing-quality-source-text-wiring-assertions, testing-quality-tests-that-cannot-fail, backend-common-change-impact-call-site-enumeration, backend-common-change-impact-widening-a-closed-value-table, qa-process-evaluating-review-feedback, infrastructure-containers-host-cgroup-visibility, testing-quality-behavior-not-implementation +Pages read: qa-process-llm-review-pipelines, qa-process-evaluating-review-feedback, qa-process-adversarial-change-review, infrastructure-agent-orchestration-session-context-token-budget, security-input-validation-at-trust-boundaries, platforms-shells-option-like-argument-values, qa-exploratory-guard-true-path-coverage, qa-document-verification-spec-document-gates, infrastructure-config-keys-ahead-of-their-consumer, platforms-shells-command-text-inspected-before-execution, security-agent-exposure-in-session-tool-exposure, testing-quality-checks-that-cannot-pass, infrastructure-agent-orchestration-pane-delivery-confirmation, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts, infrastructure-agent-orchestration-session-completion-gates, infrastructure-agent-orchestration-usage-limit-paused-workers, platforms-tools-version-keyed-artifact-cache, qa-document-verification-generated-reference-drift-gates, infrastructure-ci-cd-pipeline-structure, platforms-tools-plugin-mcp-server-registration, testing-quality-tests-that-cannot-fail, testing-quality-unasserted-return-fields, testing-mocking-what-to-mock, platforms-filesystems-paths-case-and-line-endings, infrastructure-agent-orchestration-worktree-isolated-workers, testing-async-async-testing, platforms-processes-driving-a-tui-in-a-tmux-pane, platforms-processes-non-interactive-cli-invocation, qa-environments-browser-console-capture-gaps, databases-transactions-application-clock-vs-database-timestamps -Findings: +Also read on open-PR branches (not on main, so not listed above): `infrastructure-agent-orchestration-login-expiry-during-unattended-turns` and `testing-quality-assertion-scanner-false-positive-on-unittest-convention` (PR #180), `infrastructure-agent-orchestration-checkable-claims-in-an-adopted-plan` (PR #181). -- **Zero coverage** (→ new pages): changed-files-only gates, tsconfig/contextual - typing, allowlist set-difference, app-clock-vs-DB-timestamp, exec-into-a-running-container. - The concept greps returned no hits for these; the clock hits were incidental - (offline sync, token handling) and none compared an app clock to a DB column. -- **Already covered — one candidate all but retired.** The comment-stripping - insight is `source-text-wiring-assertions` step 2 verbatim ("Make the - assertion's subject the file with comments removed"), and its false-RED and - negative/count false-GREEN shapes are already edge rows. Only the *empty-slice* - consequence was new, so that alone was merged. -- **Line-cap conflict handled without breaking the invariant.** - `source-text-wiring-assertions` sits at exactly **120** body lines (the - documented cap). Rather than add a row and violate maintenance invariant 5, the - new nuance and the new field evidence were merged **in place** into an existing - edge row and an existing source bullet. Body count re-measured after editing: - still 120. -- **No conflicts found.** Nothing ingested contradicts an existing directive. - The one correction made was to a *candidate*, not to the wiki (item 9: the - `memory.events` `max` counter is approaches-to-limit, not kills). -- **Related links added both ways**: `tests-that-cannot-fail` ↔ - `mutation-harness-file-custody`; `widening-a-closed-value-table` ↔ - `compiler-as-call-site-inventory` (+ `errors-diagnostics-from-a-shared-code-path`); - `host-cgroup-visibility` → `exec-added-processes-and-the-memory-budget` - (from its existing self-monitoring row). -- **Indexes/log updated**: 4 domain indexes (+5 "load when" rows), `log.md` - appended. Root `INDEX.md` unchanged — no new domain. - -Gates run (the exact CI commands from `.github/workflows/test.yml`): -`node scripts/wiki-structure-checks.js wiki` → **pages: 270, indexes: 13, -findings: 0**; `node scripts/wiki-lint-prohibitions.js wiki` → **directives 72, -compliant 72, violations 0** (the 1 `info` is pre-existing in -`config/keys-ahead-of-their-consumer.md`, untouched); `bash scripts/check-versions.sh` -→ `ok: dev-loop 1.11.2`. The `bats tests/` job was **not** run — bats is not -installed on this machine, and this change touches only wiki markdown (no -scripts or hooks), so that suite's subject is unchanged. +- **Merged, not duplicated:** #2 → `qa/process/adversarial-change-review` (sharpens its technique 4 + confirm-before-blocking step; index row extended). #8 → `platforms/tools/version-keyed-artifact-cache` (the only main page mentioning `marketplace.json`; it stopped at "bump both by hand"). #12 → `testing/quality/unasserted-return-fields` (assertion side) + `testing/mocking/what-to-mock` (mock-authoring side, sibling of its clock row). +- **New pages** (no page with the same trigger): #1 `qa/process/fresh-context-code-review` (llm-review-pipelines is about building a CI pipeline, not which session reviews); #4 `testing/quality/gate-parsing-vs-command-execution` (spec-document-gates explicitly delegates check authoring to testing/quality); #6 `infrastructure/agent-orchestration/client-bound-pty-coordinator-loss` (sibling pages cover worker liveness, not coordinator PTY hosting); #10 `platforms/filesystems/unix-domain-socket-path-length` (paths-case-and-line-endings covers OS path limits, not the socket-address buffer); #11 `platforms/processes/sentinel-driven-repl-payloads` (driving-a-tui covers tmux echo, not the stdin sentinel race). +- **Conflicts:** none found. +- **Related links added both ways** only on main pages no open PR rewrites: session-context-token-budget ↔ fresh-context-code-review; command-text-inspected-before-execution and in-session-tool-exposure ↔ gate-parsing-vs-command-execution; pane-delivery-confirmation and session-completion-gates ↔ client-bound-pty-coordinator-loss; paths-case-and-line-endings ↔ unix-domain-socket-path-length (+1 edge row); driving-a-tui-in-a-tmux-pane, non-interactive-cli-invocation, async-testing ↔ sentinel-driven-repl-payloads; unasserted-return-fields → what-to-mock. +- **Back-links deliberately skipped** because an open PR rewrites that page's `related:` line (forward link only): llm-review-pipelines, evaluating-review-feedback, adversarial-change-review (#182); checks-that-cannot-pass (#179/#180); spec-document-gates (#179/#181); control-signals-vs-primary-artifacts, worktree-isolated-workers, tests-that-cannot-fail, what-to-mock (#179); usage-limit-paused-workers (#180); browser-console-capture-gaps (#182). ## Open-PR check -`gh pr list --repo choiyounggi/dev-loop --state open --search "head:knowledge/"` -returned **no open PRs**, and a second unfiltered `gh pr list --state open` -returned none either — the repository has zero open PRs at flush time. There -were therefore no in-flight sibling branches to diff against, and no -`git fetch origin ` / `git diff origin/main origin/ -- wiki/` -comparisons to run. - -Per-candidate verdict: **all 21 = `new`.** No `fold`, no `drop-as-pending-duplicate`. -(The 2 drops recorded below are out-of-layer drops, not pending-duplicate drops.) - -## Routing decision - -**New pages (5)** - -| Page | Domain/category | From | Why not an existing page | -|------|-----------------|------|--------------------------| -| `application-clock-vs-database-timestamps` | databases / transactions | `2b27d15d` + `bea92fdd` | No page compares an app clock to a DB column. `transactions` chosen over `schema-design` because the decisive content is transaction-time semantics (`now()` = transaction start ⇒ stamp order ≠ commit order) and the fix is a lock/isolation choice | -| `changed-files-only-gates` | infrastructure / ci-cd | `ff041061` + `4b9af3a0` | Zero grep hits. Both candidates are the same defect (a gate green with an empty subject) from two directions, so they became one page rather than two | -| `compiler-as-call-site-inventory` | backend / common / change-impact | `702dcf4e` + `94d55f2f` | `call-site-enumeration` is the sibling case (callers of a changed signature, Python positional-vs-keyword) and is at 80 body lines; the TS mechanism is *constructors of a type* with its own workflow, so per "one case per page" it is a separate page, cross-linked | -| `mutation-harness-file-custody` | testing / quality | `6a9de235` + `41fa1c87` | `harness-reverse-controls` covers scoring a harness; nothing covers the harness's custody of the tree. Both candidates are that one case (keying, and the read window) | -| `exec-added-processes-and-the-memory-budget` | infrastructure / containers | `7b9e8788` | `host-cgroup-visibility` is cross-pod read mechanics and explicitly routes self-monitoring elsewhere; `resource-limits-and-probes` is manifest authoring. This is a runtime preflight before adding load | - -No new category was created — all five landed in existing categories. - -**Merged into existing pages (5 candidates)** - -| Candidate | Merged into | Shape | -|-----------|-------------|-------| -| `91ef5d53` | `testing-quality-tests-that-cannot-fail` | +1 never-fails row, +1 Instead-of row, +1 source | -| `f189f423` | `testing-quality-source-text-wiring-assertions` | In-place extension of 1 edge row + 1 source bullet (page at the 120-line cap) | -| `bb6d8539` | `backend-common-change-impact-widening-a-closed-value-table` | +Do-this 6 & 7 (incl. a set-difference ruling table), +1 Instead-of row, +1 source | -| `60a817ee` | `qa-process-evaluating-review-feedback` | +2 edge rows, +1 Instead-of row, +1 source | -| `7b9e8788` | `infrastructure-containers-host-cgroup-visibility` | Cross-link from its self-monitoring row to the new page | +Open `knowledge/*` heads listed via `gh pr list --search "head:knowledge/"`: #179 `knowledge/choiyounggi-20260903-172728`, #180 `knowledge/choiyounggi-20260903-184706`, #181 `knowledge/choiyounggi-20260903-203836`, #182 `knowledge/choiyounggi-20260903-214027`. Each head was fetched and diffed against `origin/main -- wiki/`. -**Dropped — out of layer (2, retired)** +| Candidate | Overlapping head | Verdict | +|---|---|---| +| #7 false login expiry (9a9038861c1d93fc) | #180 `login-expiry-during-unattended-turns.md` — same symptom, but that page treats expiry as always genuine | **fold** — pushed to #180 as commit `bc1cac7` (+1 edge row, +3 sources, index row extended), PR comment posted | +| #9 Rust inline tests vs test-floor (b9e2ed778fd6d661) | #180 `assertion-scanner-false-positive-on-unittest-convention.md` — same root cause (path/name classifier with no entry for the convention) | **fold** — same commit `bc1cac7` (+When-this-applies paragraph, +Do item 6, +1 edge row, +3 sources, `applies_to` +rust, index row extended) | +| #3 copied-plan premise (412141e74acbc743) | #181 `checkable-claims-in-an-adopted-plan.md` — same page family, case not yet present | **fold** — pushed to #181 as commit `58c6bd1` (+Do item 7, +1 Finding row, +1 source, log line), PR comment posted | +| #2 allowlist bypass | #181 touches `validation-at-trust-boundaries` (spatial-range case), #182 touches `adversarial-change-review` `related:` line only | **new** (merge on main, body hunks do not collide) | +| #4 parser vs executor | #179/#181 touch `spec-document-gates` (word-boundary fix, `related:`), #179/#180 touch `checks-that-cannot-pass` `related:` | **new** page; skipped those back-links | +| #1, #6, #8, #10, #11, #12 | none | **new** | +| #5 teardown cache | none | **drop** (stale against `d37fd67`, see above) | -- `094dedf3` — a Figma MCP `inspect_node` → `get_dev_ready` children-fetch - workaround. The server is a private, org-internal MCP plugin; the behavior is - not publicly verifiable and the directive does not transfer to any other reader. -- `e165a365` — an `/rtb:review` remote-fallback runbook naming - `~/.claude/tools/rtb-remote-review.sh` and an internal pod. The transferable - kernel ("a two-provider review gate degraded to one provider is not a passed - gate") is already the subject of `qa-process-llm-review-pipelines`; what remains - is machine-specific paths. +No sibling duplicate PR was opened; this flush's PR carries only main-branch changes. -**Released back to `pending` (6)** — each needs its own page, not a row, and is -better served by a dedicated pass than by being appended here: -`81dc1f98` (naming the carrier field/type when a plan says "wire A to B"), -`b9ae304a` (`VAR="$(cmd 2>&1)"` mixing stderr into a value used as a path), -`fdd0b3c6` (monitor markers anchored at line start; delta rather than absolute -state; first cycle records a baseline), -`c2adb2be` (positional-order assertions on rendered SQL predicates), -`815e8cb9` (grep only *active* `DATABASE_URL` assignments, and confirm which -dotenv file the tool loads, before a destructive DB command), -`f1146adb` (CI ticket-key extraction scoped by changed-file intersection rather -than by mention). - -## Decision Log - -**Intent.** Drain the harvested `★ Insight` queue into reviewable wiki knowledge -without lowering the wiki's evidence bar. The queue held 21 rows accumulated over -several days; the goal was correct routing and real verification, not a high -ingest count. - -**Alternatives considered and rejected.** - -- *Ingest all 21 in this pass.* Rejected: six of them each need their own page, - and writing six more pages in one pass would have produced thin, weakly-sourced - entries. They are released to `pending`, not dropped, so the next flush takes - them with a full budget. -- *Append the two TypeScript candidates to `call-site-enumeration`.* Rejected: - that page is the sibling case (callers of a changed signature, Python - positional-vs-keyword). AGENTS.md requires one case per page, so the - constructor-enumeration case became its own page, cross-linked both ways. -- *Add a row to `source-text-wiring-assertions` for the empty-slice nuance.* - Rejected: that page is at exactly the documented 120-line body cap, so adding a - line would violate maintenance invariant 5. The nuance was merged **in place** - into an existing edge row instead; body re-measured at 120. -- *Drop the comment-stripping candidate entirely as a duplicate.* Rejected: its - directive is already the page's step 2, but the empty-slice consequence - (vacuous **green**, not the documented noisy red) was genuinely absent. -- *Claim a cross-check exemption because this PR cannot merge itself.* Rejected — - see below; the check found a real error, which is the argument against exempting. -- *Push to `origin`* as the skill's snippet does. Not available: this contributor - has no write access to `choiyounggi/dev-loop` (403). Used the pre-existing - `fork` remote, which is how every prior knowledge branch here was published. -- *Branch name from `git config user.name`.* The skill's ASCII sanitisation of a - Korean name yields an empty string → `anon`, defeating the attribution the - branch name exists for. Used the gh login, matching existing branch names. - -**Where reviewers should look hardest.** - -1. `infrastructure/ci-cd/changed-files-only-gates.md` — rewritten after the - cross-check. The measured table is the load-bearing part; please sanity-check - it against your own Prettier version, since the exit codes are version-visible - behaviour rather than a documented contract. -2. `databases/transactions/application-clock-vs-database-timestamps.md` step 5–6 — - the claim that timestamp order is not commit order, and that the remedy is a - lock/isolation level rather than finer clock resolution. `[추정]` on the MySQL - `NOW()`/`SYSDATE()` row: taken from general MySQL semantics, not fetched this - session like the PostgreSQL pages were. -3. `widening-a-closed-value-table.md` Do-this 6–7 — this inserts a security-shaped - concern (allowlist widening) into a page whose original subject was value - tables. If that reads as two cases, it should be split. -4. The 2 dropped candidates — if you consider private-tooling runbooks in scope - for this wiki, they should be restored rather than retired. - -## Cross-Check - -Independent adversarial pass via `claude` CLI headless (separate process, no -shared context), prompted to refute rather than confirm, over the five new pages' -technical claims. - -**It found a real error, and the page was rewritten because of it.** The reviewer -challenged the claim that `prettier --check` exits 0 on an empty match set, -arguing an unmatched pattern errors by default and that exit-0 belongs to the -ignore-filtered case. I resolved it by measurement rather than by argument — -running all seven cases against Prettier 3.7.4 — and the reviewer was right: -an unmatched operand exits **2** (while still printing the success sentence), -whereas the genuine silent vacuous passes are no-operands, all-ignore-filtered, -and `--ignore-unknown`-with-unsupported-extensions. The page, this report's -rows 3 and 5, and the `log.md` entry were all corrected. - -Verdicts on the other five claim groups: **sound** (PostgreSQL clock semantics — -noted as if anything *understated*; zsh word-splitting; TS contextual typing; -`tsc` program membership incl. `exclude`-does-not-stop-imports; cgroup v2 -`max` vs `oom_kill` and `kubectl exec` cgroup placement). - -Stated limits of the check: the reviewer's sandbox denied it read access to -`~/.dev-loop/repo/wiki`, so it adjudicated the six claims as quoted in its prompt -and could **not** audit (b) whether each `Sources` quote supports the directive it -is cited for, or (c) whether any page contradicts its own edge-case rows. Those -two dimensions remain unreviewed by an independent party and are the residual -risk in this PR. A first attempt also returned only the session's Stop-hook -output rather than a verdict; that run was discarded rather than read as -"no findings". - -## Review notes +## Routing decision -- PR-only, as required: no merge, no push to `main`. -- Commit is under the contributor's own ambient git identity - (`최영기 `, gh `dch0202-rsquare`); no assistant identity - and no `Co-Authored-By` trailer. The branch uses the gh login because - sanitizing the Korean `user.name` to ASCII yields an empty string, which the - skill's snippet would have turned into `anon` — that would have defeated the - attribution the branch name exists for. -- Scope purity: only `wiki/**`, four domain indexes, `log.md`, and this report. - Two untracked leftovers from earlier flushes - (`.dev-loop/CROSSCHECK_FINDINGS.md`, `.dev-loop/fold-note-73.md`) were left - untouched and unstaged. +| Candidate | Target | Why | +|---|---|---| +| #1 fresh-context review | `qa/process/fresh-context-code-review` (new) | Review-process decision; peer of llm-review-pipelines / evaluating-review-feedback with a distinct trigger | +| #2 allowlist bypass construction | `qa/process/adversarial-change-review` (merge) | Same trigger family — high-risk diff review at integration stage | +| #3 copied-plan premise | `infrastructure/agent-orchestration/checkable-claims-in-an-adopted-plan` (fold, PR #181) | Plan-adoption checks live there | +| #4 parser vs executor | `testing/quality/gate-parsing-vs-command-execution` (new) | spec-document-gates states check authoring belongs to testing/quality | +| #5 teardown cache | dropped | Fixed on main 2026-09-02 (#170) | +| #6 coordinator PTY | `infrastructure/agent-orchestration/client-bound-pty-coordinator-loss` (new) | Sibling of the tmux/orchestration lifecycle pages | +| #7 false login expiry | `infrastructure/agent-orchestration/login-expiry-during-unattended-turns` (fold, PR #180) | Same symptom page | +| #8 version drift gate | `platforms/tools/version-keyed-artifact-cache` (merge) | Only page on the plugin.json/marketplace.json pair; adds the CI-prevention half | +| #9 Rust inline tests | `testing/quality/assertion-scanner-false-positive-on-unittest-convention` (fold, PR #180) | Same scanner-gap root cause | +| #10 sun_path | `platforms/filesystems/unix-domain-socket-path-length` (new) | Path-length is a filesystem concern per the domain description | +| #11 REPL sentinel | `platforms/processes/sentinel-driven-repl-payloads` (new) | Driving an external process, matching the category's existing pages | +| #12 mock timestamps | `testing/quality/unasserted-return-fields` + `testing/mocking/what-to-mock` (merge rows) | Assertion side and mock-authoring side each have an exact host page | + +No new category was needed. Lint: `wiki-structure-checks.js wiki` → 280 pages, 0 findings; `wiki-lint-prohibitions.js wiki` → 0 violations (1 pre-existing info). diff --git a/log.md b/log.md index 3f8265b..49b754e 100644 --- a/log.md +++ b/log.md @@ -103,3 +103,4 @@ Append-only. Format: `## [YYYY-MM-DD] `, repeated upward). A + coordinator whose chain ends in `claude ← zsh ← tmux server` survives a + client disconnect; one ending in `claude ← zsh ← ` + is reclaimed when that relay tears the PTY down. +3. **Read a documented reconnect grace period as a bound on survivable + client disconnects, not as persistence.** Orca's SSH targets, for example, + default to 5 minutes, configurable per target; a coordinator meant to run + for hours goes under tmux regardless of the configured grace period. + +## Edge cases + +| Case | Then | +|------|------| +| The product's grace period is generous (several minutes) | It still expires on any disconnect longer than the window, or on a relay/daemon restart; tmux's server has no such window, so prefer it for any run that can outlive the grace period | +| Reattaching would race an orchestrator wait-loop that already marked the coordinator dead | Pause that wait-loop before reattaching, then resume it — otherwise the orchestrator's own liveness logic double-drives the same coordinator | +| tmux is not available on the remote host | Use the relay/terminal product's own persistent-session primitive if it has one, distinct from a live PTY lease, and verify its survival semantics with the same parent-chain check — "remote" does not imply "persistent" | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Run the coordinator directly in the remote/SSH terminal card | Start `tmux new -s coordinator` first, run the coordinator inside it, and use the card only to `tmux attach` | The card's PTY is leased to the client connection; tmux's server is a separate process a client disconnect never touches | +| Assume a vanished coordinator crashed and re-dispatch its work from scratch | Check the PTY's parent chain and the relay's own connection/reconnect log first | A relay-reclaimed PTY leaves no crash signal in the transcript — re-dispatching from scratch discards a coordinator that was mid-run, not failed | + +## Sources + +- https://man7.org/linux/man-pages/man1/tmux.1.html — "Each session is persistent and will survive accidental disconnection (such as ssh connection timeout) or intentional detaching"; "a session is displayed on screen by a client and all sessions are managed by a single server. The server and each client are separate processes" +- https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap11.html §11.1.10 — "If a modem disconnect is detected by the terminal interface for a controlling terminal ... the SIGHUP signal shall be sent to the controlling process for which the terminal is the controlling terminal" — the general mechanism behind a client-bound PTY's process dying on disconnect +- https://www.onorca.dev/docs/ssh — "A short grace period (5 minutes by default, configurable per target) gives the relay time to ride out a quick reconnect before tearing down detached sessions"; reconnected "leased PTYs are restored to their tabs in the attached state, with their scrollback intact" — a bounded grace window, not indefinite persistence +- Field evidence 2026-09-02 (measured in a linkly-crew orchestration run): a coordinator transcript ended after an `away_summary` with no crash; OOM, relay restart, and daemon kill were each ruled out; the death window coincided with relay socket churn, a relay-watcher restart, and a new remote shell being created. The grace-time value reported for that run (0s) could not be confirmed against the product's public default (5 min) and is a local configuration detail, not a documented default diff --git a/wiki/infrastructure/agent-orchestration/pane-delivery-confirmation.md b/wiki/infrastructure/agent-orchestration/pane-delivery-confirmation.md index 1e3008f..386d754 100644 --- a/wiki/infrastructure/agent-orchestration/pane-delivery-confirmation.md +++ b/wiki/infrastructure/agent-orchestration/pane-delivery-confirmation.md @@ -8,7 +8,7 @@ sources: - https://man7.org/linux/man-pages/man3/termios.3.html - https://man7.org/linux/man-pages/man1/tmux.1.html last_verified: 2026-08-25 -related: [platforms-shells-option-like-argument-values, infrastructure-agent-orchestration-session-completion-gates, platforms-processes-non-interactive-cli-invocation, infrastructure-agent-orchestration-unattended-worker-questions, platforms-processes-driving-a-tui-in-a-tmux-pane] +related: [platforms-shells-option-like-argument-values, infrastructure-agent-orchestration-session-completion-gates, platforms-processes-non-interactive-cli-invocation, infrastructure-agent-orchestration-unattended-worker-questions, platforms-processes-driving-a-tui-in-a-tmux-pane, infrastructure-agent-orchestration-client-bound-pty-coordinator-loss] --- # Confirming a Keystroke Sent to a Terminal Pane Was Actually Consumed diff --git a/wiki/infrastructure/agent-orchestration/session-completion-gates.md b/wiki/infrastructure/agent-orchestration/session-completion-gates.md index 137c5bc..50a05f5 100644 --- a/wiki/infrastructure/agent-orchestration/session-completion-gates.md +++ b/wiki/infrastructure/agent-orchestration/session-completion-gates.md @@ -8,7 +8,7 @@ sources: - https://code.claude.com/docs/en/hooks - https://csf.tools/reference/nist-sp-800-53/r5/ac/ac-5/ last_verified: 2026-08-13 -related: [infrastructure-agent-orchestration-pane-delivery-confirmation, infrastructure-agent-orchestration-worktree-isolated-workers, platforms-processes-tool-diagnostics-without-a-failing-exit-code, infrastructure-agent-orchestration-dispatching-after-a-completion-report] +related: [infrastructure-agent-orchestration-pane-delivery-confirmation, infrastructure-agent-orchestration-worktree-isolated-workers, platforms-processes-tool-diagnostics-without-a-failing-exit-code, infrastructure-agent-orchestration-dispatching-after-a-completion-report, infrastructure-agent-orchestration-client-bound-pty-coordinator-loss] --- # A Gate That Blocks a Worker Session from Ending Mid-Workflow diff --git a/wiki/infrastructure/agent-orchestration/session-context-token-budget.md b/wiki/infrastructure/agent-orchestration/session-context-token-budget.md index c23e719..995f7b2 100644 --- a/wiki/infrastructure/agent-orchestration/session-context-token-budget.md +++ b/wiki/infrastructure/agent-orchestration/session-context-token-budget.md @@ -13,7 +13,7 @@ sources: - https://platform.claude.com/docs/en/build-with-claude/vision - https://www.anthropic.com/engineering/multi-agent-research-system last_verified: 2026-08-21 -related: [infrastructure-agent-orchestration-worktree-isolated-workers, infrastructure-agent-orchestration-usage-limit-paused-workers, backend-common-llm-context-window-budget] +related: [infrastructure-agent-orchestration-worktree-isolated-workers, infrastructure-agent-orchestration-usage-limit-paused-workers, backend-common-llm-context-window-budget, qa-process-fresh-context-code-review] --- # Token Budget for Long-Lived Coordinator and Worker Agent Sessions diff --git a/wiki/infrastructure/index.md b/wiki/infrastructure/index.md index f781a71..5c05e16 100644 --- a/wiki/infrastructure/index.md +++ b/wiki/infrastructure/index.md @@ -22,6 +22,7 @@ Match your situation to a "load when" line; load only matching pages. | [worktree-isolated-workers](agent-orchestration/worktree-isolated-workers.md) | Authoring the brief/output contract for parallel workers each confined to its own git worktree; workers stall at the same phase with no task-level error; deciding where shared or produced artifacts live and which direction (read vs write) a worktree guardrail stops; a guardrail escalates on read-only access to another worktree; the isolation guard is a Bash-command hook while workers also edit files with native Edit/Write tools | | [autonomous-decision-rulings](agent-orchestration/autonomous-decision-rulings.md) | An unattended agent hits a decision its plan does not answer and must choose between stopping to ask and proceeding; a run stalls on questions no human needed to see; deciding which decision categories require a human; recording autonomous decisions for audit; resuming after interruption/compaction without re-dispatching completed work | | [session-context-token-budget](agent-orchestration/session-context-token-budget.md) | Planning or running long-lived coordinator/worker agent sessions and deciding when to compact or clear context; a run's cost is dominated by cache reads; screenshots or large file reads are entering a long-lived session; choosing slot counts / per-phase token budgets for an orchestrated run | +| [client-bound-pty-coordinator-loss](agent-orchestration/client-bound-pty-coordinator-loss.md) | A long-running orchestration coordinator runs directly inside a remote/SSH terminal card or relay-owned PTY instead of tmux; a coordinator session vanishes with no crash, OOM, or task-level error in its transcript; deciding whether a vanished coordinator crashed or its PTY was reclaimed by a client-lifetime-bound relay; a relay/terminal product's reconnect grace period is being treated as long-run persistence | ## ci-cd diff --git a/wiki/platforms/filesystems/paths-case-and-line-endings.md b/wiki/platforms/filesystems/paths-case-and-line-endings.md index 0cf95c1..b0c3031 100644 --- a/wiki/platforms/filesystems/paths-case-and-line-endings.md +++ b/wiki/platforms/filesystems/paths-case-and-line-endings.md @@ -9,7 +9,7 @@ sources: - https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file - https://git-scm.com/docs/gitattributes last_verified: 2026-07-10 -related: [platforms-shells-portable-shell-scripts, platforms-environment-unicode-text-matching] +related: [platforms-shells-portable-shell-scripts, platforms-environment-unicode-text-matching, platforms-filesystems-unix-domain-socket-path-length] --- # Files That Break When a Repo Moves Between macOS, Windows, and Linux @@ -62,6 +62,7 @@ a script fails with `bad interpreter`, or generated paths break on Windows. | A file must keep CRLF (`.bat`, `.ps1` consumed by cmd) | Per-pattern override in `.gitattributes`: `*.bat text eol=crlf` | | Long paths are unavoidable on a Windows target | The 260 limit is removable via registry/Group Policy and the `\\?\` prefix, but tools without long-path support still fail — shallow trees remain the fix that works everywhere | | Case-renamed file loops as modified/untracked in git | `git config core.ignorecase` disagrees with the filesystem — redo the rename as two steps and leave `core.ignorecase` at git's auto-detected value | +| The path being generated will be used to bind a unix domain socket, not just opened as a file | The filesystem's own path limit does not apply — `sun_path` is a fixed 104-byte (macOS) / 108-byte (Linux) buffer regardless of the OS's general path limit ([platforms-filesystems-unix-domain-socket-path-length]) | ## Instead of diff --git a/wiki/platforms/filesystems/unix-domain-socket-path-length.md b/wiki/platforms/filesystems/unix-domain-socket-path-length.md new file mode 100644 index 0000000..bb5a679 --- /dev/null +++ b/wiki/platforms/filesystems/unix-domain-socket-path-length.md @@ -0,0 +1,75 @@ +--- +id: platforms-filesystems-unix-domain-socket-path-length +domain: platforms +category: filesystems +applies_to: [macos, linux, node, general] +confidence: verified +sources: + - https://man7.org/linux/man-pages/man7/unix.7.html + - https://nodejs.org/api/net.html + - macOS SDK `sys/un.h` — `char sun_path[104];` +last_verified: 2026-09-04 +related: [platforms-filesystems-paths-case-and-line-endings, infrastructure-agent-orchestration-worktree-isolated-workers, platforms-processes-non-interactive-cli-invocation] +--- + +# A Unix Domain Socket Bind Failing Only Inside a Deep Worktree Path + +## When this applies + +Binding or listening on a unix domain socket (an IPC test, a dev-server +socket) fails with "Failed to listen", `listen EINVAL`, `ENAMETOOLONG`, or +"AF_UNIX path too long" only when run from inside a git worktree +(`.worktrees//...`) or another deeply nested directory, while the +identical code passes from the main checkout or a shorter path. Also when +choosing where to place a socket file for a test suite or IPC channel. + +## Do this + +1. **Compare against the fixed socket-address limit before treating this as + a code bug.** Unix domain socket addresses use a fixed-size `sun_path` + buffer, not the filesystem's general path-length limit — macOS: 104 bytes; + Linux: 108 bytes. A path well within the OS's normal file-path limit + (1024 on macOS, 4096 on Linux) still overflows this buffer. +2. **Measure the candidate socket path's byte length** + (`printf '%s' "$path" | wc -c`) before debugging further — a + `.worktrees//node_modules/.cache/...` prefix alone consumes + 60–80 bytes before the socket's own filename is appended. +3. **Choose the fix by where the path is generated:** + +| Case | Do | +|------|----| +| Socket path comes from `os.tmpdir()` or another fixed short root | Keep it there regardless of cwd; derive nothing from the project directory | +| Socket path is derived from the repo/worktree root (`path.join(__dirname, ...)`, `process.cwd()`) | Redirect it to a short, fixed location outside the repo so worktree depth cannot affect it | +| Verifying whether a suite failure is this limit or a real regression | Run the identical suite from the main checkout (short path) alongside the worktree run; a pass in the short path and a listen failure in the worktree path is this limit | +| A path is already short but still fails | Check for a symlinked or resolved absolute path longer than the literal one written in code — `realpath` can lengthen a path that looked short | + +4. **Read the failure from the library's own docs when in doubt.** Node's + `net`/`http` server "will throw an error when the length of pathname is + greater than the length of `sizeof(sockaddr_un.sun_path)`. Typical values + are 107 bytes on Linux and 103 bytes on macOS" (one less than the raw + buffer because the null terminator is excluded); on macOS the error + surfaces as `listen EINVAL`, not as a path-length message. + +## Edge cases + +| Case | Then | +|------|------| +| The failing suite runs under several worktrees at once (parallel orchestration) | Give each worktree's sockets a short, run-scoped root outside any worktree path, keyed by run id rather than by worktree path | +| The socket must live inside the worktree for cleanup-on-delete semantics | Create a short symlink from a fixed short path into the worktree location, and point the socket at the symlink | +| Switching to `os.tmpdir()` still overflows | `TMPDIR` itself can be a long per-user path (macOS `/var/folders/...`, some CI images) — measure its length before assuming it is short | +| The error message is generic ("Failed to listen") with no path-length mention | The wrapping library does not always surface `bind(2)`'s errno clearly — reproduce the failing bind call directly (`node -e` with `net.createServer().listen(path)`) to see the underlying error code | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Debug application logic because a socket test fails only in CI or worktree checkouts | Compare the socket path's byte length against 104 (macOS) / 108 (Linux) first | The failure is a fixed protocol-level buffer limit unrelated to the code under test — nesting the checkout one directory deeper is enough to cross it | +| Derive a unix socket's path from the project or worktree directory "for locality" | Derive it from a short, run-scoped path outside the repo | The socket address buffer is far shorter than the filesystem's own path limit, so any project-rooted path is one refactor away from crossing it | + +## Sources + +- macOS SDK `sys/un.h` (confirmed on this host 2026-09-04, `/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/un.h`): `char sun_path[104]; /* [XSI] path name (gag) */` +- https://man7.org/linux/man-pages/man7/unix.7.html — `struct sockaddr_un { sa_family_t sun_family; char sun_path[108]; }` +- https://nodejs.org/api/net.html — "It will throw an error when the length of pathname is greater than the length of `sizeof(sockaddr_un.sun_path)`. Typical values are 107 bytes on Linux and 103 bytes on macOS." +- Local reproduction 2026-09-04 (macOS, Node v26.7.0): `net.createServer().listen(<131-byte path>)` failed with `listen EINVAL: invalid argument ` — no `ENAMETOOLONG` and no mention of length in the message +- Field evidence 2026-08-19 (measured in a linkly-crew orchestration run): an IPC-listen test suite failed 14 cases from a git-worktree checkout and passed all of them from the main checkout; comparing the socket path's byte length against the worktree prefix explained the split diff --git a/wiki/platforms/index.md b/wiki/platforms/index.md index 8200fae..96cd8b6 100644 --- a/wiki/platforms/index.md +++ b/wiki/platforms/index.md @@ -30,7 +30,7 @@ Match your situation to a "load when" line; load only matching pages. | [bsd-vs-gnu-cli](tools/bsd-vs-gnu-cli.md) | A command works on Linux but fails on macOS or vice versa (`date`, `sed -i`, `timeout`, `seq`, `grep -P`, `readlink`, `stat`); writing a script or CI step that must run on both userlands; deciding whether to install GNU coreutils on macOS or write POSIX-only | | [harness-mediated-tool-results](tools/harness-mediated-tool-results.md) | A plugin or hook in your agent harness returned substitute content for a built-in tool (truncated read, redaction, a note telling you to call something else); deciding whether a short result is interception or a genuinely small/empty file; the hook's suggested workaround failed too; briefing spawned worker sessions about a known-degraded tool | | [plugin-mcp-server-registration](tools/plugin-mcp-server-registration.md) | A plugin-bundled MCP server is missing from `/mcp` or its tools are uncallable while the plugin's skills work; deciding whether the fault is registration, config shape, or the server itself; about to change a plugin `.mcp.json` (bare map vs `mcpServers` wrapper, `command`/`args`/`env`) to make a server appear; relaying a shell variable into a server with `${VAR}` and the server reads the wrong value; a plugin updated mid-session and its server still runs the old path | -| [version-keyed-artifact-cache](tools/version-keyed-artifact-cache.md) | Shipping a code update to a distribution system that caches artifacts by a version string (a Claude Code marketplace plugin, or any tag-pinned cache) and the update runs but the old behavior persists; deciding why `/plugin update` reports "at latest" yet new code never runs; locating and clearing a stale `~/.claude/plugins/cache////` | +| [version-keyed-artifact-cache](tools/version-keyed-artifact-cache.md) | Shipping a code update to a distribution system that caches artifacts by a version string (a Claude Code marketplace plugin, or any tag-pinned cache) and the update runs but the old behavior persists; deciding why `/plugin update` reports "at latest" yet new code never runs; locating and clearing a stale `~/.claude/plugins/cache////`; setting up a plugin repo's `plugin.json`/`marketplace.json` pair so future drift fails CI instead of shipping silently | | [unpacked-extension-source-reload](tools/unpacked-extension-source-reload.md) | You edited a file of an unpacked Chromium extension (content script, service worker, manifest) and are about to judge the change in the browser, or the edit appears to have no effect; deciding which of extension-reload and host-page-reload a given file requires; adding a running-build version marker; automating the `chrome://extensions` reload control from an agent or CDP script | @@ -47,6 +47,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [paths-case-and-line-endings](filesystems/paths-case-and-line-endings.md) | A repo moves between macOS/Windows/Linux and files disappear or collide; an import resolves locally but fails on Linux CI (casing); renaming only the case of a file; diffs show every line changed or a script dies with `bad interpreter: ^M` (CRLF); setting up `.gitattributes` line-ending policy; generating file names or paths that must be valid on Windows (reserved names, path length) | +| [unix-domain-socket-path-length](filesystems/unix-domain-socket-path-length.md) | A unix-domain-socket bind/listen fails with "Failed to listen", `listen EINVAL`, `ENAMETOOLONG`, or "AF_UNIX path too long" only inside a deep path (git worktree, nested cache dir) and passes from a shorter path; choosing where to place a socket file for a test suite or IPC channel; verifying whether a suite failure is the fixed `sun_path` buffer limit (104 bytes macOS, 108 Linux) or a real regression | | [permissions-and-exec-bits](filesystems/permissions-and-exec-bits.md) | "Permission denied" running a script that exists; a script loses its executable bit through git/Windows/zip/CI artifacts; surprise file-mode diffs in git (`core.fileMode`); docker bind-mount files root-owned or unreadable (host/container uid mismatch); pipeline stages can't read each other's artifacts (umask); setting up a shared directory for several users/daemons; reviewing file-permission handling in a repo or pipeline | | [deleted-file-recovery-on-apfs](filesystems/deleted-file-recovery-on-apfs.md) | Someone asks you to recover a file or folder deleted on macOS and you are about to recommend a recovery tool; deciding whether free-space carving is available at all (TRIM/APFS) before spending time on it; ordering the copy sources (Trash, APFS local snapshot, Time Machine, cloud trash); recovering an exact path from an app's stored bookmark data when the remembered name is wrong | @@ -59,6 +60,7 @@ Match your situation to a "load when" line; load only matching pages. | [parsing-cli-structured-output](processes/parsing-cli-structured-output.md) | About to write automation that parses another CLI/tool's `--json` output (field names, nesting) — wrapping a desktop app's CLI, an orchestrator, a cloud tool; deciding how to confirm exact field paths without guessing; making the parser unit-testable without the live tool via a captured fixture | | [cloud-cli-invocation-bounds](processes/cloud-cli-invocation-bounds.md) | About to run `aws`/`gcloud`/`az`/`kubectl` from a script, hook, or agent turn with flags recalled from memory; a list command floods the session; a command returned nothing and absence cannot be told from an auth failure; a command turned out to have targeted the wrong account, project, or region; turning off the pager and interactive prompts for an unattended caller | | [driving-a-tui-in-a-tmux-pane](processes/driving-a-tui-in-a-tmux-pane.md) | Sending prompts or keystrokes into a long-lived interactive program in a tmux pane (`send-keys`) and needing to know it was consumed; a `capture-pane` before/after diff as delivery evidence; a payload that begins with `-` or comes from a variable; choosing between polling pane text and an out-of-band status artifact; a collapsed paste placeholder is not inside the fixed tail window you are searching | +| [sentinel-driven-repl-payloads](processes/sentinel-driven-repl-payloads.md) | Driving an external REPL/CLI (a browser-automation REPL channel, `node -i` fed via stdin) by writing payload lines and waiting for its own completion sentinel; every step reports `ok` with empty or wrong stdout; deciding whether to wrap an async payload in an IIFE or a top-level `await`; unit-testing the driver against a fake REPL that cannot reproduce the fast-empty-output failure | | [tool-diagnostics-without-a-failing-exit-code](processes/tool-diagnostics-without-a-failing-exit-code.md) | Wiring a compiler/linter/type-checker/validator into a hook, CI step, or agent loop so its complaints reach the author; the tool prints warnings to stderr but exits 0 so an exit-code-only wrapper reports success; choosing the redirection order that captures stderr without the build artifact; choosing which exit code and stream actually deliver text to the model; adopting a warnings-as-errors promotion (`-Werror`, `--strict`, `--max-warnings 0`) on a platform whose valid usage legitimately warns (deprecations, accept-and-warn declarations) | | [non-interactive-cli-invocation](processes/non-interactive-cli-invocation.md) | Calling a tool that can prompt (agent CLI, ssh, git, package manager) from a script, CI step, hook, or agent session, including with its own `-p`/`--print`/`--yes` flag; such a call produced no output and never returned; deciding whether a hang belongs to the client, the network, or the far-side service; choosing the stdin/timeout/fail-fast switches for an unattended call; a TTY-detecting tool changes its output format under automation; driving its interactive REPL with injected keystrokes (tmux `send-keys`/`expect`) and a pasted prompt never submits | diff --git a/wiki/platforms/processes/driving-a-tui-in-a-tmux-pane.md b/wiki/platforms/processes/driving-a-tui-in-a-tmux-pane.md index 96f67ee..406249d 100644 --- a/wiki/platforms/processes/driving-a-tui-in-a-tmux-pane.md +++ b/wiki/platforms/processes/driving-a-tui-in-a-tmux-pane.md @@ -9,7 +9,7 @@ sources: - https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap12.html - https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap11.html last_verified: 2026-08-25 -related: [platforms-processes-non-interactive-cli-invocation, platforms-processes-background-services, platforms-shells-portable-shell-scripts, infrastructure-agent-orchestration-pane-delivery-confirmation] +related: [platforms-processes-non-interactive-cli-invocation, platforms-processes-background-services, platforms-shells-portable-shell-scripts, infrastructure-agent-orchestration-pane-delivery-confirmation, platforms-processes-sentinel-driven-repl-payloads] --- # Sending Input to a TUI Running in a tmux Pane diff --git a/wiki/platforms/processes/non-interactive-cli-invocation.md b/wiki/platforms/processes/non-interactive-cli-invocation.md index 3073b8d..371b2d0 100644 --- a/wiki/platforms/processes/non-interactive-cli-invocation.md +++ b/wiki/platforms/processes/non-interactive-cli-invocation.md @@ -12,7 +12,7 @@ sources: - https://man7.org/linux/man-pages/man1/timeout.1.html - https://en.wikipedia.org/wiki/Bracketed-paste last_verified: 2026-08-04 -related: [platforms-processes-background-services, platforms-tools-bsd-vs-gnu-cli, platforms-shells-portable-shell-scripts, debugging-methodology-hypothesis-testing, platforms-processes-parsing-cli-structured-output, platforms-shells-env-var-off-switches, infrastructure-agent-orchestration-pane-delivery-confirmation, testing-strategy-signal-delivery-to-a-process-under-test, qa-deliverables-command-transcripts-in-a-document] +related: [platforms-processes-background-services, platforms-tools-bsd-vs-gnu-cli, platforms-shells-portable-shell-scripts, debugging-methodology-hypothesis-testing, platforms-processes-parsing-cli-structured-output, platforms-shells-env-var-off-switches, infrastructure-agent-orchestration-pane-delivery-confirmation, testing-strategy-signal-delivery-to-a-process-under-test, qa-deliverables-command-transcripts-in-a-document, platforms-processes-sentinel-driven-repl-payloads] --- # Invoking a Prompt-Capable CLI from a Script or Agent Harness diff --git a/wiki/platforms/processes/sentinel-driven-repl-payloads.md b/wiki/platforms/processes/sentinel-driven-repl-payloads.md new file mode 100644 index 0000000..ea3af86 --- /dev/null +++ b/wiki/platforms/processes/sentinel-driven-repl-payloads.md @@ -0,0 +1,68 @@ +--- +id: platforms-processes-sentinel-driven-repl-payloads +domain: platforms +category: processes +applies_to: [node, general] +confidence: verified +sources: + - https://nodejs.org/api/repl.html +last_verified: 2026-09-04 +related: [platforms-processes-driving-a-tui-in-a-tmux-pane, platforms-processes-non-interactive-cli-invocation, testing-async-async-testing, testing-quality-tests-that-cannot-fail] +--- + +# Driving an External REPL by Piped Payload Lines and a Completion Sentinel + +## When this applies + +Automating an external REPL or CLI (a browser-automation REPL channel, +`node -i` fed via stdin, any tool that evaluates one line and prints its own +ok/error marker) by writing JavaScript statements to its stdin and waiting for +that marker before treating a step as done. Also when every step reports +success with empty or wrong stdout, or a fake-REPL unit test cannot catch a +failure the real REPL exhibits. + +## Do this + +1. **Send the payload with a top-level `await` on the outermost expression**: + `await (async () => { ... })();`, rather than a bare async IIFE. A REPL with + top-level-await support suspends the whole line's evaluation on that + `await`, so the sentinel prints only once the awaited work finishes. +2. **After the sentinel, assert on the payload's own stdout or return value, + not only the ok/error marker.** The marker proves the REPL finished + evaluating the line, not that the async work inside it ran to completion. +3. **Choose the check by what the REPL exposes:** + +| REPL behavior | Do | +|---------------|----| +| Prints the expression's result on the same line as the sentinel | Assert that result is not an unresolved `Promise { }` | +| Only prints stdout/console output, no expression value | `console.log` an explicit marker inside the awaited path and require that exact string in captured stdout | +| Exposes a way to query real state (a `listTabs()`-style introspection call) | After the sentinel, make a separate call reading the state the payload was supposed to create, and assert on that | + +4. **When unit-testing the driving code itself, replay real REPL transcripts** + (both the unawaited-IIFE failure shape and the awaited success shape) + rather than canned stdout — a fake that always returns scripted "ok" output + can never exercise the fast-sentinel, no-output failure mode this page + describes ([testing-quality-tests-that-cannot-fail]). + +## Edge cases + +| Case | Then | +|------|------| +| The async work must run several dependent steps | Chain them with `await` inside one payload, or send one line per step and wait for each line's own sentinel — one awaited unit at a time, rather than several un-awaited calls fired at once | +| The target REPL has no top-level `await` | Wrap the payload as `void (async()=>{...})().then(()=>console.log(SENTINEL))` and gate on that printed sentinel instead of the REPL's own marker, since the marker still fires early | +| The payload's `await` throws | Node's REPL surfaces it as `Uncaught ` on the same line — treat that as the step's own failure, distinct from "no output" | +| Payload sent over `tmux send-keys` rather than piped stdin | Same consumption-vs-echo risk in a different shape → [platforms-processes-driving-a-tui-in-a-tmux-pane] | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Send `(async () => { await doThing(); console.log("done") })();` and wait for the sentinel | Send `await (async () => { await doThing(); console.log("done") })();` | The bare IIFE returns an unawaited promise; the REPL evaluates the call expression synchronously and prints its ok sentinel in milliseconds while the async body is still pending | +| Treat the ok/error sentinel as proof the step's side effect happened | Assert on the payload's own stdout or a follow-up state query | The sentinel reports the line finished evaluating, which for an unawaited promise happens before the async work completes | +| Unit-test the driver with a fake REPL replaying scripted stdout for each command | Replay real transcripts including the fast-empty-output failure shape | A canned-stdout fake cannot produce the exact failure this page exists to catch | + +## Sources + +- https://nodejs.org/api/repl.html — "Support for the `await` keyword is enabled at the top level" in the Node.js REPL; disabled via `--no-experimental-repl-await` +- Local reproduction 2026-09-04 (Node.js v26.7.0, macOS, `node -i` fed via piped stdin): `(async()=>{await new Promise(r=>setTimeout(r,300));console.log("done-unawaited")})();` followed by `.exit` printed only `Promise { }` — the 300 ms `console.log` never appeared; the same work as `await new Promise(r=>setTimeout(r,300)); console.log("done-awaited");` printed `done-awaited` before the REPL returned control +- Field evidence 2026-08-25 (measured in a linkly-crew orchestration run, browser-automation REPL channel): `(async () => { await openTab(url); console.log(...) })();` → ok sentinel in ~10 ms, no output, no tab in a follow-up tab-listing query; the same call with a leading `await` → output printed, ok sentinel in ~800 ms, tab present in the follow-up query diff --git a/wiki/platforms/shells/command-text-inspected-before-execution.md b/wiki/platforms/shells/command-text-inspected-before-execution.md index 45be5be..2b416ef 100644 --- a/wiki/platforms/shells/command-text-inspected-before-execution.md +++ b/wiki/platforms/shells/command-text-inspected-before-execution.md @@ -8,7 +8,7 @@ sources: - https://code.claude.com/docs/en/hooks - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html last_verified: 2026-08-06 -related: [platforms-shells-portable-shell-scripts, platforms-environment-path-resolution, platforms-shells-escapes-in-shell-string-literals, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts, platforms-tools-harness-mediated-tool-results, platforms-processes-tool-diagnostics-without-a-failing-exit-code, platforms-tools-agent-permission-classifier-denials] +related: [platforms-shells-portable-shell-scripts, platforms-environment-path-resolution, platforms-shells-escapes-in-shell-string-literals, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts, platforms-tools-harness-mediated-tool-results, platforms-processes-tool-diagnostics-without-a-failing-exit-code, platforms-tools-agent-permission-classifier-denials, testing-quality-gate-parsing-vs-command-execution] --- # Commands Read as Text by a Gate Before the Shell Runs Them diff --git a/wiki/platforms/tools/version-keyed-artifact-cache.md b/wiki/platforms/tools/version-keyed-artifact-cache.md index 740c28b..7ba2cb1 100644 --- a/wiki/platforms/tools/version-keyed-artifact-cache.md +++ b/wiki/platforms/tools/version-keyed-artifact-cache.md @@ -9,7 +9,8 @@ sources: - https://github.com/anthropics/claude-code/issues/45542 - https://github.com/anthropics/claude-code/issues/17361 - https://github.com/anthropics/claude-code/issues/61954 -last_verified: 2026-08-04 + - https://github.com/mattpocock/skills/blob/main/scripts/sync-plugin-version.mjs +last_verified: 2026-09-04 related: [platforms-toolchains-version-management, platforms-tools-plugin-mcp-server-registration, platforms-tools-unpacked-extension-source-reload] --- @@ -48,6 +49,21 @@ consumer's "update" to deliver the new code. The update runs, reports success or reinstall. Known updater bugs leave the cache stale even after a version bump; a manual clear is the reliable fallback. +5. **Prefer one manifest as the version's source of truth; gate the pair with + CI only if you keep both.** Claude Code reads `plugin.json`'s version and + ignores `marketplace.json`'s when both are set, so declaring it in one + place removes the drift risk entirely. If you keep both anyway — for + example so the marketplace listing shows a version without an install — + add a script with a `--check` mode that compares the two, matched by + plugin **name** (a self-referential marketplace's own entry has a git-URL + `source`, not a local path, so a path-keyed comparison resolves nothing), + and run it in CI on every push and before any automated release bump. + +| Case | Do | +|------|----| +| You control only `plugin.json` | Leave `version` unset in the marketplace entry; there is nothing left to drift | +| You need both fields populated (display, tooling) | Add a `--check`-mode sync/compare script and run it in CI; treat a mismatch or a missing matching entry as a build failure, not a warning | + ## Edge cases | Case | Then | @@ -71,3 +87,6 @@ consumer's "update" to deliver the new code. The update runs, reports success or - https://github.com/anthropics/claude-code/issues/17361 — cache never refreshes; Claude reads the stale cached copy even with autoUpdate - https://github.com/anthropics/claude-code/issues/61954 — `plugin update` reports "at latest" while the cache stays stale vs. a refreshed marketplace - Observed 2026-08-04: `~/.claude/plugins/cache/` holds per-version sibling dirs (`figma/2.2.81`, `2.2.87`, `2.2.88`; `dev-loop/0.8.0`…`0.11.0`), confirming the cache is keyed by version string +- https://code.claude.com/docs/en/plugin-marketplaces — "Avoid setting `version` in both `plugin.json` and the marketplace entry. Claude Code always uses the `plugin.json` value without warning, so a stale manifest version can mask a version you set in `marketplace.json`" — the single-source-of-truth alternative to gating +- https://github.com/mattpocock/skills/blob/main/scripts/sync-plugin-version.mjs — public reference implementation: `--check` mode compares `plugin.json`'s version against `package.json`'s and exits 1 with a fix-it message on drift, wired as `npm run check-plugin-version` +- Local observation 2026-09-04 (this repo, dev-loop): `scripts/check-versions.sh` implements this gate — matches `marketplace.json` entries to `plugin.json` by plugin name (not source path, since this marketplace's one entry is self-referential with a git-URL source), fails the build on any mismatch or on zero matching entries, and runs as a named "Version gate" step in `.github/workflows/test.yml` and inside `auto-release.yml`'s automated bump; the workflow step's comment records that "this pair has drifted unenforced before (a real release incident)" diff --git a/wiki/qa/index.md b/wiki/qa/index.md index 18a4e9a..f81daca 100644 --- a/wiki/qa/index.md +++ b/wiki/qa/index.md @@ -22,8 +22,9 @@ Match your situation to a "load when" line; load only matching pages. | [defect-class-resweep-after-review](process/defect-class-resweep-after-review.md) | Addressing review findings when the remediation itself adds code (a new function, branch, call site, formatter); about to hand that diff to a verifier, an audit, or CI as "review comments addressed"; deciding whether a bot's acknowledged/resolved thread means the class is closed; reporting fix completeness as a class-with-method count rather than a per-finding list | | [completion-claims](process/completion-claims.md) | About to report work as done, fixed, or passing — to a human, a coordinator, a PR, or a commit message; about to write "should work" or "tests pass" without a fresh run; deciding what evidence a completion claim requires; relaying a subagent's success report | | [evaluating-review-feedback](process/evaluating-review-feedback.md) | Review findings arrived (human, bot, or reviewer agent) and you are deciding what to implement; a finding is unclear; a reviewer proposes robustness or features nothing uses; you disagree with a finding and are deciding how to respond | -| [adversarial-change-review](process/adversarial-change-review.md) | Reviewing a diff ≥50 changed lines or one touching auth/payments/migrations/external APIs; a checklist review found nothing on a high-blast-radius change; deciding review depth for a risky diff; constructing failure scenarios (assumption violation, composition, cascade, abuse) | +| [adversarial-change-review](process/adversarial-change-review.md) | Reviewing a diff ≥50 changed lines or one touching auth/payments/migrations/external APIs; a checklist review found nothing on a high-blast-radius change; deciding review depth for a risky diff; constructing failure scenarios (assumption violation, composition, cascade, abuse); per-task lens reviews all passed on a diff that builds a command/query/path from user input through an allowlist | | [llm-review-pipelines](process/llm-review-pipelines.md) | Building or configuring an automated LLM code-review pipeline (CI review bot, review skill, PR-reviewer agent); review token cost grows with changeset size; deciding which stages run as deterministic code vs model judgment; keeping review rules as matchable data not prompt prose; model-written comments land on wrong lines; measuring a precision-vs-recall threshold change | +| [fresh-context-code-review](process/fresh-context-code-review.md) | Reviewing code, a plan, or a document an LLM session just produced; deciding whether the producing session, a repeated same-session pass, or a context-sharing subagent may serve as reviewer; designing which session an automated review stage dispatches to | ## deliverables diff --git a/wiki/qa/process/adversarial-change-review.md b/wiki/qa/process/adversarial-change-review.md index b4c5a59..ed5a589 100644 --- a/wiki/qa/process/adversarial-change-review.md +++ b/wiki/qa/process/adversarial-change-review.md @@ -6,7 +6,10 @@ applies_to: [general] confidence: field-tested sources: - https://github.com/EveryInc/compound-engineering-plugin -last_verified: 2026-08-22 + - https://cwe.mitre.org/data/definitions/88.html + - https://doc.rust-lang.org/cargo/commands/cargo-test.html + - https://doc.rust-lang.org/cargo/reference/config.html +last_verified: 2026-09-04 related: [qa-process-evaluating-review-feedback, qa-process-regression-scope, qa-process-post-release-verification] --- @@ -53,6 +56,7 @@ must go before merge. | A scenario needs prod-only conditions to trigger | Record it as a post-release monitoring item with the signal to watch ([qa-process-post-release-verification]) | | All four techniques return nothing on a high-risk diff | State the null result with what was searched (techniques × assumptions listed) — a stated null and an omitted one read the same to the merge decision, so state it | | The diff is high-risk and also very large | Partition by risk surface (auth paths first, then money paths) and run the techniques per partition, so depth lands where the blast radius is | +| Per-task lens reviews all passed on a diff that builds a command, query, path, or argv from user input through an allowlist | Task an integration-level review explicitly with technique 4 (abuse cases): construct a string that satisfies the allowlist while doing something the allowlist should have forbidden. The coordinator, not the reviewer, then reproduces the candidate string directly against the real validator before treating it as a finding — a contract gap (e.g. no rule for a trailing or extra token) passes every task-scoped review because each task correctly implemented its own contract; construction-and-reproduction, not reading the code, is what surfaces the gap | ## Instead of @@ -60,7 +64,14 @@ must go before merge. |---------------------|-----------------|-----| | Clear a payment/auth diff with a style-and-correctness checklist alone | Run the four techniques at the depth the table sets | Checklists find known single-point patterns; constructed scenarios find interactions between correct-looking parts | | Report "potential race condition" as a finding | Construct the interleaving: which two operations, which shared state, what wrong outcome | An unconstructed finding cannot be verified, prioritized, or fixed | +| Accept "all task-scoped lenses passed" as clearance for an allowlist, argv-builder, or path-assembly merge | Task the integration review with constructing a bypass string, then reproduce that exact string yourself against the real validator | Task review checks "built to contract"; an allowlist's gap is a hole in the contract itself, which a faithful per-task implementation still carries and which construction plus reproduction exposes | +| Hand a reviewer's proposed bypass string straight to a worker as a confirmed defect | Reproduce it yourself first; reject examples that fail reproduction | A reviewer's attack string can itself be wrong (e.g. it violates the sink's own syntax) — un-reproduced, it sends a worker chasing a threat that does not exist | ## Sources - https://github.com/EveryInc/compound-engineering-plugin — adversarial-reviewer persona: depth calibration by size/risk, the four scenario-construction techniques, scenario-oriented finding titles, advisory-to-human routing; field-tested in the plugin's shipped review workflow +- https://cwe.mitre.org/data/definitions/88.html — CWE-88, "Improper Neutralization of Argument Delimiters in a Command ('Argument Injection')": untrusted argument-delimiting input gains the command unintended arguments +- https://doc.rust-lang.org/cargo/commands/cargo-test.html — `--manifest-path` selects the package defined by that manifest, so an out-of-repo manifest path redirects package selection there +- https://doc.rust-lang.org/cargo/reference/build-scripts.html — Cargo compiles a package's build script "just before a package is built" and "will then run the script" — `cargo test` runs `build.rs` for the selected package +- https://doc.rust-lang.org/cargo/reference/config.html — `--config KEY=VALUE`: "The argument should be in TOML syntax"; an unquoted bare value is not valid TOML, which is why a naive second bypass attempt using an unquoted path fails reproduction +- Field evidence 2026-08-31 (measured in a linkly-crew orchestration run): after 4/4 task-scoped lens reviews passed, an integration review proposed `cargo test --manifest-path=`; the coordinator reproduced it directly — the string passed the allowlist's `vet()`, exited 0 (definition-of-done "passed"), and ran the out-of-repo `build.rs`. The reviewer's second example, `--config build.rustc=/path`, failed reproduction (TOML requires quoting the value; the allowed character set excludes `"`) and was rejected rather than escalated diff --git a/wiki/qa/process/fresh-context-code-review.md b/wiki/qa/process/fresh-context-code-review.md new file mode 100644 index 0000000..9aaea29 --- /dev/null +++ b/wiki/qa/process/fresh-context-code-review.md @@ -0,0 +1,69 @@ +--- +id: qa-process-fresh-context-code-review +domain: qa +category: process +applies_to: [general] +confidence: verified +sources: + - https://arxiv.org/abs/2603.12123 +last_verified: 2026-09-04 +related: [qa-process-llm-review-pipelines, qa-process-evaluating-review-feedback, qa-process-adversarial-change-review, infrastructure-agent-orchestration-session-context-token-budget] +--- + +# Reviewing LLM-Produced Work From a Session That Did Not Produce It + +## When this applies + +Reviewing code, a plan, or a document an LLM session just produced, and +deciding whether the producing session (or a subagent spawned from it) may +also serve as the reviewer; about to ask a session to "double check your +work"; designing which session or subagent an automated review stage +dispatches to. + +## Do this + +1. **Run the review in a session that never received the production + conversation** — a fresh session, or a subagent invoked with only the + artifact (diff, plan, document) and the review instructions, not the + reasoning trail that produced it. arXiv:2603.12123 ("Cross-Context + Review", CCR) measured this at 28.6% F1 across 360 reviews over 150 seeded + errors in 30 artifacts, against 24.6% F1 for same-session self-review + (p=0.008). +2. **Buy the gain with isolation, not repetition.** Asking the same session + to review twice (SR2) scored 21.7% F1, not significantly different from a + single same-session pass (p=0.11). One fresh-context pass replaces any + number of same-session re-reads. +3. **Give a subagent reviewer the artifact only.** A subagent that inherited + the producing session's context (SA) scored 23.8% F1, statistically on par + with same-session review — the subagent boundary by itself buys nothing; + the missing production context is what does. +4. **Return only the verdict to the original session**, not the reviewer's + reasoning trail, so a correction loop does not re-mix the production + context back into the next round's review. +5. **Budget one extra session, not a pipeline.** The paper's method "requires + no special infrastructure and costs only one additional session." This + repo's own `test-quality-auditor`, `integration-reviewer`, and + `plan-reviewer` subagents already implement it — each exists "so the + session that wrote the code does not grade its own tests" / to review + "from a fresh context the coordinator's own session never reaches." + +## Edge cases + +| Case | Then | +|------|------| +| The reviewer needs shared background (repo layout, conventions) but not the production reasoning | Give it the artifact and repo access, not the transcript — [infrastructure-agent-orchestration-session-context-token-budget] already isolates subagent context for cost; the same isolation carries this page's accuracy gain | +| The review stage is a deterministic CI bot, not a chat session | Isolation still applies — [qa-process-llm-review-pipelines] bundles files into isolated review units; keep the producing agent's own narrated reasoning (commit-message chain-of-thought, plan rationale) out of the unit's input | +| The change is high-risk (auth, payments, migration) | Weight fresh-context review highest here: CCR's gain concentrated on critical errors (+11 percentage points) and code artifacts (+4.7 F1 points), consistent with [qa-process-adversarial-change-review]'s depth-by-risk table | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Ask the session that wrote the code or plan to "double-check your work" | Dispatch a fresh session or artifact-only subagent | Same-session review scored 24.6% F1 vs 28.6% for a fresh session on the same 150-seeded-error benchmark (p=0.008) | +| Ask the producing session to review twice for extra confidence | Run one fresh-context review | Repeated same-session review (21.7% F1) is not significantly better than one pass (p=0.11) | +| Hand review to a subagent spawned with the same conversation | Spawn the subagent with only the artifact, not the production transcript | Context-sharing subagents scored 23.8% F1 — statistically on par with same-session review | + +## Sources + +- https://arxiv.org/abs/2603.12123 — Song, "Cross-Context Review: Improving LLM Output Quality by Separating Production and Review Sessions" (Mar 2026): 30 artifacts, 150 seeded errors, 360 reviews; abstract reads "CCR reached an F1 of 28.6%, outperforming SR (24.6%, p=0.008, d=0.52), SR2 (21.7%, p<0.001, d=0.72), and SA (23.8%, p=0.004, d=0.57)"; SR2 vs SR not significant (p=0.11); gains concentrated on critical errors (+11pp) and code artifacts (+4.7 F1) +- Local reproduction 2026-09-04 (this repo, `agents/test-quality-auditor.md`, `agents/integration-reviewer.md`, `agents/plan-reviewer.md`): each is a fresh-context reviewer invoked "so the session that wrote the code does not grade its own tests" / "from a fresh context the coordinator's own session never reaches" — a production instance of the paper's mechanism diff --git a/wiki/security/agent-exposure/in-session-tool-exposure.md b/wiki/security/agent-exposure/in-session-tool-exposure.md index 4d4bcbb..f15888b 100644 --- a/wiki/security/agent-exposure/in-session-tool-exposure.md +++ b/wiki/security/agent-exposure/in-session-tool-exposure.md @@ -8,7 +8,7 @@ sources: - https://developer.chrome.com/docs/ai/webmcp/secure-tools - https://webmachinelearning.github.io/webmcp/ last_verified: 2026-08-18 -related: [frontend-agent-interfaces-agent-facing-tool-surfaces, security-input-validation-at-trust-boundaries, security-agent-exposure-authorization-scope-persistence] +related: [frontend-agent-interfaces-agent-facing-tool-surfaces, security-input-validation-at-trust-boundaries, security-agent-exposure-authorization-scope-persistence, testing-quality-gate-parsing-vs-command-execution] --- # Exposing Executable Tools to an Agent in a User's Session diff --git a/wiki/testing/async/async-testing.md b/wiki/testing/async/async-testing.md index 49b91e1..dec3aec 100644 --- a/wiki/testing/async/async-testing.md +++ b/wiki/testing/async/async-testing.md @@ -11,7 +11,7 @@ sources: - https://testing-library.com/docs/dom-testing-library/api-async/ - https://martinfowler.com/articles/nonDeterminism.html last_verified: 2026-08-29 -related: [testing-quality-tests-that-cannot-fail, testing-flaky-diagnosing-flaky-tests, testing-data-test-data-and-isolation, testing-quality-injected-clock-duration-assertions] +related: [testing-quality-tests-that-cannot-fail, testing-flaky-diagnosing-flaky-tests, testing-data-test-data-and-isolation, testing-quality-injected-clock-duration-assertions, platforms-processes-sentinel-driven-repl-payloads] --- # Testing Asynchronous Code Deterministically diff --git a/wiki/testing/index.md b/wiki/testing/index.md index 4e65ac9..28b1fd4 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -40,13 +40,14 @@ Match your situation to a "load when" line; load only matching pages. | [behavior-not-implementation](quality/behavior-not-implementation.md) | Deciding what a test should assert; a behavior-preserving refactor broke tests; tempted to expose privates for testing; deciding whether a snapshot test is appropriate; a value-preserving move of a literal into config needs a test that detects re-inlining without asserting an internal | | [tests-that-cannot-fail](quality/tests-that-cannot-fail.md) | Reviewing tests that always pass; a bug shipped through an area the suite reported as covered; auditing a suspiciously green suite; judging whether an assertion, error-path test, or mock-based test can actually detect a defect; a shell-test (bats) assertion passes mid-test on code it should fail | | [checks-that-cannot-pass](quality/checks-that-cannot-pass.md) | Authoring a check whose target does not exist yet (grep/regex gate on an unwritten file or doc section, lint/scan rule, schema assertion on an unbuilt endpoint, a plan's verification command) and it has only ever been observed failing; reviewing a plan's gates before adopting them; separating "target missing" from "content missing" in a gate's exit status. This owns unwritten-target gate patterns for spec/RFC documents too — qa/document-verification does not | +| [gate-parsing-vs-command-execution](quality/gate-parsing-vs-command-execution.md) | Writing or reviewing a gate script that reads a document/plan containing command strings and deciding whether the parser may run them directly (`sh -c`) or must hand them to a dedicated timeout-bounded executor; proving a parser does not execute embedded commands | | [spec-artifact-checks](quality/spec-artifact-checks.md) | Authoring or reviewing the check itself: that a mapping table covers every rule/field/enum case, that ids resolve across documents; deciding whether a green check earned "verified" or only "present"; designing one negative control per check in a multi-check harness; parsing Markdown table rows programmatically in a doc-as-spec repo (deciding whether a passing gate is enough to *accept the deliverable* → wiki/qa/document-verification/spec-document-gates.md) | | [schema-additions-under-a-golden-gate](quality/schema-additions-under-a-golden-gate.md) | Adding a node kind, variant, discriminator value, or field to a document format (IR, JSON Schema, spec artifact) whose only automated gate builds its negatives by mutating one committed golden example; the gate or the whole suite comes back green right after a schema change; deciding which negative each new schema keyword needs, and whether a green suite that never loads the schema is evidence at all | | [policy-at-several-return-sites](quality/policy-at-several-return-sites.md) | One function applies the same policy at more than one of its own success returns (a CLI handler computing an exit code under `--strict`/`--check` and returning it from several branches, a controller stamping one header on several 200s); judging whether a suite covers such a flag when only one green test exists; choosing test cases by exit path rather than by input boundary; proving each site with a reversion of that site alone | | [generated-sql-property-assertions](quality/generated-sql-property-assertions.md) | Asserting on the SQL a query builder renders (`.toSQL()`, `sqlToQuery()`, a compiled-string snapshot) because CI has no database, and the property is semantic rather than textual ("a missing value must stay missing", "the tenant filter must stay applied"); such an assertion is green while a hand-seeded mutant that refills the missing value survives; choosing between top-level structural anchors, a function-name check, an aggregate-occurrence count, and a captured-alias binding — and knowing which refill position each one owns | | [harness-reverse-controls](quality/harness-reverse-controls.md) | You built a harness that scores how well something is verified (mutation run, doc/spec gate suite, CI check matrix) and are about to cite its score in a commit, PR, README, or report; its verdicts come out uniform (every case caught, or every case green); deciding what control run proves the harness discriminates, how to score errored/never-ran cases, and what the harness's isolated working tree must contain | | [value-preserving-refactor-assertions](quality/value-preserving-refactor-assertions.md) | Adding or reviewing the regression test for a refactor that replaced a hardcoded literal with a read from config/constants/SSOT while the value stayed the same; deciding what such a test can assert when the rendered output is byte-identical before and after; choosing a sentinel and the seam to substitute it through, and how to restore it; deciding when the value is too private for a test and the guard belongs in a static check instead; a sentinel assertion is red on code you believe correct | -| [unasserted-return-fields](quality/unasserted-return-fields.md) | Testing a function that returns a composite (dict/record/tuple) of computed fields while assertions read only some of them; judging whether a large assertion count covers such a return; auditing which fields no assertion mentions and confirming it by mutation; asserting invariants that bind fields to each other (`lo ≤ point ≤ hi`, parts sum to total) and choosing between an input grid and generated inputs | +| [unasserted-return-fields](quality/unasserted-return-fields.md) | Testing a function that returns a composite (dict/record/tuple) of computed fields while assertions read only some of them; judging whether a large assertion count covers such a return; auditing which fields no assertion mentions and confirming it by mutation; asserting invariants that bind fields to each other (`lo ≤ point ≤ hi`, parts sum to total) and choosing between an input grid and generated inputs; a mock/simulated source emits a sequence of timed events and only each event's field existence is asserted, not the delivery-order relation between events' timestamps | | [stale-artifact-baselines](quality/stale-artifact-baselines.md) | Using a previously published output file (JSON/CSV export, approved snapshot) as the "before" baseline for measuring a code change's impact; its totals match the current run and you are about to treat the comparison as clean; dating an artifact's generation from its schema fields; rebuilding a before side by reverting only the change under measurement; reporting previously published numbers beside a new delta | | [surviving-mutant-equivalence-triage](quality/surviving-mutant-equivalence-triage.md) | A mutation run (PIT, Stryker, or a hand-seeded mutation) left a mutant alive on code you own and you are deciding what to change; a reviewer asks for a test covering a specific surviving mutant; a defensive branch carries a comment explaining why it is needed and its mutant survives; separating a missing test from an equivalent mutant from an uncovered line | | [mutation-harness-file-custody](quality/mutation-harness-file-custody.md) | You wrote a script that backs up, mutates and restores source files rather than using PIT/Stryker; choosing how it keys backups and how it proves a restore; reading a `git diff`/`grep` of a tree while such a harness runs; a mutant verdict flipped after a restore bug was fixed; same-named files in different directories (`routers/deal.ts` vs `schemas/deal.ts`) | @@ -68,7 +69,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [destructive-operations-on-shared-daemons](mocking/destructive-operations-on-shared-daemons.md) | The code under test enumerates and deletes a machine-wide daemon's resources by name/pattern (tmux sessions, docker containers, systemd units, namespaces) and that daemon runs on the test machine; proving a sweep deletes the targets and spares bystanders; keeping a scope bug from destroying the dev environment instead of failing the test; giving a shell script a substitution seam for the tool it shells out to | -| [what-to-mock](mocking/what-to-mock.md) | Deciding whether to mock/stub/fake a dependency or use the real one; mocks breaking on refactors; testing handling of a third-party's failure modes; the same mock setup is copy-pasted across tests | +| [what-to-mock](mocking/what-to-mock.md) | Deciding whether to mock/stub/fake a dependency or use the real one; mocks breaking on refactors; testing handling of a third-party's failure modes; the same mock setup is copy-pasted across tests; a mock/simulated source replays several events over an interval and must decide when to stamp each event's timestamp | | [captured-call-arguments](mocking/captured-call-arguments.md) | Writing the spy/stub test that holds a fix to one argument of one wiring call (constructor, factory, server startup); such a test is green while a mutation of a *different* argument of the same call survives; the fix extracted the value into a resolver and you are choosing what to assert; deciding between asserting a constant's value and asserting that the call site passes it on; choosing how to record an argument you deliberately leave unpinned | ## flaky diff --git a/wiki/testing/mocking/what-to-mock.md b/wiki/testing/mocking/what-to-mock.md index 17f7fa2..82a9955 100644 --- a/wiki/testing/mocking/what-to-mock.md +++ b/wiki/testing/mocking/what-to-mock.md @@ -10,7 +10,7 @@ sources: - https://abseil.io/resources/swe-book/html/ch12.html - https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import - https://nodejs.org/api/test.html -last_verified: 2026-08-06 +last_verified: 2026-09-04 related: [testing-strategy-test-level-choice, testing-quality-behavior-not-implementation, testing-quality-tests-that-cannot-fail, testing-mocking-captured-call-arguments] --- @@ -32,6 +32,7 @@ where mocks are breaking on refactors. | Pure in-process collaborators you own (domain objects, helpers, services without I/O) | Use the real objects — replacing them tests your guess of your own code | | External I/O you don't control: third-party HTTP APIs, payment providers, email/SMS gateways | Stub/mock at the boundary **you own** — your client wrapper interface — with canned responses per case (success, error, timeout) | | Nondeterminism sources: clock, randomness, UUID generation | Inject them and substitute a fixed clock / seeded generator in tests | +| A mock or simulated source that replays several events over an interval (event-stream/SSE mock, scripted fixture timeline) | Stamp each event's timestamp immediately before delivery, inside the per-event emit/`setTimeout` callback, rather than once when the whole event array is built; a build-time stamp gives every event the same instant regardless of when it is actually delivered | | Your own DB, when the test's subject is query behavior (SQL shape, mapping, constraints) | Real test database — a mocked DB asserts your assumption of the contract, not the contract ([testing-strategy-test-level-choice]) | | A dependency that is real-capable but too slow/stateful for every unit test (your DB behind a repository, a queue) | An in-memory **fake** implementing the same interface, kept honest by running the contract's own integration tests against the real one | | Command sent to an external boundary is itself the behavior (charge card, publish event) | Mock the owned boundary interface and assert the **outbound contract**: which command, with what arguments — not internal call sequences leading up to it. Assert the recorded call whole, so no argument of it stays unchecked ([testing-mocking-captured-call-arguments]) | @@ -73,3 +74,4 @@ where mocks are breaking on refactors. - https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import — imported bindings are read-only live bindings; reassignment by the importer throws - https://nodejs.org/api/test.html — `mock.module()` requires the `--experimental-test-module-mocks` CLI flag - Field evidence 2026-08-06 (ESM CLI, `node:test`; the two ESM/tripwire rows are field-tested on top of the doc-verified mechanism): boundary test passed with `PATH=""` while a real `spawnSync("npm")` under the same emptied PATH returned ENOENT; 365/365 suite green +- Field evidence 2026-08-28 (measured in a linkly-crew orchestration run, commit `530e7e2`): a mock event source computed its entire replay array — including each event's `ts` — in one synchronous build pass, so injecting a fixed or mocked clock at test time could not have separated correct from broken behavior; the defect was in *when* the source's own stamping ran relative to delivery, not in whether the clock was real or injected ([testing-quality-unasserted-return-fields] carries the assertion-side fix) diff --git a/wiki/testing/quality/gate-parsing-vs-command-execution.md b/wiki/testing/quality/gate-parsing-vs-command-execution.md new file mode 100644 index 0000000..4b7ce17 --- /dev/null +++ b/wiki/testing/quality/gate-parsing-vs-command-execution.md @@ -0,0 +1,74 @@ +--- +id: testing-quality-gate-parsing-vs-command-execution +domain: testing +category: quality +applies_to: [general] +confidence: verified +sources: + - https://cwe.mitre.org/data/definitions/78.html + - https://man7.org/linux/man-pages/man1/timeout.1.html +last_verified: 2026-09-04 +related: [testing-quality-checks-that-cannot-pass, qa-document-verification-spec-document-gates, platforms-shells-command-text-inspected-before-execution, security-agent-exposure-in-session-tool-exposure] +--- + +# Separating a Document/Plan Parser From the Command Executor It Feeds + +## When this applies + +Writing or reviewing a gate script that reads a document or plan file +containing command strings (a plan's verification command, a runbook's repro +command, a spec's example invocation), and deciding whether the parser may +run those strings directly (`sh -c "$cmd"`) or must hand them to a separate +executor. + +## Do this + +1. **Keep the parser's job to two questions**: does the target exist, and + can it be parsed. A parser that also runs `sh -c` on a string it just + extracted doubles as an untrusted-command executor — any document author + (including an LLM producing the plan) gains a code-execution surface the + moment their prose is treated this way (CWE-78: "constructing operating + system commands using externally-controlled input without properly + neutralizing special characters"). +2. **Route every command that must actually run through one dedicated + executor**, bounded by a timeout and required to record its evidence + (exit code plus matched output) in a durable ledger line — one executor, + rather than `sh -c` calls scattered across parsing code. `timeout(1)`'s + contract is the shape to copy: run the command, kill it when it outruns + its budget, and surface a distinct status (124) for "timed out" versus the + command's own exit code; a bare `sh -c "$cmd"` with no timeout hangs the + whole gate on one bad line. +3. **Prove the non-execution property with a fixture, not a code read.** + Write a fixture document whose embedded command would fail loudly if run + (`CHECK: false`, or a command naming a path that does not exist), feed it + to the parser-only path, and require it to still report `ok`. A fixture + whose "would-fail" command changes the parser's own verdict proves the + parser executed it. +4. **When parser and executor are one binary in two modes** + (`--status` / `--run`), state the mode boundary in the usage banner and + put the timeout only inside the executing mode, so a future edit cannot + move execution into the read-only mode without showing up as a one-line + diff of that path. + +## Edge cases + +| Case | Then | +|------|------| +| The document embeds a command as prose or example, not as a thing to run | Keep it out of any field the parser scans for `CHECK`/executable markers — [platforms-shells-command-text-inspected-before-execution] covers passing such text through a non-executing channel | +| The same script must support both a fast pre-flight parse and a full execution pass | Split into two invocation modes on one binary (`--status` vs `--run`), not two conditionals inside one code path — a caller that only wants existence/parseability should never be one flag away from executing | +| A gate has never been observed passing (unwritten target) | Validate the check's own correctness per [testing-quality-checks-that-cannot-pass] as well — that page's known-good/mutated-input proof and this page's non-execution proof are different properties of the same script | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Call `sh -c "$cmd"` inside code that parses a plan or document for existence and structure | Extract the command into a ledger line and hand it to a dedicated executor with its own timeout | Parsing and executing are different trust levels; merging them turns every document the parser reads into a command-execution surface (CWE-78) | +| Trust a code read ("the parser has no `sh -c`, so it is safe") as proof of non-execution | Add a fixture with a deliberately failing command and require the parser-only path to return ok on it | A read can miss a call buried behind a helper; the fixture is evidence, the read is a hypothesis | +| Let a slow or hanging embedded command stall the whole gate | Bound every executed command with a timeout and a distinct timed-out exit status | An unbounded `sh -c` on a bad line blocks the harness indefinitely, indistinguishable from a hang elsewhere in the pipeline | + +## Sources + +- https://cwe.mitre.org/data/definitions/78.html — CWE-78, "Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection')": constructing OS commands from externally-controlled input without neutralizing special elements +- https://man7.org/linux/man-pages/man1/timeout.1.html — "run a command with a time limit"; exit status 124 "if COMMAND times out", distinct from the command's own exit status +- Local reproduction 2026-09-04 (this repo, `skills/loop-implement/scripts/gate-check.sh`): the usage banner documents `--status` as "parse + report; never executes CHECK" versus `--run`, which executes runnable gates' CHECK bounded by `GATE_CHECK_TIMEOUT` seconds (default 120) via a perl alarm and records EVIDENCE per gate — a production instance of the split this page recommends +- Field evidence 2026-09-02 (measured in a linkly-crew orchestration run, review t1-plan-gate-r1): a plan-gate script's direct `sh -c` execution of a document-embedded command was flagged in review; the repair commit removed the `sh -c` call and added a non-execution test (a fixture whose command would fail, returning `ok` from the parser), re-reviewed at 87/87 green diff --git a/wiki/testing/quality/unasserted-return-fields.md b/wiki/testing/quality/unasserted-return-fields.md index ab857f8..a471ffc 100644 --- a/wiki/testing/quality/unasserted-return-fields.md +++ b/wiki/testing/quality/unasserted-return-fields.md @@ -10,8 +10,8 @@ sources: - https://arxiv.org/abs/2211.12003 - https://hypothesis.works/articles/what-is-property-based-testing/ - https://abseil.io/resources/swe-book/html/ch12.html -last_verified: 2026-08-06 -related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, testing-quality-harness-reverse-controls, testing-quality-write-path-assertions, testing-quality-value-preserving-refactor-assertions, testing-quality-expectation-sets-with-one-distinct-value] +last_verified: 2026-09-04 +related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, testing-quality-harness-reverse-controls, testing-quality-write-path-assertions, testing-quality-value-preserving-refactor-assertions, testing-quality-expectation-sets-with-one-distinct-value, testing-mocking-what-to-mock] --- # Return Fields No Assertion Reads @@ -85,6 +85,7 @@ count for such a function and you are judging whether that count is coverage. | Two fields are computed by the same expression | Assert each field's value independently, and either drop the relation between them or record it as non-discriminating with that reason — a relation between two copies of one expression cannot fail on a defect that moves both, so keeping it unlabelled adds an assertion that cannot fail | | The invariant is violated on inputs the caller cannot produce | Fix the field's domain guard or narrow the grid to reachable inputs, and state which it was — a violation on unreachable input is not a bug in the formula | | The composite is returned across a boundary you own (HTTP/JSON) | Assert the fields on the deserialized response, not the internal object, so the serialization is covered too ([testing-quality-write-path-assertions]) | +| The composite is one of many objects emitted over time by a mock or simulated source (a replayed event stream, an SSE mock, a scripted fixture timeline) rather than a single call's return | Assert the temporal relation across the sequence — timestamps strictly increase in delivery order — in addition to each event's own field existence; a source that stamps every event at build time rather than at emit time passes an existence-only assertion while every event carries the identical value | ## Instead of @@ -103,3 +104,4 @@ count for such a function and you are judging whether that count is coverage. - https://hypothesis.works/articles/what-is-property-based-testing/ — property-based testing as "the construction of tests such that, when these tests are fuzzed, failures in the test reveal problems with the system under test that could not have been revealed by direct fuzzing of that system" — why an invariant plus generated inputs finds what per-case assertions miss - https://abseil.io/resources/swe-book/html/ch12.html — test the behaviors (guarantees) a unit makes; a cross-field invariant is one such guarantee and needs its own assertion - Field reproduction 2026-08-05 (manday estimation engine): the suite held 58 passing assertions over a function returning `lo`/`sp`/`hi`; `lo` and `hi` appeared in none of them. Four mutations of their formulas (percentile `p25`→`p75`, `p75`→`p25`, scope factor large→small) each left all 58 assertions passing, while the no-op control survived — confirming the harness discriminated. Adding `lo ≤ sp ≤ hi` over the full discrete input grid surfaced 13 combinations with `sp > hi` +- Field evidence 2026-08-28 (measured in a linkly-crew orchestration run, commit `530e7e2` "stamp MockEventSource ts at delivery, not build time"): a mock event source's `buildScenario()` stamped every event's `ts` synchronously at build time, so 30–60 replayed events showed an identical timestamp instead of progressing over a ~10 s replay; the existing 63-test suite asserted only that `ts` was present, never that it progressed. The fix left `ts` as a placeholder at build time and stamped the real clock in the per-event delivery callback immediately before each event fired; the added regression test asserts two delivered events' `ts` values differ and increase in delivery order, and was confirmed to fail against the pre-fix code before merging