Skip to content

docs: record the run that proves the paging fix, and de-rot the backlog header - #155

Open
oratis wants to merge 2 commits into
mainfrom
docs/sitemap-shards-verification
Open

docs: record the run that proves the paging fix, and de-rot the backlog header#155
oratis wants to merge 2 commits into
mainfrom
docs/sitemap-shards-verification

Conversation

@oratis

@oratis oratis commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Two documentation corrections. No behaviour change in either.


1. Record the run that proves #154's paging fix

#154 said "verified against production" without saying what was run. That leaves the next person to either trust it or re-derive it — so this records the actual measurement in the docstring.

… 100,000 scanned,  50 shards, 1 pages, 196s (511/s)
… 200,000 scanned, 100 shards, 2 pages, 418s (478/s)
… 300,000 scanned, 150 shards, 3 pages, 640s (469/s)
▸ 300,000 docs → 149 shards in 640s (3 pages)          [exit 0]

Against the real 18,968,589-doc collection, via --max=300000 (which refuses --apply, so nothing was written — _meta/sitemap_shards is still absent and the job stays disarmed).

The number that matters is the wall clock, not the id count. 640s is past the 600s total_timeout_millis that killed the single-stream version, so this run is the regression test: the old code could not have reached the end of it at any throughput.

The corollary is the part that's easy to get wrong when re-checking, and it's now stated explicitly: reproducing this needs a connection slow enough for 300k ids to take ten minutes. A fast in-region runner finishes 300k well inside the budget and therefore proves nothing — a green in-region run is not evidence the bug is fixed.

Also pins the home-connection throughput the existing note called only "far less": ~470–510 ids/s vs ~13,700 in-region, putting a full sweep at ~23 min in-region against ~11 h from a laptop.


2. Drop the stale "currently empty" backlog header

The pending section said "Currently empty (all seeded), so the daily cron idles until you add more." True when written (6aaee6b, 2026-07-05, backlog drained). suggest-topics began filling the queue five days later (4d1b6f5, 2026-07-10), and it has been wrong for the ~six weeks since — including a stretch where the queue held 13 topics and a reader checking whether the cron had work would have concluded it didn't.

Replaced with the invariant instead of a snapshot, and the note now says its lack of a count is deliberate — a count sitting next to the thing it counts will always rot.

Second, subtler reason to reword. The old line spelled status: 'pending' } exactly as a real entry does, so it was itself matched by greps meant to count queued topics. That caused a real off-by-one during this week's pipeline review — the backlog was reported as 13 pending when it held 12. After this:

grep -c "status: 'pending' }"  → 15
real entries                   → 15     # now agree

tsc clean · eslint clean (1 pre-existing warning) · 18/18 suites pass

🤖 Generated with Claude Code

oratis and others added 2 commits August 21, 2026 23:11
#154 said "verified against production" without saying what was run, which
leaves the next person to either trust it or re-derive it. This records the
actual measurement.

The number that matters is the wall clock, not the id count. A capped run on
2026-08-21 walked 300,000 ids in 640s across 3 page boundaries and exited 0.
640s is past the 600s `total_timeout_millis` that killed the single-stream
version, so that run IS the regression test — the old code could not have
reached the end of it at any throughput.

Worth stating explicitly, because it is the part that is easy to get wrong when
re-checking: reproducing this needs a connection slow enough for 300k ids to
take ten minutes. A fast in-region runner finishes 300k well inside the budget
and therefore proves nothing, so a green in-region run is not evidence the bug
is fixed.

Also pins the home-connection throughput (~470-510 ids/s) the existing note
referred to only as "far less", giving the ~28x gap and both full-sweep
estimates (~23 min in-region, ~11 h from a laptop).

Docstring only; no behaviour change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pending section's header said "Currently empty (all seeded), so the daily
cron idles until you add more". That was written in 6aaee6b on 2026-07-05, when
the backlog genuinely was drained. suggest-topics began filling the queue five
days later in 4d1b6f5, and the line has been wrong ever since — about six weeks,
across a stretch when the queue held 13 topics and a reader checking whether the
cron had work to do would have concluded it did not.

Replaced with the invariant rather than a snapshot: where new topics go, that
auto-author drafts from the top, and that suggest-topics keeps it fed. A count
in a comment next to the thing it counts will always rot; the note now says
explicitly that its absence is deliberate.

Second, subtler reason to reword: the old line spelled `status: 'pending' }`
exactly as a real entry does, so it was itself matched by greps meant to count
queued topics. That produced a real off-by-one during this week's pipeline
review — the backlog was reported as holding 13 pending when it held 12. The
replacement does not contain the literal, and `grep -c "status: 'pending' }"`
now agrees with the entry count.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oratis oratis changed the title docs(sitemap): record the run that proves the paging fix docs: record the run that proves the paging fix, and de-rot the backlog header Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant