docs: record the run that proves the paging fix, and de-rot the backlog header - #155
Open
oratis wants to merge 2 commits into
Open
docs: record the run that proves the paging fix, and de-rot the backlog header#155oratis wants to merge 2 commits into
oratis wants to merge 2 commits into
Conversation
#154 said "verified against production" without saying what was run, which leaves the next person to either trust it or re-derive it. This records the actual measurement. The number that matters is the wall clock, not the id count. A capped run on 2026-08-21 walked 300,000 ids in 640s across 3 page boundaries and exited 0. 640s is past the 600s `total_timeout_millis` that killed the single-stream version, so that run IS the regression test — the old code could not have reached the end of it at any throughput. Worth stating explicitly, because it is the part that is easy to get wrong when re-checking: reproducing this needs a connection slow enough for 300k ids to take ten minutes. A fast in-region runner finishes 300k well inside the budget and therefore proves nothing, so a green in-region run is not evidence the bug is fixed. Also pins the home-connection throughput (~470-510 ids/s) the existing note referred to only as "far less", giving the ~28x gap and both full-sweep estimates (~23 min in-region, ~11 h from a laptop). Docstring only; no behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pending section's header said "Currently empty (all seeded), so the daily cron idles until you add more". That was written in 6aaee6b on 2026-07-05, when the backlog genuinely was drained. suggest-topics began filling the queue five days later in 4d1b6f5, and the line has been wrong ever since — about six weeks, across a stretch when the queue held 13 topics and a reader checking whether the cron had work to do would have concluded it did not. Replaced with the invariant rather than a snapshot: where new topics go, that auto-author drafts from the top, and that suggest-topics keeps it fed. A count in a comment next to the thing it counts will always rot; the note now says explicitly that its absence is deliberate. Second, subtler reason to reword: the old line spelled `status: 'pending' }` exactly as a real entry does, so it was itself matched by greps meant to count queued topics. That produced a real off-by-one during this week's pipeline review — the backlog was reported as holding 13 pending when it held 12. The replacement does not contain the literal, and `grep -c "status: 'pending' }"` now agrees with the entry count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two documentation corrections. No behaviour change in either.
1. Record the run that proves #154's paging fix
#154 said "verified against production" without saying what was run. That leaves the next person to either trust it or re-derive it — so this records the actual measurement in the docstring.
Against the real 18,968,589-doc collection, via
--max=300000(which refuses--apply, so nothing was written —_meta/sitemap_shardsis still absent and the job stays disarmed).The number that matters is the wall clock, not the id count. 640s is past the 600s
total_timeout_millisthat killed the single-stream version, so this run is the regression test: the old code could not have reached the end of it at any throughput.The corollary is the part that's easy to get wrong when re-checking, and it's now stated explicitly: reproducing this needs a connection slow enough for 300k ids to take ten minutes. A fast in-region runner finishes 300k well inside the budget and therefore proves nothing — a green in-region run is not evidence the bug is fixed.
Also pins the home-connection throughput the existing note called only "far less": ~470–510 ids/s vs ~13,700 in-region, putting a full sweep at ~23 min in-region against ~11 h from a laptop.
2. Drop the stale "currently empty" backlog header
The pending section said "Currently empty (all seeded), so the daily cron idles until you add more." True when written (
6aaee6b, 2026-07-05, backlog drained).suggest-topicsbegan filling the queue five days later (4d1b6f5, 2026-07-10), and it has been wrong for the ~six weeks since — including a stretch where the queue held 13 topics and a reader checking whether the cron had work would have concluded it didn't.Replaced with the invariant instead of a snapshot, and the note now says its lack of a count is deliberate — a count sitting next to the thing it counts will always rot.
Second, subtler reason to reword. The old line spelled
status: 'pending' }exactly as a real entry does, so it was itself matched by greps meant to count queued topics. That caused a real off-by-one during this week's pipeline review — the backlog was reported as 13 pending when it held 12. After this:tscclean · eslint clean (1 pre-existing warning) · 18/18 suites pass🤖 Generated with Claude Code