Fix the open bug list, and the two milestones whose remaining work was code - #479
Open
tamnd wants to merge 62 commits into
Open
Fix the open bug list, and the two milestones whose remaining work was code#479tamnd wants to merge 62 commits into
tamnd wants to merge 62 commits into
Conversation
…471) Eight sections of the Vietnamese corpus held "Unusual activity has been detected from your device. Try again later." and a request id as their whole body, under a full set of headers whose source hash matched. Every measure the corpus has called them finished translations, so nothing was ever going to look at them again. The detection was not the gap. textguard already reads those lines and the audit already reported them, as gateway and refusal leaks under RuleCommentary. The gap was the waiver: -raw waives commentary, because a model that announced its translation did still translate, and the same waiver let a rate limit page through. So the two provider kinds move to a rule of their own, RuleRefusal, which takeRaw leaves alone the way it leaves transport alone. A complaint of that rule is not an answer the audit disliked and could be fixed later; it is not an answer at all. Narration stays where it was and stays waivable.
marksOf already said what to do with it, that "the section sign and the dagger are misread pilcrows and mark what the pilcrow marks", and exNumRE already carried the control word \S a text layer writes. Only the sign itself was missing, from both. So "**§ 4.**" was not a marker, exercises() went on looking for a 4 it would never find, and the rest of the section was appended to the body of exercise 3. § 7 of chapter I of Algebra reported 3 exercises where the volume prints 40, and its third exercise file is 34043 bytes against 236 and 439 for the two in front of it. Nothing was lost when a § stalled, only misfiled, which is why this is a re-split and wants no PDF. A section heading spells a § and a number too, and three things keep this off them. itemStart takes a marker only where it carries the number the § is up to. The heading of a § is not inside the block exercises() reads. And a cross reference on a line of its own, "§ 4.3", cannot match, because the pattern wants a space, a bold or a dollar after the full stop and a reference puts a digit there. What is left is 21 historical notes listing their own contents, and those reach the pattern only through itemOpen, whose two callers use it to refuse to glue two blocks together. A table of contents is not the second half of a sentence broken over a page, so refusing is right anyway. Checked by reassembling all 44 books against what is committed. 29 sections gain, none loses, 279 exercises come back out of the bodies they were glued into. 14 of the 29 land exactly on the count the other printing of the same book assembles on its own. The largest disagreement left is top IX § 5, which comes out at 30 against the French 7, and the French is the one that is wrong: the tail it glued still holds markers numbered 5 through 30.
… are The same misreading as the dagger, one letter further. A page image read as a picture gives the pilcrow as a capital, and where it does it gives it that way throughout: every page in the corpus that opens an exercise with one of these four carries no pilcrow anywhere on it, so the letter is the only thing left to read the mark off. They are selective the way a mark is and a numbering is not. Page 109 of Espaces vectoriels topologiques in French runs "T 27) 28) T 29) 30) T 31)", three of five. And the other printing agrees: of the 37 exercises marked this way that the other language also holds as an exercise of its own, 28 are starred there, which is the other reading of the same page finding a pilcrow. The 9 that are not do not disagree, they sit on pages carrying no exercise mark of any kind. The letter must have a space after it. That is what tells a mark from a label, and the corpus writes "C1.", "S2.", "A1." and "E1)" 307 times with no space and never means a mark. A is left out although it has the space: its only four uses are in the proof of § 3 of chapter V of Commutative Algebra in French, numbering the cases of an argument, the second opening "Cas général". Those four are the whole of the capital-and-space form outside the four letters taken here, so the set is the corpus counted rather than a guess at what a model might do. Checked by reassembling all 44 books. 24 sections gain, none loses, 259 exercises come back out of the bodies they were glued into. 15 of the 24 land exactly on the count the other printing assembles on its own, and two more agree once both languages are written: § 1 of Algebra VI was 34 against 20 and is 36 in both, § 4 of Algebra VII was 9 against 8 and is 25 in both. S11, which reports an exercise one printing has and the other lacks, falls from 694 findings over 94 sections to 508 over 77.
sentenceEnd held a page reference and little else, so every other shape of cross reference the books print read as a marker. "(IX, p. 112, exerc. 7)" started the seventh exercise of the section it was cited in, and § 5 of chapter IX of Topologie generale in French came out with seven exercises against the thirty the English has, three of them halves of a sentence cut at a citation and two of those a body of one full stop. Every abbreviation added was counted in the corpus first, in the one shape that can do the damage, the citation closing on its own number, and every distinct match of the small ones was read: prop 13090, exerc 5595, th 4445, cor 2180, def 1440, Exer 680, Ex 234, mod 82, rem 6, App 6, Sect 4. None of them can cost a real marker, because a marker at the head of a block is matched at offset zero and never reaches sentenceEnd, and a marker set inside a paragraph directly behind the word "prop." is not a thing the books print. Taking the false markers away left two sections with nothing at all, both of them held together only by a citation. One is a marker form the pattern did not allow: extraction writes the mark and the number inside one span of bold and the bracket outside it, "**T 1**)", and § 3 of chapter IV of Topological Vector Spaces opens on it. The corpus holds seven of that form and three are the first exercise of their section. The other is a reading that dropped three numbers off a page and is put right in the corpus.
content/ is generated from pages/, and assembly is a pure function of the pages. A correction made in the section rather than in the page it came from therefore survives exactly until the next assemble of that book, and is then replaced without a word. That had reached 80 files across 21 books before anything reported it, and the only way it was ever found was by diffing the tree after every run. So a run that would replace a committed file with text it does not reproduce now stops and names every one of them, and says where the correction belongs. -overwrite is the way to say that assembly is right and the file is not, which is what a genuine re-extraction wants. A run that reproduces what is there, or writes a section for the first time, is unchanged and costs nothing extra. Fixes #478.
…alled Coverage keyed on path.Base, and a filename is the title slugged, so a translation is counted against the English manifest under a name it can never have: 01_s1_laws_of_composition.md and 01_s1_luat_hop_thanh.md are one § under two spellings. The two printings also slug differently within one language. A lane built from the French pages reported 8 of 30 against a manifest that had all thirty. The key is now the chapter, the kind of file, and the § number, none of which is a matter of spelling. The path is still what the message names, since that is what a reader has to open. Fixes #472.
A build over twenty thousand pages takes ten minutes and returned on the first file it could not render, so learning about two unclosed math spans in one volume cost two full passes to be told two line numbers, and the two were the same fault twice. The build now carries on, writes every page it can, keeps the list, and the command prints all of it and exits non-zero. A failure to write still returns at once: a full disk is not a finding about the corpus. Also stops the informational line claiming a site is not publishable when the run is about to exit 0. Whether marked formulae are enough to stop a build is what -max-broken says, and it says it three lines further down. Fixes #446. Fixes #449.
Elements d'histoire des mathematiques and the French Varietes differentielles are continuously paginated and each declares the single chapter "1", and neither prints a chapter numeral anywhere. So the fit found no opener, named no row, and chapterSpans looked for rows saying "1" and handed back nothing: the chapter column came out empty on all 374 rows of the one and all 190 of the other, and anyone running pagemap build with no -book would have overwritten two maps that had been pinned by hand. The corpus is keyed on that column, content/fr/hist/1/ being where the sections live, so an empty one is a loss. Where the declaration names one chapter and no row can be fitted to it, the body is now named the way a volume declaring no chapters at all is named: one run per fascicule, numbered from one in the order they are bound. That gives the declaration back on a volume of one run and keeps the span each on the Varietes, which is two fascicules and prints its page 6 twice. The steps and the conflicts take the name of the row they stand on, since they were recorded before the rows had one and Validate compares them against it. All 44 maps rebuild, all 44 validate, and the only rows that move are the ones that gain a chapter: hist, hist-fr and var-fr reproduce their committed page and confidence columns byte for byte. Fixes #475.
A refusal that arrives as an HTTP error, or as an empty body, is caught on the write path and the file is left alone. A refusal that arrives as a well formed answer whose text happens to be an error message is written with full front matter, a translation_model, a translation_run and a source_content_sha256, and from then on nothing can tell it from a translation that worked. It survives -stale and -redo-small, and no translate run will ever visit it again: it is current and its source hash matches. Ten were in the corpus and they were found because somebody was reading one of them for an unrelated reason. So the check has to be a scan of what is committed rather than a flag on translate. textguard already holds the phrases, since the same answers reach the reading side. What is checked here is the three kinds that are the provider talking instead of the model working, and an empty body, which the same fault produces. The kinds textguard keeps for a model narrating its work are left alone: those are sentences a translation of mathematical prose can honestly contain, and this rule is hard. Fixes #471.
Counting delimiters can only see mathematics that was set as mathematics. A page whose reader dropped the dollars altogether balances at zero and rule 2 had nothing to say about it, so the mathematics went into the corpus as text and nothing downstream noticed. checkBareTeX walks the page tracking inline and display state and reports a backslash followed by letters when it is outside both. It runs last, after the two delimiter checks, so an unclosed dollar is still reported as the dollar rather than as confused span state. Escaped characters are stepped over, which is what keeps a \$ from opening a span and a backslash at the end of a line from eating the newline the line count is made of. A markdown escape is a backslash and one character that is not a letter and is never reported. Over all 44 volumes this reports 594 commands, and every one of them is mathematics: \in 120 times, \leq 48, \mathbf 39, \alpha 28, \otimes 27, and a tail of Greek letters and operators one and two at a time. Not one text-mode command is left in the list. Getting there needed the allow-list in proseTeX. The footnote three are #448, which records them as the conventions the pages carry and leaves the choice between them open; reporting them now would put 71 pages in the rejected column for a decision nobody has made. The rest is the corpus setting text, which is what those commands are for: \textsuperscript in 54 places, nearly all French ordinals, \emph in 41, nearly all \emph{ibid.} and \emph{loc. cit.}, \textbf inside a footnote holding a journal volume number, and \v three times, every one of them Stone-\v{C}ech. Page 67 of the French Varietes differentielles, which the issue names, is not among the findings and should not be. Its 38 control sequences are all inside spans; the count in the issue is of every command on the page rather than of the bare ones. The ten real cases in that volume are pages 100, 102, 135, 143, 156, 158, 160, 170, 172 and 173. Fixes #444
The note was keyed by volume id, so the second printing of a Book
inherited none of the notation the first was given. Theory of Sets is
read twice and only ens-i-iv got ocr_ens.md; ens-i-iv-fr was read with
the shared prompt and nothing else for all 349 of its pages. The
front matter says so plainly: 344 of the 349 carry prompt hash
97b470c1, which is the hash of the shared prompt with no note on it,
the same one alg-i-iii carries.
What it cost is in the counts. The printing that was told the script T
"is never \mathcal" wrote \mathcal{T} zero times and \mathscr{T} 843;
the printing that was not told wrote \mathcal{T} 738 times and
\mathscr{T} 41. The note calls the pilcrow common and it was found 249
times against 47 over a comparable number of pages. \square against
\Box goes 26/0 against 10/5. The rows the shared prompt already covers
track each other, which is what says the difference is the note.
So bookNote is keyed by the Book the manifest already carries, and
ocr_ens.md holds what is true of the Book: the Hilbert operator, the
criteria labels, the disjunction sign, the removed-letter square, the
pilcrow, the script T.
Two of its fourteen lines were about the printing and each is wrong for
the other printing rather than merely absent from it, so they move to
volumeNote as ocr_ens_en.md and ocr_ens_fr.md. The English heads a
section with a bare number, the French with a § and the number, both
checked against the pages: content of the French volume has "## § 4.
RÉUNION ET INTERSECTION D'UNE FAMILLE D'ENSEMBLES" where the English
has "## 4. UNION, INTERSECTION, PRODUCT". Telling the French reader not
to add a sign the page does not print would have it drop one that is
printed. And the English is foot-number where the French is head-label
with an E I.15 style label in the head, so the sentence saying to leave
the folio at the foot would send the French reader to the wrong edge of
the page. 24 of that volume's 44 current rejections are the head rule.
Both printings go stale, which is the point of the change rather than a
side effect: they were read against a prompt missing rules that apply
to them. ens-i-iv already carries five distinct prompt hashes across
its pages, so this is not a new kind of cost for it.
OCRFor and OCRAnything now take the volume and the Book. A caller with
only the volume can pass an empty Book and gets the volume note alone.
Fixes #476
There was no way to ask for a re-read of a page the runner calls alreadyRead. The three doors that existed are doors for something else: a changed image, a changed prompt, or a rejected page a stronger reader should take. -reread-protected only reaches needsABetterReader, which is a state a page can only be in after it has been rejected, and -unread narrows the set rather than widening it. A page that passes every rule was unreachable and stayed unreachable however wrong it was. state was right that written is not the same as read. What it assumed is the converse, that a page which passes the rules was read correctly. The rules are structural: they see a broken heading, a loose control sequence, an unbalanced delimiter. They cannot see a page that is well formed and wrong. alg-x-fr is the case that has been parked. 163 of its 222 pages came off the weakest reader in the fleet and 59 off gpt-5, every page passes, so every page was alreadyRead and Fill skipped all 222. Page 65 has running_head RÉSOLUTIONS where every neighbouring page of the chapter says ALGÈBRE HOMOLOGIQUE, because the reader filed the section title as the running head, and no page_label at all. Both wrong, neither visible to a rule. outranked stays in force and that is what makes this safe. The 142 pages lost before were lost to a weaker reader writing over a stronger one, and outranked stops that whatever put the page in the queue. Run against alg-x-fr the 163 weak readings queue and the 59 gpt-5 ones are held and counted, which is the behaviour the case wants and not a special case written for it. The queue needed telling too. The job id is the content address of the work, so a page read at this image and this prompt is a job the queue has already done and Add refuses one of those; without a reset the flag would say the page is work and then queue nothing. Reset is the door translate -force already uses and it keeps the id, so the record of what the last answer was and which host gave it stays on the file. The flag refuses to run without -f and -l. The pages it puts back already passed, and the cost of pointing it at a whole volume by accident is a volume of fleet time. Fixes #477
A run of remarks or examples came apart into one statement per member only where the printing put a blank line between them. cutRunMember was already there for the case where it did not, and it could never see the one that matters: it is asked before the block is read, when next is still 0, because the run is not open until the block carrying its head has been read as its first member. So a page setting "Remarques. — 1) ... 2) ... 3) ..." gave one statement and members 2 and 3 went into the body of the first, with no tag, no reference target and no translation unit of their own. The cut is made again after the first member has been read, out of the body rather than out of the block, because by then the head has been taken off and the body is what the member is. It looks for one number and not for any number, which is what keeps "voir 2) ci-dessus" in the body of member 5 out of it, and the line has to open on the marker. Over the 44 volumes this is 254 statements, and every one of them is a Remark or an Example: 60 Remarque 2, 49 Exemple 2, 28 Exemple 3, 24 Remarque 3, 19 Remark 2, 19 Example 2 and a tail. No other kind moves, 34 previously unnumbered Remarks and Examples become numbered members of their run, and 288 minus 34 is the 254. M01 stays at 7 and P04 stays clean, so nothing was cut through a formula. no. 11 of § 5 of chapter IV of Integration is the case the issue names. The French printing sets Remarque 1 and then "2) La topologie induite par la topologie de la convergence en mesure ..." on the line under it, and only Remarque 1 came out. Eight of the members open on the star that brackets a passage in small type, and splitting them dropped it, five of those closing the passage on the same line and so leaving the star at the far end without its pair. The head branch has put that mark back on the body since it was written; the member branch never did, because it never had to before. It does now, and it needed the escaped spelling: smallTypeOpen knows $*$ and $^*$, which is how a printing that sets the mark as mathematics writes it in a head, and every one of the eight members writes it \*, escaped so that Markdown does not read it as emphasis. closingMarks had to learn the same spelling for the same reason. That puts the mark back on 67 bodies, the 8 from the split and 59 that had been losing it quietly. Fixes #445
reports/eval.json is the current estimate of the judges, the number the scorecard quotes and the one M8 is judged on. solve eval -write put whatever the run had in hand into it, and a run that is stopped has part of a set in hand. A five hour run was stopped after putting 68 questions to the fleet with 20 answered, and what it left read as a clean eval: 4 outcomes, 2 accepts, 2 rejects, a false accept rate of 0 and a false reject rate of 0, against the committed 20 outcomes with 4 false rejects in 10. That is the exit criterion passing on a fifth of the questions, and nothing in the file said it was a fragment. The committed copy came back only because it is tracked. Two changes. A run now records Of, the number of cases it set out to put, against the outcomes it has, so Partial is arithmetic rather than the absence of a field; a run written before Of existed has it at zero and reads as whole, which is right, because those are the complete ones. And -write refuses a partial rather than writing it, saying how many of how many it got and pointing at -json, which prints the same run and puts the fragment where the caller asked for it instead of over the number everything else reads. Save also writes beside the file and renames, the way writeOCRReport does, so a run killed while it saves leaves the last good result rather than half of a new one. The eval runs for hours against a fleet answering well under half of what it is asked, which is the condition a run gets abandoned in. The scorecard says so too. A partial should not reach it now, but the file can be put there by hand and the scorecard is what quotes it, so it reports a part of a set as a part of a set. Fixes #400
Validate catches a map that contradicts itself, and a map fitted from one anchor contradicts nothing. top-v-x is 372 pages, foot-number, scanned with no text layer, and the reading kept the foot number on 2 of them. The offset from those carried across the other 364 validates clean, gets written, and from then on the volume reads as mapped: the coverage table counts its pages and every page the fitter got wrong is quoted back by everything that asks. A volume that steps anywhere at all, which most of them do, is wrong from the step to the end and nothing says so. The file was being kept out of the corpus by hand. Two measures, because the obvious one is not enough on its own. The share of body pages whose number was read says how much evidence there is; the longest single run of pages carried by arithmetic says where it is, and ten anchors spread evenly are worth more than ten in the first chapter for the same share. Both thresholds come from the 44 maps rather than from a guess. Share read is 0.5% for top-v-x and then nothing until 15.3% for lie-i-fr and var-fr, so the floor is 5%, the round number in the middle of an empty gap. The longest unread run is 73% of the body for top-v-x and then nothing until 26.5% for alg-ix-fr and 24.3% for lie-i-fr, so the cap is half the volume. Neither is tight: this stops a map being published that is not a fit at all, and the coverage table is where a map that is thin but honest gets reported. Neither share is asked below 20 body pages, because a share over a handful of pages is not a measurement. The smallest volume in the corpus is the 61 body pages of lie-vii-viii-fr. Run over all 44 volumes, exactly one is refused and it is top-v-x, on both counts. Nothing else moves. The cure for that volume is a folio read rather than a check that catches it, which is its own thread. Fixes #403
When an OCR reading replaces a native one the script capitals go.
\mathscr{T} comes back as \mathcal{T} and the corpus spells one printed
glyph two ways. Page 76 of Algebra VIII is the shape of it: thirty
script capitals in the native reading, thirty calligraphic in the one
that replaced it, and nothing else about the mathematics changed. Over
the 48 pages of the two born digital volumes rewritten in one morning,
9 swapped the face.
The general rule that the run reports a face and does not correct it is
right and stays. This is a narrower claim standing beside it: on a born
digital volume the extractor reads the font name out of the PDF and the
model guesses a typeface from pixels at 300 dpi, so the text layer is
not merely right more often, it has evidence the model does not have at
all.
Narrowed four ways. Only against a native reading, which is the
condition carry already computes. Only that pair of faces. Only that
direction. And only for a letter the native reading set in script and
did not also set calligraphic, so a page that really did distinguish
the two is left alone. Everything the OCR reading is better at is kept,
including the 17 fuller running heads recovered in the same pass.
Whether any page legitimately prints a calligraphic capital distinct
from a script one was the thing to check first, and the corpus says no.
139 pages use both commands and on 137 of them no letter takes both
faces, which is two different letters and not a distinction. The two
that remain are both OCR readings spelling the same letter both ways
within the page, \mathcal{C} nine times against \mathscr{C} four on one
and \mathcal{F} four against \mathscr{F} one on the other. That is one
reader being inconsistent, and neither page is native, so neither is
reachable from here.
mathcal joins the faces the report counts. Without it this swap read as
thirty losses of \mathscr{T} with nothing gaining them, a report that
says a face went and cannot say where.
The 48 pages already written this way are a separate sweep.
Fixes #332
A reader can collapse distinct letters onto one command and nothing
downstream can tell. olmOCR emitted \mathcal{G} 237 times over 25 pages
of top-v-x-fr with three different Fraktur capitals underneath it: F
for the filter in the chapter VII exercise on rearranging a series, G
for its partner in the Baire hierarchy of chapter IX, and S for the
S-convergence chapter X is about. The TeX compiles, the audit passes,
and the book is wrong about which set system it is discussing.
No substitution repairs that, which is why this reports and does not
correct. Page 228 of that volume carries Fraktur S and Fraktur F in the
same paragraph and the reader mapped the first to \mathcal{G} and the
second to \mathcal{F}, so the mapping is not consistent inside one page.
fvr-i-vii-fr has the mirror image, \mathfrak{g} where the printing has
Fraktur F, 72 times over 9 pages with the interleaved pages correct.
What made that findable was the ratio and the isolation: 237 times in
one volume and nowhere else in the corpus. So the books column counts
how many volumes write the same face and argument, and -alone keeps the
ones no other volume uses. Over the whole corpus that is five lines:
ac-x-fr \mathfrak{y} 13 on 2 pages
top-i-iv-fr \mathcal{Z} 6 on 1
top-i-iv-fr \mathcal{CV} 2 on 1
lie-i-fr \mathfrak{st} 1 on 1
top-v-x-fr \mathcal{CX} 1 on 1
against 1056 face and argument pairs in the corpus. The two-letter ones
read like a complement C\mathcal{V} that lost its brace rather than a
letter the printing has, which is the kind of thing this is for.
The argument is taken whole and not a letter at a time, because
\mathfrak{su} is a thing the books set. Every volume is read even when
-book asks for one, since the books column cannot be counted from a
single volume.
Fixes #466
A twenty six chunk section cost 377 asks. All twenty six were answered by one route inside nineteen minutes and the file was still unwritten thirty five minutes later: the surplus rows in the usage log were two chunks sent over and over to a second route, every one refused. That route was not broken. Nine of its ten accounts were in a rate limit cooldown and the tenth was locked by a live solve, so it had nothing free and said so at once, and the router picked it anyway, because the host list is a preference order and a host that refuses is still a host. The lane that retires itself when the provider never reads the question has been here all along. It was not enough, and the reason is the shape of the run rather than the shape of the failure: lanes are started inside translateFile, which is called per file, so a route retired while one file was being translated was started again for the next one and the one after. So the breaker is built once in runTranslate and passed down. Three refusals running put a route aside for twenty minutes, after which it is let back to try once: a route whose cooldowns have expired answers that probe and is back with its count cleared, one whose have not refuses it and is put away again, so being wrong costs one ask every twenty minutes rather than every file. Only the refusals where the question was never read count, which transportOnly already tells apart, and any answer at all clears the count. Nothing is lost when a route goes: the chunk is released with its attempts intact for whichever route is still answering. The count is not gated on outOfTurns, because the route that cost the 377 was not out of turns in any of the three wordings that test knows. It had accounts, all of them were busy, and it said so in its own words. Two smaller things the same measurement wanted. The run summary now says how many asks it cost, which is the number that says whether a run went well and is not the number of chunks: 377 against 26 looks like every other run in every other line the command prints, and finding it out meant counting rows in the usage log. And a chunk asked more than six times in one run says so on the line as it happens, six being two asks a lease times three leases, the most a chunk that is merely difficult can cost. The test that carries it translates two files through one breaker against a route that answers nothing, and measures the second file's ask count. With the retirement neutered it is 1 and the test fails, which is the bug in miniature.
… ask Two things out of the morning that wrote 199 exercises and not one section. -deadline is per ask. The usage said so and the only bound the command offered was still that one, so a runner written on the belief that it bounded the call was not being careless: a section is 25 to 50 chunks, and at the longest -deadline accepts, one file is up to sixteen hours in a call that prints nothing until it returns. Two lanes sat seventy two minutes inside their first file while the supervisor above them cleared their leases twice for looking dead. An exercise is one chunk, which is why exercises looked healthy: for them the two numbers are the same. So -within, which stops the run when the time is up and reports what was written. It costs only the chunks in flight, because every answered chunk is recorded done in the queue with its answer beside it long before the file is assembled, and the next run reads those back and asks for the rest. A run that spends the time it was given is not an error and does not return one; an interrupt still is, because the two are different things to whatever is above it. The usage now says which bound is which and what a section costs, since that is the sentence whose absence cost the day. And the chunk name. ChunkChars and ChunkSpans decide where a section is cut, so loosening either of them re-cuts every unfinished section and chunk 7 becomes a different passage carrying the same number. The queue keys a job on a hash of the body and readAccepted compares that hash before it takes an answer, so both of those miss and ask again, which is right and already has a test. chunkID was the one name left keyed on the number alone, and it names the scratch directory the question is asked in, so a re-split would have put the new chunk 7 where the old one was asked and read back whatever was lying there. It carries six digits of the body now. A re-split should miss, not mis-pair. What is not done here is raising the chunk size, which is the thing those two changes were wanted for. That needs the refusal rate measured against the new size and is its own piece of work.
Every box in the pool was a headed Chrome under xvfb-run against a signed in ChatGPT profile for long enough that nothing here asked what a host was. "Can this host read" and "does this host have a browser and an account pool" were the same question, and CanOCR asked the second one. A machine with a card came into the pool. It serves its own weights and reads page images against them, it has no browser, and it has never had an account signed into it because it does not need one. Asked the browser question it answers no on every count. The ban board printed "no signed in profile, so this host can read nothing" about the host that reads the most pages in the fleet, and CanOCR gated the only box with a GPU on an Xvfb it will never start. Lanes was worse than wrong: it divided by browser profile memory, so a reader with 900 MB free and forty gigabytes of disk got zero lanes while holding the weights it was told it could not use. So Kind is a field on a host and both the gate and the report branch on it. CanOCR splits: canBrowse is the old body unchanged, canRead wants the reader program, rsync, somewhere to put the images, and a model server that answered on the endpoint the route names. Lanes for a reader is disk, which is what actually bounds it. The probe asks about the endpoint only of a host that has one, because a curl to nowhere costs five seconds on every probe of every other box. fleet accounts grows two more row shapes. A reader row says which model it is serving and whether the endpoint answered, which is the fact somebody deciding where to send a page needs; printing zeroes for it in the account columns would be true and misleading. And a host that timed out is its own row, because it is a different state from a host that answered and has nothing ready, and it used to have its transport error printed into a table of counts. Wait learns the same thing. A reader that is answering means there is somewhere to send a page now, so it ends the sleep; a reader that is down has no cooldown to contribute and is not counted either way. Without that the sweep would wait out a browser cooldown with the fastest reader in the fleet idle. The zero kind is a browser, so every caller that predates readers is judged exactly as before. Fixes #474
A head erratum reached the page text and nothing else, and the page text is not where the number is on a labelled volume. readAnchors takes the label out of the front matter over any head on the page, so a mangled label is not overruled by a sound head beside it. It is preferred to it, and then thrown out by the filter that keeps only the volume's own Book prefix, and the page is left with no anchor at all. Writing a head erratum for such a page corrected a head nobody was going to read. Algebre commutative chapitre 10 is the volume, seven times over. Its verso head reads AC X.42 and then the chapter title, and the reader cut it in the wrong place on pdf 41, 63, 73, 125, 131, 139 and 171. It put the A into the running head, where it survives as the lone Greek capital lambda in front of PROFONDEUR, and wrote C X.42 as the label. C is not the prefix the volume prints, so all seven labels went in the bin, and the heads they were preferred to carry the chapter title and no number. None of the seven had the wrong number in the end, because 119 other labels agree on the offset and the fit interpolates across them. What the loss cost was the evidence: seven pages the printing puts a number on that the map had to call interpolated. With the errata applied the volume goes from 124 labels to 131 and from 71.5% of its body printed on the page to 75.4%, and not one printed page number moves. An erratum now applies wherever its says is found, on the label or in the page, and is an error only when it is found in neither, on the same reasoning the check has always had: a correction nobody applied is a person having believed it was in force. A says that is on the page more than once is still an error even where the label carried it too, since the head it means is then undecided. Fixes #402.
…olds fleet accounts read 21 verified profiles with 10 of them ready and idle, which is a table saying there is spare capacity. There was none. Three extra translate lanes went at those ten ready profiles and wrote nothing at all, and the reason was in the run logs and nowhere else: 995 asks across the fleet had ended "stopped without writing an answer: browser: ... ERROR ChatGPT never accepted the prompt", 720 of them on that host and 262 on another. The session loads, the profile counts as verified and ready, the prompt is composed, and the composer never takes it. Each one costs the whole deadline before the lane gives up, so a profile in that state is worse than an absent one: it looks available, gets leased, and burns twenty minutes. So the outcome of every ask is written down against the host it went to, in ~/.config/bourbaki/asks.json beside the fleet state, and the board reads it back as a taking column: answered out of recently asked. Where the record says most of the recent asks never reached a composer, the last column says that instead of saying the host is free now, since the last column is what somebody deciding where to send a page reads, and the run prints a line saying the profiles want signing in again rather than more lanes. Two failures that turn up beside it are honest and are counted apart: all verified slots busy is a box that really is full, and a usage limit is a quota that comes back on its own. Neither says the host cannot take a prompt, and a table that lumped them in would send the fleet away from the hosts it has. A dash is a host nothing has asked lately, which is not the same as a host that failed everything. It is per host and not per profile, and it has to be. The account table prints the email address of every slot and none of it is kept, on purpose, so there is no key to hang a profile on and no honest way to say which of the ten ready slots is the one that will not compose. Nothing here gates. The run level breaker is what actually stops asking a host that has stopped answering, because that is measured inside the run it protects; this is a record of runs already over, and a record that refused a host outright would have no way back, since the only evidence that a host is composing again is an ask that was answered. The window is two hours, shorter than the eight hour cooldown a rate limited profile sits out, so a host cannot be condemned on a ban that has since lifted, and a verdict wants five asks, because two failures in a row is a bad minute and every host has those. Two runs on this laptop at once is ordinary, so a run merges its notes into the file rather than writing over it. Only the translate driver notes its asks so far. Wiring ocr run and fleet ask into the same record is worth doing and is not here. Fixes #447.
bourbaki audit -only M01 took a minute, and that is why nobody used it. M01 is the rule that finds an unclosed math span, and it has been there and correct the whole time. People went on finding those with publish -check instead, which walks the corpus for over ten minutes and stops at the first fault, so two of them cost two full runs and forty minutes to learn two line numbers. The reason was not that a scan was missing. It was that asking took long enough that the file the question was about had been closed again. Timed stage by stage over the corpus, a full load is 67 seconds and 63.5 of them are refs.Build. Every manifest and every Markdown body of every language together is 3.2. So the graph is built only when the run has a rule in it that reads one, which is the References group. Asked by group rather than by naming R01 to R03, so a fourth reference rule is loaded for by being registered. bourbaki audit -only M01 59.7s -> 2.9s Nothing else changes. A run with no -only takes every rule and so builds the graph, which is what a full audit, bourbaki report and CI all do, and the two places outside the R rules that read Corpus.Refs, the report's summary line and its JSON, already test it for nil and leave the reference counts out when it is absent. That is the honest answer: a run narrowed to one rule counted no references. The other half of the issue, the message naming the end of the file rather than the line the delimiter sits on, was already repaired: publish/markdown.go and book/latex.go both report r.at(unclosed.Line) now. And the two other loose scanners it lists are rules as well, L07, L11 and L16 for a paragraph left in the source language and L13 for a word in an alphabet the target does not use. All three are now cheap enough to run. Fixes #473
The round robin driver rendered a window of pages, called ocr run over that
range, swept the images and wrote last+1 into a cursor file per book. It stopped
when every cursor was past its book's last page.
A window that died part way through advanced the cursor anyway. lie-iv-vi-fr
read 99 pages of a 240 page window and ended on "context canceled", and the
pages in that window that were never leased stayed pending in the queue for
ever, because the cursor had gone past them and the driver never came back.
Counted against the queue that was 320 pages sitting pending behind their book's
cursor, 130 of them in hist, between pages 1 and 299 of a volume that read as
finished.
The cursor should not exist. The queue already knows which pages of a volume
have not been read, and the only reason to window at all is that this laptop
cannot hold a whole volume of images at 300 dpi, which is a rendering constraint
and not a reading one. So:
bourbaki ocr run -book B -window 240
renders the next 240 pages the queue is still waiting on, reads them, sweeps
their images and asks the queue again, until the volume has none left. A window
that dies leaves its pages exactly where a window that never ran would have left
them, and the next one takes them. That is one call per book with no state
outside the queue.
A page is outstanding when the queue holds an unfinished job for it, pending or
leased, or when the corpus holds no page file for it at all, which is what Fill
finds on a page nothing has ever queued. Dead is deliberately not outstanding:
the queue gave up on those and queue retry is the door back, and a run that
swept them up would refuse the same pages until it was killed. A page also gets
one window per invocation, which is what makes the loop end; the driver's answer
to that was a cap of three sweeps, which is the same idea with a number on it.
Three things underneath:
- Runner.Only bounds leasing to the exact pages of the window. A range cannot,
because a window is not contiguous and the pages between its pages have had
their images swept, so leasing one sends a batch at a file that is not there.
- writeBlanks is held to the pages a render named. The manifest it walks is
the merged one, so without that a windowed run would walk every page file of
the volume once per window.
- ocrSetupFor lets a windowed run start against a volume with no render
manifest, which is the ordinary starting state when the run renders its own
pages. Every other caller still gets "has not been rendered".
-limit bounds the run rather than the window, so a pilot of twenty pages is one
window of twenty. -sweep is on by default and is the reason a window is a
window. -dry prints the window the queue would give and renders nothing.
Fixes #329
38 of the 129 built volumes printed a footnote marker as its own source,
905 times over. The EPUB set a paragraph reading "[^1]: Notably
Democritus, ..." and the pdf, having escaped the caret on the way out,
set "[\textasciicircum{}1]: Notably Democritus, ...". A footnote printed
as its source is worse than a footnote dropped: the reader has to work
out what it is before deciding to ignore it.
The corpus writes a note three ways. 436 places use the Markdown form, a
[^1] where the printing has the mark and a [^1]: line further down
holding the text, which is what the OCR normaliser makes of the (*) and
the (†) the pages actually carry. 118 places write an inline
\footnote{...}. 33 write a bare \footnotetext{...}. Only the first
records which note belongs to which call, and it was the one neither
builder could read.
So the Markdown form is read, once, before either writer sees the body:
the definitions come out and each note's text goes to its call as a
\footnote. Nothing downstream learns a third spelling, because the
corpus already writes \footnote and the pdf has set one since it was
written. The EPUB gets a reading for it at last: a mark that links to a
note at the end of the document and a note that links back, which is
what an EPUB can do and what epub:type="noteref" is for. A bare
\footnotetext becomes a note with no number there, which is what the
page says: this note belongs somewhere on these words and the reading
did not find where.
Four things had to be repaired along the way.
The exercise pointer was eating notes. A note is written at the foot of
the body it belongs to and the assembly parks it at the foot of the
file, which is under the pointer to the exercises whenever the § has
any. Cutting at the pointer took 94 notes in 68 files with it, 22 per
cent of the corpus's footnotes, each called from a mark in the prose
above. StripExercisePointer now carries them across.
The EPUB only escaped the argument of a text mode command where the
LaTeX writer renders it as prose. That was enough while the only thing
inside a \footnote was words, and stopped being enough the moment the
Markdown notes arrived, because a note carries formulae like any other
sentence and the masking puts a NUL placeholder where each one was.
Chapter IV of Set Theory would not parse as XML.
A note is a div and not a p, because two notes in Set Theory alone hold
a display and a p cannot.
And the audit had nothing to say about any of it. A whole book had to be
read to find 905 markers, so there is now a check that names them.
Across all 44 volumes: no XML fault anywhere, and 24 markers left in 7
volumes. Every one is a defect in content/ rather than in the build. 65
calls in the corpus have no note behind them and are a superscript the
reading lost, a (p) over a letter in Topological Vector Spaces V coming
back as [^1] between two formulae; those are left exactly where they
stand, because inventing a note would put an empty one at the foot of
the page and link the reader to it. One definition in the historical
notes lost its colon.
What is not done is the corpus half. The 33 \footnotetext places need
the marker split off and the call put where the scan has it, which means
looking at each scan, and three of them read \footnotetext{9}{...} with
the number as a separate argument. That is a hand pass and not a rewrite
rule.
Fixes #448
…ot ask Two faults that both hid behind a run that stopped early or said nothing. bourbaki toc build returned out of its loop on four different per-volume failures, so whichever volume failed first ended the run for every volume after it in the manifest. int-i-iv-fr is the one that does it today: it carries three contents errata written against a widened overlay that was later backed out, so the first of them matches no page and the sweep died there, thirty-odd volumes short of the end. Each of the four steps now runs inside readContents, there is one place that decides what a volume it cannot read means, and the volume is reported against its own name. The three volumes whose contents was read off their own pages are no longer counted as failures. ac-x-fr and lie-vii-viii-fr print no contents this can read, and alg-iv-vii-fr prints one whose middle leaf is not in the scan: the file runs from chapter V § 13 at the foot of pdf 423 to chapter VII § 1 at the head of pdf 424, so the leaf that listed the end of chapter V and the whole of chapter VI is missing, and the parse comes back with three of the four chapters. The pdf text layer agrees with the reading, so this is the scan and not the OCR. All three were read by bourbaki toc body and their manifests say so in their grammar, which is now a named constant rather than a word spelled twice. Taking the partial reading over the body one would have thrown a chapter away. ocr run -contents took the flag on a host that cannot honour it. The flag swaps in a prompt that keeps the leader dots and the printed page numbers in the column the printing sets them in. A host with a reader named on it runs a program that is not chatgpt-tool, with a fixed prompt in front of a card, and it takes --prompt and answers its own question anyway. Three pages of the French Integration came back in forty-four seconds as an ordinary reading, and nothing in the answer said the question had been changed. Those hosts are now dropped from a contents run with a line saying which and why, and the run is refused when that leaves none. Eleven of the thirteen volumes #385 lists now read. What is left is two volumes and neither is a contents parser: top-v-x has no page map, because it numbers its pages at the foot and 0.5% of them were read, which is the folio thread and not this one; and int-i-iv-fr needs pdf 282 to 284 read with the contents prompt, which is the data half of #395 and wants a host that can be asked. Updates #385 Updates #395
tamnd
changed the title
This was referenced Sep 3, 2026
M13 asks whether the corpus assembles back into the books it came from. bourbaki book answered that for one volume at a time, which is the wrong grain: a check run by hand on the volume somebody is looking at never finds the volume nobody is looking at. The last full sweep had twenty one builds that set no file of content/ at all, each producing a four page PDF and passing eighteen of twenty one checks, because almost every check is about text and there was no text to be wrong about. bourbaki book check builds every volume in every language that has enough of it and writes reports/books.md and reports/books.json, the way the extraction and translation reports are written. One malformed file does not stop it: the first run over the shelf died on its fifth volume, on one unclosed math span, and reported nothing about the other hundred and thirty builds. A volume that will not assemble is now a row of the report and the sweep goes on, and the command exits non zero at the end so that nobody mistakes such a report for a clean one. A floor refuses rather than reports. It is the one thing here that does, because everything else the audit finds is a book with something wrong with it and this is the case where there is no book. A tenth was measured, not picked: the twenty one empty builds run from 0 to 7 per cent and the lowest build holding any text at all is at 12, so anything between 8 and 11 refuses exactly those and nothing else. A stopped typesetter used to be the quietest thing that could happen. The command nilled the build out when it had no pages and the audit returns on a nil build, so a volume that died inside TeX produced an audit with no typesetting section: the same shape as a volume nobody asked to typeset. Two checks now fail on it and the build is kept for them to fail on. A TeX error named a line of book.tex, which is half a megabyte of macro calls this package generates and deletes on every build. "l.41207" tells a reader nothing they can act on. The writer now drops a comment before every body it sets, the line numbers are read back off the finished document, and an error is reported against the file under content/ somebody can open. The cover was drawn twice, once by the class and once in SVG from the same eight measured numbers written out a second time. Two copies of eight numbers is two covers, and the day somebody moves the title on the printed one is the day the two books stop being the same book. The EPUB now takes page one of the PDF the run just set. The SVG stays as the fallback for -no-pdf and for a machine without poppler, and the audit says which it got. Also the half title, the leaf between the cover and the title page. Every one of these printings has one and it is not decoration: it is the first recto, so leaving it out put the title page on a verso, which no book does.
A whole-shelf toc build still changed one file after the last commit: it dropped the historical note of chapter V of evt-i-v, page 80, pdf 338, and said nothing. That note is real. pdf 338 has the running head HISTORICAL NOTE and carries a manual flag recording that the opening was filed by hand, because the reading had put the title through as prose and the note and all the back matter behind it had been assembled into the exercise above. The manifest entry is the correction; the rebuild undid it. Chapter.Historical, Chapter.Exercises and Section.Exercises are the three optional locators, and absent is nil in each. Nil means what zero means one field over: not a reading that came back different but a reading that did not come back. So a locator the manifest has and the rebuild does not is kept and reported, and one the rebuild found where the manifest had none is taken, since there was no reading there to undo. A kept locator's printed page goes through the page map like any other. bourbaki toc build over the whole shelf is now a no-op.
…none int-i-iv-fr has been the one volume the contents route could not read, filed as "the contents re-read hit the colophon, not the contents". The contents had not moved and nothing had hit a colophon. Its erratum reads "§ 2. Support d'une mesure ... 62" with a typewriter apostrophe, pdf 283 carries "Support d’une mesure" with a typographic one, and the corpus writes the typographic one everywhere. One character of an 88 character line. The message was "is on 0 pages of the volume, want exactly one". That is true and it sends the reader looking for a page rather than for a character. So when the count is wrong the pages are searched again with the quote marks folded together and the runs of blanks squeezed, and the line that comes back is printed under the failure with the offset of the first byte that differs. It only reports. Folding the quotes for the match itself would let an erratum typed with the wrong apostrophe apply, and applying it writes the correction into the page, so a corpus that uses U+2019 everywhere would quietly acquire a U+0027 on the corrected line. The erratum is the thing to fix and this says which character to fix.
The contents the book sets comes out of content/. manifests/toc/ is an independent reading of the volume's own contents pages, corrected by hand against the printing wherever the scan got it wrong. Two sources for one fact, and until now nothing held them against each other, so a title that drifted in content/ would be set in the built book and nobody would hear about it. Chapter.Listed and Section.Listed carry the printed reading through the load, alongside Section.Contents, which is the same thing one level down and was already read. A new audit check compares them, names every disagreement, and says how many it compared, so a build in a language the volume was not printed in reports zero compared rather than a pass for a check that did not run. Running it found something on the first try, and not drift. Chapter VIII of Algebra prints twenty one sections and then four appendices numbered from one again, and loadContentsTitles keyed the printed contents on the number alone, so appendix 1 overwrote section 1 and the four sections opening the chapter took the appendices' subsection titles into their contents lines: the built book listed "Algebras without Unit Element" under a section called "Artinian Modules and Noetherian Modules". The consuming loop skipped appendices, which hid it from the appendix side while leaving the sections wrong. Four sections in each of the two printings of Algebra VIII, two in the French Topologie IX, one in Lie VII, seventeen in the library. printedKey now puts the appendix flag in the key, where it belongs, because Bourbaki numbers appendices from one alongside the sections and corpus.Section holds both in one field. With that fixed the whole shelf agrees: 989 titles compared over every volume and language, 134 chapters and 855 sections, none differing. The check is a gate on that from here.
A printed Bourbaki volume ends with an index of notation and an index of terminology, and the build had neither. It ended on the last chapter's historical note, which is 41 pages short of what the English Algebra I to III actually is. The indexes are transcribed rather than generated, because the printed index references chapter, section and number rather than a page: "Abelian group: I, § 4, no. 2." A transcription therefore survives a rebuild that paginates differently, which is the whole reason the printing set it that way, and a generated index would have had to be regenerated against a page numbering the corpus does not own. manifests/books.yaml gets notation_index and terminology_index on a book, in the same shape the reader note and the introduction already have, so a printing that has neither says nothing and gets nothing. corpus.Chapterless grows the two new kinds, so that the front matter and the back matter answer the one question the structure check is really asking. The assembler needed a stop: chapterEnd walks forward until a missing page or the next chapter head, so with nothing after the last chapter it swallowed the back matter whole. The historical note of Algebra III was pdf 679 to 734 and 1805 lines, and held the bibliography, eight pages of the index of notation, thirty-three pages of the index of terminology and the publisher's back cover blurb. Clamped to the first back matter page it is 679 to 691 and 137 lines. The bibliography stays with the note, where the printing puts it. An index is set one entry to a line, not as prose. oneEntryToALine walks lines rather than blocks because the two readings disagree about blank lines: the notation index puts one between every pair of entries down to page 676 and then none at all, and the terminology index has none anywhere. The first cut kept any block holding $$ intact, which left the packed tail, 130 entries with four displays in it, as one paragraph, and split a fence off its formula. Tectonic reported that as "! Missing $ inserted." and refused the PDF. \bsection takes a fourth argument, the contents line, so that the built contents comes off manifests/toc/ where the volume was printed in this language and off the head where it was not. That is the rule the numbered subsections have had since they were set. alg-i-iii in English: 29 of 29 checks, 755 pages against the printing's 734.
Three repairs the back matter asked for once the two indexes were bound
into it.
book: a thirtieth check, that every reference in the index of notation
and the index of terminology names a chapter the volume actually has.
Bourbaki's indexes point at a place in the argument rather than at a
page, "Abelian group: I, § 4, no. 2.", so a wrong numeral is a reference
that goes nowhere and no page number will catch it. The check read
11663 references across the library and rejected six, one in each of
top-i-iv, top-v-x and ens-i-iv-fr and three in ac-v-vii-fr. It also
turned up a defect it could not see: the English Commutative Algebra
index had 140 references whose chapter numeral had been read as digits,
"7, 1, 1" for "VII.1.1", which the pattern skipped in silence. With
those repaired the library reads 11803 references and rejects none.
mathtex: Split treated a dollar inside \text as the end of the span, so
\text{$\Gamma$, $\Gamma'$ correspondances} in the ens-i-iv-fr index of
notation closed its span three characters early and the rest reached
XeTeX as "! Missing $ inserted.". Split now carries a stack of open
braces, one bool each, and only a brace opened as the argument of a
\text-family command suspends the span. Any other open brace still lets
the dollar close, because M04 has to keep reporting $\mathbf{Q$.
book: the two double angle brackets, which an extraction writes around
a stretch of page it could not read. The corpus has exactly one, the
assemblage of primitive signs in the footnote of Théorie des ensembles
II, § 1, no. 7.
…ented one A run of remarks or examples comes apart into one statement per member, and two of the three ways a page can hide a member were already cut. The third is the block that carries the next member without opening on one: the last paragraph of a member runs over the foot of the page, so the block opens on neither member and both cuts decline. Example 2 of no. 1 of § 4 of chapter I of Functions of a Real Variable is the case, where the display for $x^2$ ends the block and "3) The function $|x|$ is convex" opens the line under it. Ten of the thirteen places were hidden a second way as well. The reading indents a member the printing set as a hanging paragraph, and four spaces were enough to put the marker out of reach of patterns anchored at the head of the line, so the cut declined and, where it did not, the piece it cut off was not recognised as a member either. The indent now comes off with the cut. Together they recover 26 statements over 14 files, because a member cut out of the one before it makes the member after it visible in its turn. Three statements are renumbered, and one of those moves a tag onto the member the volume numbers: 03TP was on the unnumbered example that follows Example 6 of no. 1 of § 6 of Theory of Sets II, and Example 6 is now a statement of its own. The new cut is guarded, because a run stays open across the statements printed between its members and next is still counting a long way past the last one. Without the guard the steps of every proof after a Remarque 1 are read as Remarque 2, which put two statements under lie-iii-s1-n8-rem-2 and stopped chapter III of Lie assembling at all. ocr: \footnotetext and \footnotemark come out of the prose TeX list now that the corpus has none of either, so a reading that writes one is a page to read again rather than a note that prints under whatever number the last real footnote left behind.
Boxes 16 and 17 of M13 were implemented in the writer and watched by nothing. Every other check in audit.go is anchored to a chapter, a § or a statement, so a volume could lose its whole front matter and pass all of them: the chapters the manifest names would still be there, the §§ would still run without a gap, and four leaves are half of one per cent of the text length. The way a missing title page showed up was that somebody opened the PDF, which is what this file exists to stop being the way anything shows up. Two checks. The front one asks for the five commands the writer always sets and for the edition line where the manifest carries one. The back one is conditional on the volume rather than a fixed list, because not every printing has a historical note and not every one carries both indexes; a volume with none of the three passes with nothing to bind. The historical note is asked for by its anchor and not by its heading. Writing the test found that out: the heading is the note own title where it has one and \bhistoricalname only where it does not, so the command would have passed every volume whose notes are untitled and failed every volume whose notes are titled, which is the opposite of a check. The anchor carries the chapter, so a volume that lost the note off one chapter of six fails with the chapter named. Run over the shelf: 17 of 17 on every volume tried, with the front matter at 5 or 6 of 6 and the back matter from 0 of 0 on the Historical Note volume to 9 of 9 on the French Topology V to X.
Owner
Author
|
Final state of this branch. Issues closed: 22 bugs, plus milestones #39 (M12) and #411 (M13). The audit went from 30 checks to 32. The two added here are the front matter and the back matter being bound into the volume, which were the two parts of the build with nothing watching them — a volume could lose its whole front matter and still pass every other check, because the chapters would all be there, the §§ would run without a gap, and four leaves are half of one per cent of the text length. The back-matter check asks for the historical note by its anchor rather than by The shelf: 44 volumes, 140 builds, 140 PDFs and 140 EPUBs, 47,906 pages, 2,348,386 formulae, nothing that failed to assemble. Of the 32 checks, 30 now pass on every one of the 140. The last defects cleared on this branch were:
What still fails, and why it is not a code fix: 26 builds hold less text than the printing does, and one build (
|
Bourbaki numbers the parts of an exercise with Latin letters and the members of a list of equivalent conditions with Greek ones. The English scan loses the Greek sometimes and not always: α comes back as a, β as b, γ as y and δ as the digit 8, so ac III § 3 exercise 23 reads 'conditions (a), (β) and (y)' with a correct (γ) four words later in the same sentence. Fifteen lists across the corpus were wrong this way and the French lane has none, which is what says these are misreadings and not a convention. Looking for a Latin label beside a Greek one finds them, and also finds far more of the ordinary shape, which is an exercise whose parts (a) and (b) sit around a Greek list. What separates the two is that a list of conditions is consecutive. Map each Latin label back to the Greek it was read from and ask whether the run then walks the alphabet without repeating or skipping: (a) (β) becomes α β and is a finding, while (a) (α) (β), which is part (a) and its own two conditions, becomes α α β and is not. That rule sorted all nineteen mixed runs in the corpus correctly, the fifteen real ones and the four that were the ordinary shape. Two things this got wrong first. The Greek letters are two bytes each, so strings.Index put β one past α by two rather than by one and nothing was ever consecutive; the order is a map now. And Volume.Pieces does not return the exercises, which is where every one of the fifteen was, so the first cut passed the whole library while ac IV § 2 exercise 10 still read (a) (b) (y) (δ). Both have a test. The check passes on all 140 builds of the shelf.
fixSeal parsed its positional arguments and dropped them on the floor: the return of parseFlags went to _, so -lang was the only thing that narrowed the walk. A run asking for one Vietnamese section resealed 209 of them and 4 more in en-mt, taking out every hand edit anyone had in the tree, and undoing it meant reverting file by file against git status. The paths now bind. With none given the behaviour is what it was, the whole of content/ under -lang, since that is what the sweep after an assembly wants. A named path the walk never offers is an error rather than a run that reads nothing and reports success: exercises carry no hash of their own and eachSection skips them, and a path in one language under -lang another is a contradiction, so both now say so. Five tests, one of them the incident: two unsealed sections, one named, and the other has to come back untouched. It fails without the filter.
The terminology rule reads a chunk for a glossary term the translation left standing in English, and it was reading the names of cited papers. Exercise 15 of § 2 of Topological Vector Spaces I is a line of prose and a footnote, the footnote is a citation, and the only occurrence of the word space in the file is inside "The space of p-adic norms". The rule asked for "không gian" in the name of a paper, the chunk was refused on all three attempts, and the file was one of fifteen a 160-file run could not land. bibEntryRE cannot help here: it opens on the number a note cites a work by, as "12.", and a footnote marker is a bare 1 with no period after it. So read the citation by its tail instead, the volume and the year in parentheses and the pages, which is the part no running sentence has. Over content/en that shape is on 268 lines of 259,261; bibEntryRE already covers 88 and the other 180 are the bracket-numbered references of the historical notes, whose German and French titles were being read the same way. Three tests. The two exemptions fail without the change, and the third is the guard: an ordinary sentence that leaves a term in English is still refused.
An index of notation writes its entries as bare LaTeX with no dollar signs
anywhere: "\sum_{i=p}^q \sum_{j=r}^s x_{ij}, \prod_{i<j} x_{ij} : \text{I, § 1,
no. 5}". mathtex.Strip toggles on the dollar, so with none on the line it takes
nothing out and the entry arrives whole in the text the terminology rule reads.
The glossary then found the word sum inside \sum and asked for "tổng" in a list
of symbols. Four of the five indexes of notation a 160-file run could not land
died on that, and the largest was refused on every attempt over two models.
Take the control words out after Strip, where a word inside a formula is already
gone and what is left is either prose or bare mathematics that Strip could not
see. The comment in Strip about left, right and square being read as prose is
the same fault from the other side: it was fixed for a display written $$...$$
on one line, and a control word outside dollars altogether was still read.
Two tests, and the second is the guard: taking the control words out must not
take the prose with them, so a line that really does leave a term in English is
still refused.
An index of notation writes "Map(M, N), Pol_A(M, N), Pol(M, N) : IV, p. 57", and Map there is the functor set upright, the same thing as Hom, End and Aut, which weldedRE counts at 2165 occurrences over 155 names. weldedRE cannot help because it wants the dollars a welded name is written with and an index writes none, so the glossary read the word map and asked for "ánh xạ" in a list of symbols. That was the last terminology refusal standing over the tail of the Vietnamese gap-fill. Take what stands before the locator. Over content/en the shape matches 3398 lines in the fourteen index_of_ files and four lines anywhere else, and those four are one section's headings, "Applications : I. Canonical decompositions ...", where the I numbers a part rather than naming a volume. Headings are left alone for them, which is the third test.
All eighteen remaining fix subcommands parsed their positional arguments and threw them away. What that reads like from the outside is a command that took the path and did as it was told, and it is not: -lang and -book were the only things narrowing the walk. That is exactly how fix seal came to reseal 209 files in one lane and 4 in another when it had been asked for one. fix seal now takes paths, because naming the file a hand correction was in is the whole use of it. The rest are sweeps by design: fix padding tightens every padded span in the book and fix dollars every solution, and three of them write content/ through helpers shared with other commands, so giving them paths is a redesign rather than a repair. What they must not do is be silent about being handed something they will not use, so they refuse it and name it. Two tests: four sweeps refuse a path and say why, and the flags still work, so the refusal is about the path and not about having been given anything at all.
staticcheck reads b.Refused("a") || b.Refused("a") as identical
expressions either side of the operator (SA4000) and fails the build,
and it is right to: the || only makes the second call when the first
came back false, so the test holds only while the code under it is
correct.
A count is not something an answer can be corrected from. Two exercises of the last Vietnamese run stood on one for eleven attempts across two hosts and three sittings: 54 inline spans in 2685 characters, most of them a single letter, and the answers came back holding 52, then 53, then 52 nine times over. "has 52 math spans and the English has 54" says nothing about which two went, so the re-ask askChunk makes is another draw from the same urn. The spans are aligned as a longest common subsequence rather than compared as sets, for the reason auditMath already compares them in order: a reflowed paragraph moves a formula without losing it. What falls out is named on both sides, and it supersedes the span-by-span report when the counts differ, which used to point at the first span after the drop and call it changed when the model had copied it exactly.
The index of notation of Algebra VIII could not get past its first
span. The English opens with $A_M$ and the answer came back $A\_M$,
twice running on two sittings. In TeX those are different things and
RuleMath is right to refuse it, but the model is doing something
reasonable one level up: an underscore in Markdown prose does want
escaping, and a file that is symbols the whole way down gets escaped on
line 3 every time.
A repair on the same terms as Respace, and as one-sided. It works span
by span against the English, fires only where taking the escape off
makes the span the English span, and does nothing where the two sides
do not line up. Because the test is that the result is the English, it
cannot turn a wrong formula into an accepted one.
Only \_ and \* are unescaped. Over the 316374 math spans of content/en,
\{ occurs 3252 times, \} 3228, \\ 1739 and \& 32, all of them TeX doing
its job, while \_ and \* occur 0 times each.
The verifier run of the twenty-case set put false rejects at 5 of the 10 right answers, against the 30 per cent spec 07 section 6 tolerates. The other half is fine: the reference-blind audit judge agreed with the person on 9 of 10, and neither judge accepted any of the 10 wrong answers. It is not calibration. Three of the four the truth judge turned down had worked the whole checklist through and cleared it. One wrote DISCHARGED against all three obligations, DOES NOT FALL against all four failure modes, PASSED against all three falsification checks and found both citations honest, then wrote COMPLETE: NO, SELF_CONTAINED: NO, VERIFIABLE: NO, SCORE 5/7 and FAIL, naming nothing. Another discharged 5 of 5 and passed 4 of 4 and scored it 4 out of 7. The summary block was being answered fresh from an impression and the review above it was doing no work. "Your default is to fail" is what produced that: right for a step the judge has not checked, wrong for one it has checked and found sound, and the prompt did not distinguish them. That sentence is now scoped to unchecked steps, and every NO has to name what it comes from. Requiring a reason for a rejection cannot raise the false accept rate, which is the number that decides whether a verdict here is worth anything and is the one currently met. Nothing about PASS is loosened, and a test holds the gate in place.
An entry of an index of notation lists the notations that share a definition, and the English sets the list as one span with a comma in it. A model reading that as prose sets each notation in its own dollars and writes the comma between them as text: the same line on the page and a different file. The index of notation of Integration I-VI was refused on it fifteen times, at 51 spans against the English 33. On the terms Respace and Unescape set. It fires only where the answer spans and the text between them assemble into the English span exactly, so it cannot weld two formulae the book keeps apart and cannot turn an altered formula into one that passes. Where the two sides do not line up all the way it does nothing: a partial alignment would be guessing which span went with which, and a formula put in the wrong place is worse than the refusal it was avoiding.
…ral spans" This reverts commit 88586ca. The measurement it cited was a misreading and the repair never fires. RuleMath reports a span count and then names the spans, and I read "has 51 math spans and the English has 33" on the index of notation of Integration I-VI as the answer having broken the English's compound entries apart. It is the other way about. Walking the English spans against the answer's, the first 31 line up exactly and the alignment stops dead at the 32nd: of the 20 answer spans past that point, 2 are spans of this chunk, 2 are spans of the chunk before it and 16 are nowhere in the English file at all. The model translated its chunk and then carried on writing index entries out of nothing. The refusal is the audit doing precisely what it is for, and no repair should touch it. Run over all eight of the files that will not land, against the chunk each answer was actually asked for rather than against the whole file, Remerge does not fire once. There is no case in front of me that it repairs. A function in this pipeline earns its place by naming the answer it fixed, and this one cannot. Unescape stays and is untouched: on the same eight files it takes chunk 1 of the index of notation of Algebra VIII from one problem to none, which is the $A\_M$ against the English $A_M$ it was written for.
The prompt was changed so that every NO names what it comes from, and the judge did as it was asked: "COMPLETE: NO, obligation 3 is not discharged." The line function anchors the end of the line straight after the value, so not one of those parsed. HasQuality went false on reviews that had answered every field, and the eval threw the case away and asked again. It cost six of the twenty cases of the built-in set. All six were cases the judge had decided, and losing them took the reject half of the run from ten cases to four, which is the half the false accept rate is measured on. The five lines the prompt asks to carry a reason are now read in the shape OBLIGATION and PART lines have always been read in: the value, then an optional comma or stop, then the judge's few words. Nothing reads those words. The correction loop is handed the review entire, so the reason already travels and a field for it here would be a second copy nobody asked for. The value still has to be the whole decision. A word boundary keeps TRUENESS from reading as TRUE, the line still has to begin with the key, and a field named in the middle of a sentence is still not an answer. Tests for all three, and the two that matter fail against the anchored form. prompt: assert the two halves against each other. What the prompt asks for is read out of the prompt, so a prompt that stops asking for reasons fails the test rather than making it vacuous. This is the check that was missing: the file comment has always said a prompt asking for a word the parser does not know should fail the test rather than three hundred solves, and in this direction nothing was checking it.
corpus.Root falls back to the working directory when BOURBAKI_CORPUS is unset, so a run started from the solver checkout looks for the benchmark answers under a tree that holds the benchmark code instead. Every case comes back missing and the message said the set names answers the corpus does not hold, naming the set and the first file and never the root it looked under. That reads as a broken benchmark set, and it cost a run. Every case missing at once is not a set that went wrong, it is a root that is not a corpus, and it now says so and how to point the run at one. Both messages name the root, because the root was the thing that was wrong and it was the one thing not on the screen.
-only narrows the registry before the ssh targets are gathered, so asking probe for a gateway or subscription route by name emptied the selection and reported "no route in routes.json names an ssh host" about a file holding four of them. That reads as the route file being wrong and sends you to look at it. The answer is that the route asked for is not reached over ssh and never was; it is asked directly. The message now names what -only asked for and says why there is nothing to do, and still names the route file when nothing was selected, because then the file really is the empty thing.
alignSpans knows the position of every span one side has and the other does not, and threw it away, so the rule named the text alone. The text is not always enough to find a span by. A Topology X exercise came back with 61 spans against the English's 60 and the extra one was "a", in a chunk that uses $a$ twice over legitimately, so the complaint named a letter the answer was right to have and left the model to pick which of its three was the one too many. Two models of different families picked the same way and both picked wrong; the file has stood on it since. Each span is now reported after the last span the two sides agreed on, which is one place and not three. The anchor is dropped where it narrows nothing: before the first agreement there is none, and a span anchored to its own text is a repeated pair, where "after itself" is a sentence rather than a direction. The audit messages are not in prompt_sha256, which covers the prompt template and the language rules, so nothing already written goes stale.
The card could print believed 37 and disputed 81% of the readings in the same breath and never say the two overlap. They do: 15 of the 37 solutions the judges call verified have a reader's finding written against them in the file, which is 40.5% of what this corpus currently believes. Disputed had already counted those 15, but it counted them under hand read, where a finding reads as a note on the reading rather than on the status. The number that matters is the one nobody could get off the card without opening 42 files, and it is now a line under believed, which is what it qualifies. Two of the findings are that the proof assumes the result of the preceding exercise. That is the failure a fluent wrong proof actually looks like, it is exactly what a judge reading the answer on its own cannot see, and it is on a solution standing at verified. It is reported and not subtracted. A finding is a reader's note and not a verdict, several are about the delimiters rather than the argument, and moving a status is a decision about the mathematics that belongs in the file. The card's job is to stop 37 being quoted without the 15.
… the question RenderWithin trims the pieces and Chars measures the pieces. The block that says what is in the corpus and is not in front of you is neither trimmed nor counted, so the trimmer cut a context down to the room it was given and then wrote that block out underneath it, unbounded, and the question went out at whatever length that came to. Exercise 1 of Commutative Algebra I § 1 measured 70.5k of context and left the assembler as a question of 447.9k. 422.3k of it was this block: 5154 entries at about 82 characters each, which is what the depth-2 closure of a Bourbaki cross-reference graph comes to when the citations run across volumes. The engine logged that it was sending it anyway and sent it, because from where the engine stands an exercise that will not fit is a fact about the exercise. Nothing was wrong with the exercise. The dry run said in one line that it had trimmed the question to fit 28.0k and in the line above that it was sending 447.9k, and both came off the same call. 4284 of the 4434 unattempted exercises were being asked this way. The 42 that have solutions are all in the two books whose closure happens to be small. Capped at 40, with the number of the rest printed in place of their names. A list of five thousand section names is a count written the long way and tells the model nothing the count would not, and past the cap the count is the one thing in the tail worth having. Under the cap nothing changes: a handful of references somebody could raise the cap for is still named one by one, which is what the block was written for. Measured on the same exercise: 447.9k to 24.8k, inside the 28.0k room. Across the whole printing the questions trim now, and 937 of the 4433 are still over the room on the pieces alone, which is the case the code already calls everything that could go has gone and is a different question.
The usage calls this the only honest check that a host will answer, and its verdict went to the terminal and nowhere else. One command wrote the ask record, translate, so the taking column of fleet accounts spoke for the translate driver and for nothing else. It missed the host it exists to catch. The board read 8/10 taking, first one back now, in the same minute that four asks to that host in a row died with ChatGPT never accepted the prompt, one of them a question thirty characters long. All four were fleet ask, so none were in the record, and the newest entries in it came from a translate run already finished. fleet.Classify had no caller outside its own tests until now.
solve run, review and eval each made thousands of asks and none of them reached the ask record, so fleet accounts drew its taking column from translate alone. A board printed during a solve run could call a host live on the strength of a translate run that had already finished. They kept ask-usage.jsonl, which is a different thing: this run's questions in full, read afterwards, against the last two hours across every run, read before anybody points more lanes at a host. Written per ask rather than merged at the end, because a run that is killed is the ordinary case here and is the run whose record is worth the most. noteAsk takes a mutex now. Append re-reads the file it adds to, so eight lanes landing together recorded one ask and lost seven. There is a test.
RenderWithin read limit <= 0 as no limit and returned the context whole. Every caller works the room out by subtraction, the limit less what the rest of the question already takes, so a room of zero or below means there is no room at all. It is the tightest a question is ever assembled and it was the one case that got no trimming. The truth judge is what it served. Its question carries a reference and a candidate solution as well as the instructions, and on ens-iii-s1-ex-3 those came to 54736 characters on their own against a limit of 32000. The room was -22736, so the branch attached the whole 54836 character context underneath and the question went out at 109572. 4 of the 122 truth questions in work/ went that way, carrying 105791 characters of context that had nothing holding it. One is ac-vi-s1-ex-1, which is the 41201 in the log of the run going on now. This does not bring those questions under the limit and cannot. What is over on them is the reference and the solution, and a truth judge without either is not judging anything. It stops the assembler adding a context to a question that had no room for one. The unlimited case never reached here: the engine returns Render itself when its limit is negative, before a room is worked out. TestAContextThatFitsIsSentWhole asserted the old reading and now asserts this one, deliberately. Nothing passed a literal zero to mean unlimited.
order never offers TheExercise, TheSection or Outside up, so a context trimmed until everything that could go has gone still carries this block whole. On the exercises where it is long it is most of what is left. Commutative Algebra VI and VII § 1 exercise 1 each carry 545 of these at 13723 characters against a question limit of 32000. It is the cited-and-not-shown block again and it is worse, because that one could be trimmed once it was counted and this one may not be touched at all. Measured on the floor, which is what solve context -ask 1 renders: VI § 1 exercise 1 came to 22339 characters and comes to 10651, VII § 1 exercise 1 to 27270 and 15582. The same 11688 either way, being the same list. The instruction is what the block is for and it is not a list. It says a result from a volume not shown must be named as such, and that holds for the five hundred as much as for the forty. What the tail costs is the chance to recognise one particular citation as outside rather than merely absent, and the sentence covers that in general. At the reference call this does not make the question shorter, because the trimmer was already fitting it: ac-vii-s1-ex-1 sends 27.3k before and 27.5k after. It spends the room on mathematics instead of on names.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Twenty-six commits against the open bug list. Every issue below is fixed on this branch unless the line says otherwise.
go build ./...,go vet ./...andgo test ./...are clean at every commit.Closed by this branch
OCR and the fleet
ocr run -windowrenders off the queue itself, so the driver keeps no cursor.\mathscris no longer overwritten with\mathcalby an OCR re-read.ocr checkrule 2 finds TeX set outside every math span, instead of counting delimiters.\mathcaland\mathfrakargument and where it is, which is what makes the Fraktur collapse visible.CanOCRasks that question rather than asking a reader for a browser it never starts.ocr fill -againis the way to ask for a page that already passes, which is how the backlog is expressed as a command.fleet accountssays whether asking a host has been working, not only what the profile holds.Page maps
Assembly and audit
content/, which was silently reverting 80 repaired files across 21 books.publish -checkreports every page it cannot build instead of stopping at the first.qualitydoes not build a reference graph a run never reads.Translation
-rawcannot waive a rate limit page into the corpus as a translation.-withinbounds the run as well as the ask. The third part, raising the span cap and measuring refusal against it, landed earlier as translate: sixty spans to a chunk, and an archived answer read back only for the passage it answered #398 and set the cap at sixty with the table behind it.Solver and the book builder
solve eval -writeno longer replaces the measuredeval.jsonwith a truncated one.Advanced but not closed
top-v-xhas no page map, which is the foot-number folio thread, andint-i-iv-frneeds three pages re-read with the contents prompt.toc buildalso no longer ends the whole sweep at the first volume it cannot read, and no longer counts the three volumes read bytoc bodyas failures.ocr run -contentsdrops a host that carries its own prompt and would answer the ordinary question, says which and why, and refuses the run when that leaves none. The data half is the re-read of pdf 282 to 284 and wants a host that can be asked.Milestone work
#411 (M13, books back out). Five and a half of its ten open boxes, and a working sweep over the whole shelf.
content/at all and each produced a four page PDF while passing 18 of 21 checks, because almost every check is about text and there was no text to be wrong about. Those 21 sit at 0-7% and the lowest build holding any text is at 12%, so a tenth refuses exactly them.content/it came from rather than a line ofbook.tex, which is half a megabyte of generated macro calls that is deleted on every build.bourbaki book check, the same build over every volume in every language, writingreports/books.mdandreports/books.json.The sweep paid for itself in three runs. The first died on its fifth volume on one unclosed math span and reported nothing about the other 138 builds, which is why a volume that will not assemble is a row rather than the end of the run; it then found seven broken builds across six corpus files. The second showed two Vietnamese volumes silently setting 140 pages of a 314 page book, and the new error-naming pinned it to one image link in one file in one line of output. The third is clean: 44 volumes, 140 builds, all 140 reaching a PDF, 47763 pages, 90 passing all 28 checks, nothing that will not assemble and no typesetter that stops.
#39 (M12).
toc verifyruns over all 44 volumes at 6499 of 6513 headings, 99.79%, with 34 volumes at 100% and no heading printed on a page the contents does not name.It had been measuring the corpus against the worst reading of it available. For a scanned volume it asked the PDF's own text layer, which is the scanner's OCR from twenty years ago, when this project has read the same pages far better into
pages/. The evidence is a clean split: all 21 volumes with exercise misses havetext_layer: ocrand not one of the 6 native or 4 unlayered volumes has a single one. Reading the page files where a scanned volume has them took the shelf from 6373 to 6499 and took the 31 headings reported as printed on another page to none — every one of the 31 was a lost§rather than a heading in the wrong place, so the command had been failing on its own blind spot. A native layer is left alone, and that was measured too: preferring the page files there cost the three born-digital volumes 201 of 201 down to 176, 98 down to 93 and 165 down to 160.Not in scope
#8, #9 and #34 are milestone epics whose remaining work is fleet time or is externally blocked rather than code: long
solveand translation runs for #8 and #9, and for #34 a share link that has to appear and a person who has to read an imported section against the printing. The repository has no GitHub milestones defined; those issues are where the milestone content lives.