#729 fixed the instance — the multi-char operator table in fix_spacing's third pass is now the lexer's full set, matched longest-first — and added a corpus gate asserting that every .eigs in the repo that parses before --fmt still parses after it. This issue is suggestion 2 from that report, which was deliberately left out of the fix: the pass still scans raw characters, not tokens.
Why the class is still open
The formatter now has to keep a hand-maintained operator table in sync with src/lexer.c:546-613 by convention alone. Nothing mechanical enforces it. Add an operator to the lexer and forget MULTI_OPS in src/fmt.c, and the single-char branches split it again — which is exactly how #729 happened, and it corrupts files under --write.
The corpus gate added in #729 catches this only if some .eigs in the repo happens to use the new operator. A newly added operator typically has no in-repo user yet, which is precisely when the gate is blind.
Known residual (cosmetic, not corruption)
The exponent guard distinguishes a literal from an identifier by scanning backwards for a digit run not preceded by an identifier character. That is correct for 1.5e+10 vs a+1, but an identifier ending in digits-then-e is left unspaced rather than corrupted:
x is a1e+1 # stays `a1e+1`; a token-level pass would emit `a1e + 1`
Harmless — it parses and runs — but it is the kind of thing a character-level pass cannot get right in general.
Suggested shape
Run the spacing pass over lexer.c tokens and re-emit with spacing rules per token kind. The lexer already knows what an operator is, what a numeric literal is, and where strings and comments end — pass 3 currently re-derives all of that by hand, including its own in_str / in_comment tracking duplicated across all three passes.
Then the gate worth adding on top: assert every operator token kind the lexer produces round-trips through --fmt unchanged, driven off the token list rather than a second hand-written table. That is the check that would have failed the day #729 was introduced, with no .eigs file needing to use the operator.
Not urgent
No known corruption remains — #729's gate is green across all 343 repo files. This is about retiring the bug class rather than fixing a live break, and it touches a tool with no consumer pressure on it right now.
#729 fixed the instance — the multi-char operator table in
fix_spacing's third pass is now the lexer's full set, matched longest-first — and added a corpus gate asserting that every.eigsin the repo that parses before--fmtstill parses after it. This issue is suggestion 2 from that report, which was deliberately left out of the fix: the pass still scans raw characters, not tokens.Why the class is still open
The formatter now has to keep a hand-maintained operator table in sync with
src/lexer.c:546-613by convention alone. Nothing mechanical enforces it. Add an operator to the lexer and forgetMULTI_OPSinsrc/fmt.c, and the single-char branches split it again — which is exactly how #729 happened, and it corrupts files under--write.The corpus gate added in #729 catches this only if some
.eigsin the repo happens to use the new operator. A newly added operator typically has no in-repo user yet, which is precisely when the gate is blind.Known residual (cosmetic, not corruption)
The exponent guard distinguishes a literal from an identifier by scanning backwards for a digit run not preceded by an identifier character. That is correct for
1.5e+10vsa+1, but an identifier ending in digits-then-eis left unspaced rather than corrupted:Harmless — it parses and runs — but it is the kind of thing a character-level pass cannot get right in general.
Suggested shape
Run the spacing pass over
lexer.ctokens and re-emit with spacing rules per token kind. The lexer already knows what an operator is, what a numeric literal is, and where strings and comments end — pass 3 currently re-derives all of that by hand, including its ownin_str/in_commenttracking duplicated across all three passes.Then the gate worth adding on top: assert every operator token kind the lexer produces round-trips through
--fmtunchanged, driven off the token list rather than a second hand-written table. That is the check that would have failed the day #729 was introduced, with no.eigsfile needing to use the operator.Not urgent
No known corruption remains — #729's gate is green across all 343 repo files. This is about retiring the bug class rather than fixing a live break, and it touches a tool with no consumer pressure on it right now.