check-language: scientific/technical Unicode notation flagged as foreign language (false positives) — add symbol allowlist config

[ffedoroff]

Summary

cfs check-language (rule LANG001) flags scientific / technical Unicode notation as if it were disallowed foreign-language text. On a normal STEM/CS docs tree, every Greek letter, sub/superscript, vulgar fraction, math bracket, keyboard glyph and circled marker is reported as a violation — even though none of it is a foreign language.

In our docs this produced 74 violations across 17 distinct codepoints, and all 17 are false positives — technical notation, not foreign text.

Evidence (real run, --languages en)

Engine v1.5.9, scanned via studio.utils.content_language.scan_file:

char	codepoint	n	Unicode block	name
Δ	U+0394	5	Greek and Coptic	GREEK CAPITAL LETTER DELTA
η	U+03B7	12	Greek and Coptic	GREEK SMALL LETTER ETA
⁶	U+2076	2	Superscripts and Subscripts	SUPERSCRIPT SIX
⁹	U+2079	2	Superscripts and Subscripts	SUPERSCRIPT NINE
⁺	U+207A	1	Superscripts and Subscripts	SUPERSCRIPT PLUS SIGN
₁	U+2081	14	Superscripts and Subscripts	SUBSCRIPT ONE
₂	U+2082	15	Superscripts and Subscripts	SUBSCRIPT TWO
⅔	U+2154	1	Number Forms	VULGAR FRACTION TWO THIRDS
⌘	U+2318	6	Miscellaneous Technical	PLACE OF INTEREST SIGN
⌥	U+2325	4	Miscellaneous Technical	OPTION KEY
⎘	U+2398	2	Miscellaneous Technical	NEXT PAGE
⏳	U+23F3	2	Miscellaneous Technical	HOURGLASS WITH FLOWING SAND
ⓘ	U+24D8	2	Enclosed Alphanumerics	CIRCLED LATIN SMALL LETTER I
⟨	U+27E8	2	Misc Mathematical Symbols-A	MATHEMATICAL LEFT ANGLE BRACKET
⟩	U+27E9	2	Misc Mathematical Symbols-A	MATHEMATICAL RIGHT ANGLE BRACKET
⭐	U+2B50	1	Misc Symbols and Arrows	WHITE MEDIUM STAR
️	U+FE0F	1	Variation Selectors	VARIATION SELECTOR-16

Typical lines: length = N₁ + N₂, unique operators (η₁), effort^⅔ ÷ 3000, ⌘/Ctrl-click, ⚠️ CHECK (only the trailing U+FE0F is flagged, not the ⚠ U+26A0).

Root cause

utils/content_language.py models the policy as "non-Latin script ⇒ foreign language". SCRIPT_RANGES["en"] allows Latin + a hand-picked subset of symbol blocks, and _COMMON_RANGES adds only emoji / zero-width / BOM. There is:

• no per-character allowlist,
• no notion of language-neutral "symbol sets",
• no config knob beyond the language code list,
• (related) inline code spans `…` are not skipped — only fenced ``` blocks — so `η₁` in inline code is flagged too.

This is an oversight, not a deliberate exclusion

The en table already deliberately whitelists technical notation — Mathematical Operators (U+2200–22FF: ∑ ∂ √ ≤ ≥ ∈ ∀), Letterlike Symbols (U+2100–214F: ℝ ℕ ℤ ℏ ℓ), Arrows (U+2190–21FF), Dingbats, Geometric Shapes. So ∑, ≤, ℝ, →, ², ½ pass fine. It just forgot the adjacent blocks that the same documents inevitably use. The boundary is arbitrary: ℝ (allowed) vs Δ (flagged); ² U+00B2 (allowed) vs ₂ U+2082 (flagged); ½ U+00BD (allowed) vs ⅔ U+2154 (flagged); ≤ U+2264 (allowed) vs ⩽ U+2A7D (flagged).

Which blocks are "forgotten" vs already covered

These blocks are clearly intended-to-be-allowed technical notation but are missing from en (every one produces false positives in STEM/CS docs):

Block	Range	Examples	Status
Greek and Coptic	U+0370–03FF	α β γ Δ Σ Π λ μ η θ ω	❌ forgotten
Greek Extended	U+1F00–1FFF	ᾶ ή ώ (accented Greek)	❌ forgotten
Superscripts and Subscripts	U+2070–209F	⁰ ⁿ ⁺ ₀ ₁ ₂ ₙ	❌ forgotten
Number Forms	U+2150–218F	⅓ ⅔ ⅛ Ⅰ Ⅱ	❌ forgotten
Miscellaneous Technical	U+2300–23FF	⌘ ⌥ ⎋ ⏎ ⌫ ⏳ ⎘	❌ forgotten
Misc Mathematical Symbols-A	U+27C0–27EF	⟨ ⟩ ⟦ ⟧	❌ forgotten
Misc Mathematical Symbols-B	U+2980–29FF	⦃ ⦄ ⟬	❌ forgotten
Supplemental Mathematical Operators	U+2A00–2AFF	⨉ ⩽ ⩾ ⨁	❌ forgotten
Combining Diacritical Marks for Symbols	U+20D0–20FF	a⃗ (vectors)	❌ forgotten
Enclosed Alphanumerics	U+2460–24FF	① ② ⓘ ⓐ	❌ forgotten (UI/legend)
Misc Symbols and Arrows	U+2B00–2BFF	⭐ ⬆ ⬇ ★	❌ forgotten
Variation Selectors	U+FE00–FE0F	emoji-presentation ️	❌ forgotten
Currency Symbols	U+20A0–20CF	€ ₽ ₿ ₹	❌ forgotten
Mathematical Alphanumeric Symbols	U+1D400–1D7FF	𝔽 𝕏 𝐀 𝓛	❌ forgotten
Mathematical Operators	U+2200–22FF	∑ ∫ ≤ ∈ ∀	✅ already allowed
Letterlike Symbols	U+2100–214F	ℝ ℕ ℤ ℏ ℓ ™	✅ already allowed
Arrows	U+2190–21FF	→ ← ↔ ⇒	✅ already allowed

Which should genuinely stay flagged

The check has real value — it should keep flagging:

1. Actual foreign-language letters when the language isn't declared (Cyrillic / CJK / Arabic prose). Keep as-is.
2. Homoglyph / confusable attacks — Latin look-alikes from other scripts: Cyrillic а о е р с х (U+0430…), Greek Α Β Ο (U+0391…), fullwidth Latin. These are the opposite problem: they currently get a pass when that script is allowed, but are exactly what a doc-integrity check should catch. A denied_chars / confusables list would let us keep them flagged even inside an allowed script.

So: allow notation blocks by default, keep scripts opt-in, and add an explicit deny path for confusables.

Proposed config

Mirror how check-language already reads [validation] allowed_content_languages / ignore_paths from workspace config. Add symbol governance:

[validation]
allowed_content_languages = ["en"]

# 1. Curated, named symbol profiles (language-neutral notation → vetted block sets).
#    Suggested default ON: ["math", "emoji"]. Others opt-in.
allowed_symbol_sets = ["math", "fractions", "technical", "keyboard", "arrows", "emoji"]

# 2. Explicit allowlist escape hatch for the long tail: literal chars,
#    U+XXXX codepoints, or U+XXXX-U+YYYY ranges.
allowed_chars = ["⌘", "⌥", "U+2070-U+209F", "U+2300-U+23FF"]

# 3. Always-flag list (wins over scripts/profiles) — homoglyph / confusable defense.
denied_chars = ["U+0430", "U+0435", "U+043E", "U+0440", "U+0441"]  # Cyrillic а е о р с

Suggested profile → block mapping:

profile	blocks
math	Greek+Coptic, Math Operators, Supplemental Math Operators, Misc Math Symbols-A/B, Letterlike, Number Forms, Super/Subscripts, Combining Marks for Symbols, Math Alphanumeric Symbols
fractions	Number Forms (U+2150–218F)
technical	Miscellaneous Technical (U+2300–23FF)
keyboard	the ⌘ ⌥ ⎋ ⏎ ⌫ subset of Misc Technical
arrows	Arrows, Supplemental Arrows-A/B, Misc Symbols and Arrows
emoji	current _COMMON_RANGES + Variation Selectors (U+FE00–FE0F)

Minimum viable fix

If the full config surface is too much, the smallest valuable change is:

1. Move math + technical + super/subscripts + number forms into the default-allowed set (they're notation, not language), and add Variation Selectors so ⚠️/emoji-presentation stop tripping; and
2. add a plain allowed_chars per-character allowlist so projects can self-serve the long tail; and
3. skip inline code spans `…` the same way fenced blocks are skipped.

Environment

• engine: v1.5.9
• proxy: constructor-studio 0.1.dev1 (Python 3.14)
• config surface today: [validation] allowed_content_languages, ignore_paths; CLI --languages, --ignore, --quiet