fix(i18n/audit): eliminate false positives in raw-site classification#656
fix(i18n/audit): eliminate false positives in raw-site classification#656thomwebb wants to merge 3 commits into
Conversation
thomwebb
left a comment
There was a problem hiding this comment.
Adversarial Review — Automated (code-puppy-4aeaa3)
Verdict: CONDITIONAL PASS — the two core bugs are correctly fixed; two follow-on gaps in the same bug class need addressing.
Must Fix
1. New false negative: f"{'literal string'}" silently drops from audit
The fix scans node.values of ast.JoinedStr for direct ast.Constant children. But a constant inside a FormattedValue (e.g. f"{'Error: connection refused'}") produces:
JoinedStr(values=[
FormattedValue(value=Constant('Error: connection refused')) # one level deeper
])
There is no direct ast.Constant in node.values — the check returns False → dynamic. The old code caught this correctly (return True for any JoinedStr). Verified on Python 3.14.
In practice nobody writes f"{'literal'}" instead of 'literal', so real-world impact is low — but the auditor now has a documented false-negative class. Fix or explicitly document the limitation in the docstring:
# Also recurse into FormattedValue.value for the rare f"{'literal'}" pattern
or (isinstance(v, ast.FormattedValue) and isinstance(v.value, ast.Constant)
and isinstance(v.value.value, str) and v.value.value.strip())2. Non-existent path silently returns coverage: 100%, raw: 0
_iter_py_files('/nonexistent/missing.py') → [] with no output. This is the same class of bug as the one being fixed (silent zero report), just for a different bad input. A developer who fat-fingers a path gets a misleadingly clean report.
Fix: add an explicit warning to stderr when the path is neither a file nor a directory:
if not os.path.isdir(root):
print(f"warning: {root!r} does not exist", file=sys.stderr)
returnShould Fix
3. PR description says "5 new tests" — it is actually 6
test_single_file_pure_variable_fstring_not_raw is in the diff but missing from the PR description's table. Minor credibility issue for a review-focused PR. (pytest correctly reports 25 passed — the count in the description is the error.)
4. audit_tree docstring not updated
"""Audit every Python module under ``root``.""" — the function now also accepts a single .py file path. Update to: """Audit every Python module under ``root``, or ``root`` itself when it is a ``.py`` file."""
Confirmed Non-Issues (adversarial probes that cleared)
- Nested f-strings like
f"Error in {f'{m}.{f}'}"— outerJoinedStr.values[0]is a directast.Constant('Error in '); fix handles correctly - Multiline f-strings with
\n/\t—str.strip()removes all ASCII whitespace;' \n '.strip() == ''→ dynamic - Symlinks to
.pyfiles —os.path.isfilefollows symlinks; works on macOS - Empty directories, directories with no
.pyfiles — correctly return empty iterator - Non-
.pyfile passed directly — silently skipped (acceptable; document if desired)
The core logic is correct and minimal. The any(isinstance(v, ast.Constant) and ... v.value.strip()) pattern is clean and Pythonic. The 23 false positives it removes are all genuine non-translatable pure-variable f-strings. The regression tests are real and explicitly documented.
thomwebb
left a comment
There was a problem hiding this comment.
Review feedback addressed — commit da025f70
Automated fix pass (code-puppy-4aeaa3). Both "must fix" items resolved.
Landed:
- FormattedValue recursion:
_has_string_literalforJoinedStrnow also matchesFormattedValuewhose.valueis a non-empty stringConstant. Kills thef"{'Error: connection refused'}"false negative. - Non-existent path guard:
audit_treechecksos.path.isdir(root) or os.path.isfile(root)at the top; on failure printswarning: '<path>' does not existto stderr and returns an emptyReportinstead of the misleading silent "100% coverage, 0 raw". - Docstring updated:
audit_treenow reads"""Audit every Python module under \`root``, or ``root`` itself when it is a ``.py`` file."""` - Two regression tests added:
test_fstring_wrapped_literal_is_raw— builds AST for the FormattedValue-wrapped literal case and asserts_classifyreturns"raw"test_nonexistent_path_warns_and_returns_empty— asserts empty result + stderr warning
Verification:
pytest tests/i18n/test_i18n_audit.py -q→ 27 passed (25 prior + 2 new)ruff check --fix/ruff format→ cleanuv.lockside-effect fromuv runwas reverted before commit; only the two authorized files landed
CI should re-run on the new commit.
Two bugs fixed:
1. f-string false positives (_has_string_literal)
Previously any ast.JoinedStr was unconditionally classified as 'raw',
meaning pure-variable f-strings like f"{result}" or f" {msg}" were
counted as un-extracted literals even though they contain no translatable
content. Fixed by walking the f-string's values list and only returning
True when at least one ast.Constant part has non-whitespace text.
Impact: 23 false positives reclassified as 'dynamic' (1302 -> 1279 raw).
2. Single-file path silently returning 0 sites (_iter_py_files)
os.walk(file_path) on a regular file yields nothing, so
'python -m code_puppy.i18n.audit path/to/module.py' always reported
0 sites. Fixed by detecting a file path up front and yielding it
directly if it ends in .py.
Tests added (5 new, 25 total):
test_fstring_pure_variable_is_dynamic
test_fstring_whitespace_only_literal_is_dynamic
test_fstring_with_content_and_variable_is_raw
test_fstring_with_only_arrow_is_raw
test_single_file_path_is_accepted
test_single_file_pure_variable_fstring_not_raw
All 25 audit tests pass. ruff clean.
Upstream commit f0dab86 added reasoning_context and reasoning_mode to supported_settings for gpt-5.6* models but did not update this test. Split the expected list so gpt-5.6* variants get the extra fields. This piggyback will drop from the branch once a standalone upstream fix lands.
da025f7 to
f5aa921
Compare
thomwebb
left a comment
There was a problem hiding this comment.
Rebased + piggyback test fix
Rebased onto latest main (now at f0dab866) and added a small piggyback commit fixing an unrelated broken test that CI was tripping over.
Context
Upstream commit f0dab866 (feat: add GPT-5.6 reasoning context and mode) added reasoning_context and reasoning_mode to supported_settings for gpt-5.6* models in chatgpt_oauth/utils.py but did not update tests/plugins/test_chatgpt_oauth_utils.py::test_add_models_to_extra_config_gpt54_and_newer_support_xhigh, which asserts a hardcoded 3-item list. Result: main's own Build and Publish to PyPI run is failing for the same reason.
What the piggyback commit does
Splits the assertion so codex-gpt-5.6-* variants expect the extra fields:
expected_settings = ["reasoning_effort", "summary", "verbosity"]
if model_name.startswith("codex-gpt-5.6"):
expected_settings.extend(["reasoning_context", "reasoning_mode"])
assert model_config["supported_settings"] == expected_settingsVerified locally: test passes on this branch.
Cleanup
This commit is a piggyback and will drop from the branch automatically once the same fix lands on upstream main via a rebase.
Two bugs fixed
1. f-string false positives in
_has_string_literalPreviously any
ast.JoinedStr(f-string) was unconditionally classified asraw, meaning pure-variable forms likef"{result}"orf" {msg}"were counted as un-extracted literals even though they contain zero translatable content.Fix: walk the f-string's
valueslist and only returnTruewhen at least oneast.Constantpart contains non-whitespace text (e.g.f"Error: {e}"still flags correctly;f"{var}"becomesdynamic).Impact: 23 false positives reclassified as
dynamic(raw: 1302 → 1279).2. Single-file path silently returning 0 sites in
_iter_py_filesos.walk(file_path)on a regular file yields nothing, so:always reported 0 sites (while the full-tree scan correctly found 33 in that file). Confusing and broke the per-file workflow.
Fix: detect a file path up front and yield it directly when it ends in
.py, before falling through toos.walk.Tests added (5 new → 25 total)
test_fstring_pure_variable_is_dynamicf"{var}"→ dynamictest_fstring_whitespace_only_literal_is_dynamicf" {msg}"→ dynamictest_fstring_with_content_and_variable_is_rawf"Error: {e}"→ raw (regression guard)test_fstring_with_only_arrow_is_rawtest_single_file_path_is_acceptedtest_single_file_pure_variable_fstring_not_raw