Skip to content

fix(i18n/audit): eliminate false positives in raw-site classification#656

Open
thomwebb wants to merge 3 commits into
mpfaffenberger:mainfrom
thomwebb:fix/i18n-audit-false-positives
Open

fix(i18n/audit): eliminate false positives in raw-site classification#656
thomwebb wants to merge 3 commits into
mpfaffenberger:mainfrom
thomwebb:fix/i18n-audit-false-positives

Conversation

@thomwebb

Copy link
Copy Markdown
Contributor

Two bugs fixed

1. f-string false positives in _has_string_literal

Previously any ast.JoinedStr (f-string) was unconditionally classified as raw, meaning pure-variable forms like f"{result}" or f" {msg}" were counted as un-extracted literals even though they contain zero translatable content.

Fix: walk the f-string's values list and only return True when at least one ast.Constant part contains non-whitespace text (e.g. f"Error: {e}" still flags correctly; f"{var}" becomes dynamic).

Impact: 23 false positives reclassified as dynamic (raw: 1302 → 1279).

2. Single-file path silently returning 0 sites in _iter_py_files

os.walk(file_path) on a regular file yields nothing, so:

python -m code_puppy.i18n.audit code_puppy/command_line/session_commands.py

always reported 0 sites (while the full-tree scan correctly found 33 in that file). Confusing and broke the per-file workflow.

Fix: detect a file path up front and yield it directly when it ends in .py, before falling through to os.walk.

Tests added (5 new → 25 total)

Test What it guards
test_fstring_pure_variable_is_dynamic f"{var}" → dynamic
test_fstring_whitespace_only_literal_is_dynamic f" {msg}" → dynamic
test_fstring_with_content_and_variable_is_raw f"Error: {e}" → raw (regression guard)
test_fstring_with_only_arrow_is_raw non-whitespace punctuation counts
test_single_file_path_is_accepted per-file audit finds raw sites
test_single_file_pure_variable_fstring_not_raw both fixes together on a single file

@thomwebb thomwebb left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial Review — Automated (code-puppy-4aeaa3)

Verdict: CONDITIONAL PASS — the two core bugs are correctly fixed; two follow-on gaps in the same bug class need addressing.


Must Fix

1. New false negative: f"{'literal string'}" silently drops from audit

The fix scans node.values of ast.JoinedStr for direct ast.Constant children. But a constant inside a FormattedValue (e.g. f"{'Error: connection refused'}") produces:

JoinedStr(values=[
    FormattedValue(value=Constant('Error: connection refused'))  # one level deeper
])

There is no direct ast.Constant in node.values — the check returns Falsedynamic. The old code caught this correctly (return True for any JoinedStr). Verified on Python 3.14.

In practice nobody writes f"{'literal'}" instead of 'literal', so real-world impact is low — but the auditor now has a documented false-negative class. Fix or explicitly document the limitation in the docstring:

# Also recurse into FormattedValue.value for the rare f"{'literal'}" pattern
or (isinstance(v, ast.FormattedValue) and isinstance(v.value, ast.Constant)
    and isinstance(v.value.value, str) and v.value.value.strip())

2. Non-existent path silently returns coverage: 100%, raw: 0

_iter_py_files('/nonexistent/missing.py')[] with no output. This is the same class of bug as the one being fixed (silent zero report), just for a different bad input. A developer who fat-fingers a path gets a misleadingly clean report.

Fix: add an explicit warning to stderr when the path is neither a file nor a directory:

if not os.path.isdir(root):
    print(f"warning: {root!r} does not exist", file=sys.stderr)
    return

Should Fix

3. PR description says "5 new tests" — it is actually 6

test_single_file_pure_variable_fstring_not_raw is in the diff but missing from the PR description's table. Minor credibility issue for a review-focused PR. (pytest correctly reports 25 passed — the count in the description is the error.)

4. audit_tree docstring not updated

"""Audit every Python module under ``root``.""" — the function now also accepts a single .py file path. Update to: """Audit every Python module under ``root``, or ``root`` itself when it is a ``.py`` file."""


Confirmed Non-Issues (adversarial probes that cleared)

  • Nested f-strings like f"Error in {f'{m}.{f}'}" — outer JoinedStr.values[0] is a direct ast.Constant('Error in '); fix handles correctly
  • Multiline f-strings with \n/\tstr.strip() removes all ASCII whitespace; ' \n '.strip() == '' → dynamic
  • Symlinks to .py files — os.path.isfile follows symlinks; works on macOS
  • Empty directories, directories with no .py files — correctly return empty iterator
  • Non-.py file passed directly — silently skipped (acceptable; document if desired)

The core logic is correct and minimal. The any(isinstance(v, ast.Constant) and ... v.value.strip()) pattern is clean and Pythonic. The 23 false positives it removes are all genuine non-translatable pure-variable f-strings. The regression tests are real and explicitly documented.

@thomwebb thomwebb left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review feedback addressed — commit da025f70

Automated fix pass (code-puppy-4aeaa3). Both "must fix" items resolved.

Landed:

  • FormattedValue recursion: _has_string_literal for JoinedStr now also matches FormattedValue whose .value is a non-empty string Constant. Kills the f"{'Error: connection refused'}" false negative.
  • Non-existent path guard: audit_tree checks os.path.isdir(root) or os.path.isfile(root) at the top; on failure prints warning: '<path>' does not exist to stderr and returns an empty Report instead of the misleading silent "100% coverage, 0 raw".
  • Docstring updated: audit_tree now reads """Audit every Python module under \`root``, or ``root`` itself when it is a ``.py`` file."""`
  • Two regression tests added:
    • test_fstring_wrapped_literal_is_raw — builds AST for the FormattedValue-wrapped literal case and asserts _classify returns "raw"
    • test_nonexistent_path_warns_and_returns_empty — asserts empty result + stderr warning

Verification:

  • pytest tests/i18n/test_i18n_audit.py -q → 27 passed (25 prior + 2 new)
  • ruff check --fix / ruff format → clean
  • uv.lock side-effect from uv run was reverted before commit; only the two authorized files landed

CI should re-run on the new commit.

TJ Webb added 3 commits July 20, 2026 13:27
Two bugs fixed:

1. f-string false positives (_has_string_literal)
   Previously any ast.JoinedStr was unconditionally classified as 'raw',
   meaning pure-variable f-strings like f"{result}" or f"  {msg}" were
   counted as un-extracted literals even though they contain no translatable
   content. Fixed by walking the f-string's values list and only returning
   True when at least one ast.Constant part has non-whitespace text.
   Impact: 23 false positives reclassified as 'dynamic' (1302 -> 1279 raw).

2. Single-file path silently returning 0 sites (_iter_py_files)
   os.walk(file_path) on a regular file yields nothing, so
   'python -m code_puppy.i18n.audit path/to/module.py' always reported
   0 sites. Fixed by detecting a file path up front and yielding it
   directly if it ends in .py.

Tests added (5 new, 25 total):
  test_fstring_pure_variable_is_dynamic
  test_fstring_whitespace_only_literal_is_dynamic
  test_fstring_with_content_and_variable_is_raw
  test_fstring_with_only_arrow_is_raw
  test_single_file_path_is_accepted
  test_single_file_pure_variable_fstring_not_raw

All 25 audit tests pass. ruff clean.
Upstream commit f0dab86 added reasoning_context and reasoning_mode to
supported_settings for gpt-5.6* models but did not update this test.
Split the expected list so gpt-5.6* variants get the extra fields.

This piggyback will drop from the branch once a standalone upstream fix lands.
@thomwebb
thomwebb force-pushed the fix/i18n-audit-false-positives branch from da025f7 to f5aa921 Compare July 20, 2026 20:27

@thomwebb thomwebb left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebased + piggyback test fix

Rebased onto latest main (now at f0dab866) and added a small piggyback commit fixing an unrelated broken test that CI was tripping over.

Context

Upstream commit f0dab866 (feat: add GPT-5.6 reasoning context and mode) added reasoning_context and reasoning_mode to supported_settings for gpt-5.6* models in chatgpt_oauth/utils.py but did not update tests/plugins/test_chatgpt_oauth_utils.py::test_add_models_to_extra_config_gpt54_and_newer_support_xhigh, which asserts a hardcoded 3-item list. Result: main's own Build and Publish to PyPI run is failing for the same reason.

What the piggyback commit does

Splits the assertion so codex-gpt-5.6-* variants expect the extra fields:

expected_settings = ["reasoning_effort", "summary", "verbosity"]
if model_name.startswith("codex-gpt-5.6"):
    expected_settings.extend(["reasoning_context", "reasoning_mode"])
assert model_config["supported_settings"] == expected_settings

Verified locally: test passes on this branch.

Cleanup

This commit is a piggyback and will drop from the branch automatically once the same fix lands on upstream main via a rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant