fix(cli): stats told operators to set a variable that does not exist - #160
Merged
Conversation
`agentmetry stats` answered a disabled audit export with "enable AGENTMETRY_AUDIT_EXPORT". There is no such setting. The only alias is AGENTMETRY_AUDIT_EXPORT_ENABLED, so the name appeared exactly once in the whole codebase: in the string telling somebody to set it. The cost is the worst one this project has: the operator follows the instruction, the variable does nothing, the trail stays empty, and nothing says why. A recorder that is off while the operator believes it is on is the failure the product exists to prevent. Found by contrast rather than by testing. #139 added the same message to `agentmetry detections`, named the variable correctly, and only then did anyone read the older one. So the guard is not a string comparison. `test_cli_env_var_names.py` walks every string literal the CLI prints, pulls out anything shaped like an AGENTMETRY_ variable, and checks it against what the package actually reads: the aliases declared on `Settings`, plus names read straight from `os.environ`, because AGENTMETRY_AGT_HMAC_KEY is real and never passes through Settings. Writing that second half is what stopped this test from being wrong in the other direction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
agentmetry statsanswered a disabled audit export with:There is no such setting. The only alias is
AGENTMETRY_AUDIT_EXPORT_ENABLED,declared on
Settingsincore/config.py.AGENTMETRY_AUDIT_EXPORTappearedexactly once in the entire codebase: inside the string telling somebody to set
it.
The cost is the worst shape this project has. The operator reads the message,
sets the variable, restarts, and nothing changes. The trail stays empty and
nothing says why. A recorder that is off while its operator believes it is on is
the failure the product exists to prevent.
How it was found
By contrast, not by testing. #139
added the same message to
agentmetry detectionsand named the variablecorrectly. Reviewing that commit is the only reason anyone read the older one.
The guard
A string comparison would pin this instance and miss the next. So
test_cli_env_var_names.pywalks every string literal the CLI prints, extractsanything shaped like an
AGENTMETRY_variable, and checks it against what thepackage actually reads.
"Actually reads" needed both halves. The aliases declared on
Settingsare notthe whole set:
AGENTMETRY_AGT_HMAC_KEYis read straight fromos.environatthe point of use and never passes through
Settings. The first version of thistest flagged it as fake. Getting that right is what stops the guard from being
wrong in the opposite direction and training people to ignore it.
Verified both ways: the test passes on this branch, and reintroducing the old
name fails both assertions.
Checks
Nothing frozen is touched, fingerprint unchanged at
15846a0915769d4a. 1,192tests pass, ruff clean, benchmark 0 missed and 0 false positives.