Port bin/secret and bin/herdr-reload-all to Python (selective, not wholesale) - #96
Closed
jfmercer wants to merge 1 commit into
Closed
Port bin/secret and bin/herdr-reload-all to Python (selective, not wholesale)#96jfmercer wants to merge 1 commit into
jfmercer wants to merge 1 commit into
Conversation
The selective half of the migration: port the two bin/ scripts with real logic, leave the thin wrappers in shell. Measured here, python3 starts in ~40 ms against bash's ~13 ms, and a realistic wrapper 65 ms against 26 ms -- so converting the twelve 1-13 line git-* helpers, which get typed interactively all day, would make them slower and longer for nothing. secret (134 lines of bash) earns it: the env-bundle quoting is the highest-risk logic in the repo, and it is security-relevant. herdr-reload-all earns it because the whole job was parsing a JSON envelope and branching on it -- three chained jq calls with `|| true` to stop set -e firing on expected misses. It also drops the jq dependency. secret env is on the shell-startup path via ~/.localrc, and porting it would have cost ~37 ms per shell -- a ~20% regression on the 180 ms we reached yesterday. So ~/.localrc now inlines the equivalent `security ... | base64 -d` instead of calling the wrapper. Startup measured 160 ms after the change, slightly better than before, because that also drops a bash fork. The tradeoff is that ~/.localrc now knows the storage format; `secret env` stays the canonical reader and the thing to verify against, and both files say so. Verified rather than assumed: - A/B against the old bash implementation on values containing single quotes, double quotes, $, backticks, backslashes, tabs, globs and unicode. Python reads bash-written bundles and bash reads Python-written bundles identically, and the generated export lines are byte-identical. - eval round-trip in zsh, bash and sh: values come back exactly. - ~/.localrc rewrite is lossless -- all 15 exported variables identical by name and sha256 of value, captured in a clean `env -i` shell before and after. - herdr-reload-all driven through every branch with a fake herdr on PATH: happy path (skipping agent panes), protocol_mismatch, other error codes, non-JSON output, only-agent-panes, and herdr absent. Found a pre-existing limitation while testing, now documented in the docstring: `security find-generic-password -w` returns a hex dump rather than the value when a secret contains non-printable-ASCII bytes. That happens in the CLI itself and the old shell version behaved identically, so it is not a regression -- but it means unicode or tabs do not round-trip. API tokens are ASCII, so it does not bite in practice. - CLAUDE.md: record the shell-vs-Python rule with the measurements behind it, mark both scripts as Python, and document why ~/.localrc bypasses the wrapper. Note the CI consequence: both files now fall out of shellcheck's shebang-based selection and into the py_compile step added earlier, which is why that step existed before this change.
Owner
Author
|
I don't like these changes, so I'm not accepting them. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Python half of the backlog, scoped by measurement rather than by "as much as possible".
Why only these two
bash -c truepython3 -c passbin/is on$PATH, so that's per-invocation. The twelvegit-*helpers are 1–13 lines each and get typed interactively all day — porting them would make them slower and longer for nothing. Same forbupdate(3 lines),update-discord(2),start-bloodhound(1).These two earn it:
secret— 134 lines, and the env-bundle quoting is the highest-risk logic in the repo. Security-relevant, and exactly the sort of thing that should be structured and testable.herdr-reload-all— the whole job is parsing a JSON envelope and branching on it. That was three chainedjqcalls with|| trueto stopset -efiring on expected misses. Also drops thejqdependency.The startup-path problem, and how it's handled
secret envruns on every shell start via~/.localrc. Porting it naively would have cost ~37 ms per shell (60.8 → 98.3 ms), a ~20% regression on the 180 ms reached yesterday — and it would have contradicted the very rule that scoped this migration.So
~/.localrcnow inlines the equivalent instead of calling the wrapper:Measured 160 ms after the change — slightly better than before, because this also drops a bash fork.
The tradeoff is real and worth naming:
~/.localrcnow knows the storage format, so a format change touches two places.secret envremains the canonical reader and the thing to verify against; both files carry a comment saying so, and a test asserts they agree byte-for-byte.Verification
A/B against the old bash implementation, on values containing single quotes, double quotes,
$, backticks, backslashes, tabs, globs and unicode:exportlines are byte-identicalevalround-trip verified in zsh, bash and shThe
~/.localrcrewrite is lossless — all 15 exported variables identical by name and SHA-256 of value, captured in a cleanenv -ishell before and after (env -imatters; my first attempt at that baseline was contaminated by inherited environment).herdr-reload-alldriven through every branch with a fakeherdronPATH: happy path (correctly skipping agent panes, 2 of 3),protocol_mismatch, other error codes, non-JSON output, only-agent-panes, andherdrabsent.A pre-existing limitation found while testing
security find-generic-password -wreturns a hex dump rather than the value when a secret contains non-printable-ASCII bytes — a tab, or any multibyte UTF-8. I initially read this as a port bug; it isn't. The raw CLI does it below any wrapper, and the old shell version behaved identically. Now documented in the docstring. API tokens are ASCII-printable so it doesn't bite, but don't store prose or unicode expecting it back verbatim.CI consequence
Both files now fall out of
shellcheck's shebang-based selection and into thepy_compilestep — which is why that step was added before this change rather than alongside it. Local run:zsh -n25 files clean,shellcheck22 files clean,py_compile3 Python scripts,bump-deps --self-testPASS, gitleaks clean.Not in the diff
~/.localrcis machine-local and untracked, so its rewrite isn't here — only the tooling and the documented reason. Backup at~/.localrc.pre-pyport.bak.Remaining backlog item: unit tests. This port is what makes them worth writing — the quoting behaviour verified by hand above currently lives nowhere, and
secretis now in a language where capturing it is straightforward.