Skip to content

Port bin/secret and bin/herdr-reload-all to Python (selective, not wholesale) - #96

Closed
jfmercer wants to merge 1 commit into
masterfrom
port-secret-and-herdr-reload-to-python
Closed

Port bin/secret and bin/herdr-reload-all to Python (selective, not wholesale)#96
jfmercer wants to merge 1 commit into
masterfrom
port-secret-and-herdr-reload-to-python

Conversation

@jfmercer

Copy link
Copy Markdown
Owner

The Python half of the backlog, scoped by measurement rather than by "as much as possible".

Why only these two

per invocation
bash -c true 12.7 ms
python3 -c pass 39.7 ms
realistic wrapper, sh 26.2 ms
same wrapper, Python 65.1 ms

bin/ is on $PATH, so that's per-invocation. The twelve git-* helpers are 1–13 lines each and get typed interactively all day — porting them would make them slower and longer for nothing. Same for bupdate (3 lines), update-discord (2), start-bloodhound (1).

These two earn it:

  • secret — 134 lines, and the env-bundle quoting is the highest-risk logic in the repo. Security-relevant, and exactly the sort of thing that should be structured and testable.
  • herdr-reload-all — the whole job is parsing a JSON envelope and branching on it. That was three chained jq calls with || true to stop set -e firing on expected misses. Also drops the jq dependency.

The startup-path problem, and how it's handled

secret env runs on every shell start via ~/.localrc. Porting it naively would have cost ~37 ms per shell (60.8 → 98.3 ms), a ~20% regression on the 180 ms reached yesterday — and it would have contradicted the very rule that scoped this migration.

So ~/.localrc now inlines the equivalent instead of calling the wrapper:

eval "$(security find-generic-password -a "${SECRET_ACCOUNT:-$USER}" \
        -s "${SECRET_ENV_NAME:-shell_env}" -w 2>/dev/null | base64 -d)"

Measured 160 ms after the change — slightly better than before, because this also drops a bash fork.

The tradeoff is real and worth naming: ~/.localrc now knows the storage format, so a format change touches two places. secret env remains the canonical reader and the thing to verify against; both files carry a comment saying so, and a test asserts they agree byte-for-byte.

Verification

A/B against the old bash implementation, on values containing single quotes, double quotes, $, backticks, backslashes, tabs, globs and unicode:

  • Python reads bash-written bundles identically
  • bash reads Python-written bundles identically
  • generated export lines are byte-identical
  • eval round-trip verified in zsh, bash and sh

The ~/.localrc rewrite is lossless — all 15 exported variables identical by name and SHA-256 of value, captured in a clean env -i shell before and after (env -i matters; my first attempt at that baseline was contaminated by inherited environment).

herdr-reload-all driven through every branch with a fake herdr on PATH: happy path (correctly skipping agent panes, 2 of 3), protocol_mismatch, other error codes, non-JSON output, only-agent-panes, and herdr absent.

A pre-existing limitation found while testing

security find-generic-password -w returns a hex dump rather than the value when a secret contains non-printable-ASCII bytes — a tab, or any multibyte UTF-8. I initially read this as a port bug; it isn't. The raw CLI does it below any wrapper, and the old shell version behaved identically. Now documented in the docstring. API tokens are ASCII-printable so it doesn't bite, but don't store prose or unicode expecting it back verbatim.

CI consequence

Both files now fall out of shellcheck's shebang-based selection and into the py_compile step — which is why that step was added before this change rather than alongside it. Local run: zsh -n 25 files clean, shellcheck 22 files clean, py_compile 3 Python scripts, bump-deps --self-test PASS, gitleaks clean.

Not in the diff

~/.localrc is machine-local and untracked, so its rewrite isn't here — only the tooling and the documented reason. Backup at ~/.localrc.pre-pyport.bak.

Remaining backlog item: unit tests. This port is what makes them worth writing — the quoting behaviour verified by hand above currently lives nowhere, and secret is now in a language where capturing it is straightforward.

The selective half of the migration: port the two bin/ scripts with real logic,
leave the thin wrappers in shell. Measured here, python3 starts in ~40 ms against
bash's ~13 ms, and a realistic wrapper 65 ms against 26 ms -- so converting the
twelve 1-13 line git-* helpers, which get typed interactively all day, would make
them slower and longer for nothing.

secret (134 lines of bash) earns it: the env-bundle quoting is the highest-risk
logic in the repo, and it is security-relevant. herdr-reload-all earns it because
the whole job was parsing a JSON envelope and branching on it -- three chained jq
calls with `|| true` to stop set -e firing on expected misses. It also drops the
jq dependency.

secret env is on the shell-startup path via ~/.localrc, and porting it would have
cost ~37 ms per shell -- a ~20% regression on the 180 ms we reached yesterday.
So ~/.localrc now inlines the equivalent `security ... | base64 -d` instead of
calling the wrapper. Startup measured 160 ms after the change, slightly better
than before, because that also drops a bash fork. The tradeoff is that ~/.localrc
now knows the storage format; `secret env` stays the canonical reader and the
thing to verify against, and both files say so.

Verified rather than assumed:

- A/B against the old bash implementation on values containing single quotes,
  double quotes, $, backticks, backslashes, tabs, globs and unicode. Python reads
  bash-written bundles and bash reads Python-written bundles identically, and the
  generated export lines are byte-identical.
- eval round-trip in zsh, bash and sh: values come back exactly.
- ~/.localrc rewrite is lossless -- all 15 exported variables identical by name
  and sha256 of value, captured in a clean `env -i` shell before and after.
- herdr-reload-all driven through every branch with a fake herdr on PATH: happy
  path (skipping agent panes), protocol_mismatch, other error codes, non-JSON
  output, only-agent-panes, and herdr absent.

Found a pre-existing limitation while testing, now documented in the docstring:
`security find-generic-password -w` returns a hex dump rather than the value when
a secret contains non-printable-ASCII bytes. That happens in the CLI itself and
the old shell version behaved identically, so it is not a regression -- but it
means unicode or tabs do not round-trip. API tokens are ASCII, so it does not
bite in practice.

- CLAUDE.md: record the shell-vs-Python rule with the measurements behind it, mark
  both scripts as Python, and document why ~/.localrc bypasses the wrapper.

Note the CI consequence: both files now fall out of shellcheck's shebang-based
selection and into the py_compile step added earlier, which is why that step
existed before this change.
@jfmercer

Copy link
Copy Markdown
Owner Author

I don't like these changes, so I'm not accepting them.

@jfmercer jfmercer closed this Jul 28, 2026
@jfmercer
jfmercer deleted the port-secret-and-herdr-reload-to-python branch July 28, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant