fix(dig): resolution failures raise instead of returning None; export inner_most_key - #84
Merged
Merged
Conversation
… inner_most_key Three related defects in dol.dig, all found while building an S3 adapter that needs to resolve a key through a wrapped store. 1. `inner_most_key(store, k)` returned **None** when no layer of the chain defined `_id_of_key`. Callers use the result as a key or a path, so the None surfaced far from its cause -- as a URL ending in `/None`, or as `TypeError: expected str, bytes or os.PathLike object, not NoneType`. It now raises AttributeError naming the method and the argument. Pass `default=` to opt out. 2. `store_trans_path` hardcoded `unravel_key` for its recursive step, so it applied `_id_of_key` at every layer below the first regardless of the `method` argument. That is the acknowledged TODO on `inner_most_val`: it applied `_data_of_obj` at the top layer and `_id_of_key` below. Now recurses with the method it was given, so `inner_most_val` does what its name says. 3. `inner_most_key` / `unravel_key` are now exported from `dol`. They were importable only from `dol.dig` while being the documented answer to resolving a key through a wrap, which pushed every adapter into a private-module import. Also fixes the two bugs (1) exposed in `MakeMissingDirsStoreMixin.__setitem__`, whose recovery path could never have worked: it fed the None to `os.path.dirname`, and then passed keyword-only `verbose` positionally to `ensure_dir`. Both raised TypeError. It now resolves with `default=k` -- correct, since the mixin targets persisters whose keys ARE full paths and which therefore define no `_id_of_key`. Tests: 6 new cases in test_dig.py covering the raise, the `default=` escape, the recursion fix, the export, and a MakeMissingDirsStoreMixin round-trip. Verified: full suite 511 passed / 3 skipped. Dependents gate byte-for-byte identical to HEAD for every dependent whose backend SDK is installed (xdol 4, pdfdol 23+1, chromadol 2, focal 1, dol_cookbook 1, ftpdol 1); the rest fail on absent SDKs both before and after.
This was referenced Aug 10, 2026
Adversarial-review finding on the `default=k` fallback added to `MakeMissingDirsStoreMixin`. When the mixin targets a persister with relative keys and no `_id_of_key`, `default=k` yields a relative dirname, and `ensure_dir` then creates it under the process CWD -- outside the store -- after which the write still fails. Now guarded by `os.path.isabs`: a relative dirname means the key was never resolved to a filepath, so the original write error is re-raised untouched. Severity was capped by the fact that this recovery path was 100% dead before this PR (it raised TypeError unconditionally on the keyword-only `verbose`), so nothing could have depended on it -- but "fails and leaves a stray directory in your CWD" is worse than "fails", and this is a mixin dol still exports. Test added. 517 passed / 3 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three related defects in
dol.dig, found while building an S3 adapter that needs to resolve akey through a wrapped store.
1.
inner_most_keyreturnedNoneon a resolution failureWhen no layer of the chain defines
_id_of_key,last_elementover an empty generator returnedNone. Callers use the result as a key or a path, so it surfaced far from its cause:That is what turns a presigned URL into
https://.../None, and what makesMakeMissingDirsStoreMixindie withTypeError: expected str, bytes or os.PathLike object, not NoneType. It now raisesAttributeErrornaming the method and the argument.default=optsout.
2.
store_trans_pathignored its ownmethodargument when recursingLine 43 hardcoded
unravel_key, so_id_of_keywas applied at every layer below the firstregardless of what was asked for. This is the acknowledged TODO on
inner_most_val— it applied_data_of_objat the top layer and_id_of_keybelow it. Now recurses with the given method:3.
inner_most_key/unravel_keyare now exported fromdolThey were importable only from
dol.dig, while being the documented answer to "resolve a keythrough a wrap" — which pushed every
*doladapter into a private-module import.Also: two bugs in
MakeMissingDirsStoreMixin.__setitem__Its recovery path could never have worked. It fed the
Nonefrom (1) intoos.path.dirname,and then passed keyword-only
verbosepositionally toensure_dir. Both raisedTypeError, sothe mixin turned a recoverable write error into an unrelated crash. Now resolves with
default=k— correct, because the mixin targets persisters whose keys are full paths andwhich therefore define no
_id_of_key.Compatibility
The only behaviour change is (1), and it converts a silent wrong answer into a loud one, in line
with dol's No Silent Failures convention. Ecosystem callers of
inner_most_key(xdol,tapyoca,graze's vendored copy,py2store's star re-export) all pass stores that define_id_of_key, so none of them reach the new raise.inner_most_val/unravel_valhave nocallers outside dol.
Verification
test_dig.py: the raise, thedefault=escape, the recursion fix, the export,resolution through a real wrap, and a
MakeMissingDirsStoreMixinround-trip.installed here — xdol 4, pdfdol 23+1 skipped, chromadol 2, focal 1, dol_cookbook 1, ftpdol 1.
mongodol/sqldol/hfdol/dropboxdol/sshdol fail on absent SDKs (pymongo, sqlalchemy, datasets,
dropbox, paramiko) both before and after.
Needed by the s3dol v1 redesign (i2mint/s3dol#17), which resolves keys through possibly-wrapped
stores and cannot depend on a private module or a silent
None.