Key/value transforms live outside dol's hooks: `.items()` and `.values()` on a wrapped dynamodol store silently escape the wrapper's scope

[thorwhalen]

Summary

dynamodol implements its key and value transforms in public, non-dunder methods
(format_get_key, format_get_item, extract_obj_from_data, iter_items, iter_values)
instead of in dol's _id_of_key / _key_of_id / _data_of_obj / _obj_of_data hooks — as the
source's own TODOs already acknowledge. Because dol wraps by delegation, every one of those
methods keeps running bound to the inner, unwrapped store once someone puts a key codec on top.

The consequence is not academic. DynamoDbBaseReader also ships its own nested ValuesView /
ItemsView (dynamodol/base.py:97-109) that route through iter_values / iter_items, and
dol.base.Store.__init__ copies those view classes onto the wrapper. So on any wrapped
dynamodol store:

list(store)          -> correctly scoped
dict(store)          -> correctly scoped
dict(store.items())  -> NOT scoped: rows from outside the wrapper's key space
list(store.values()) -> NOT scoped, and value decoders are skipped

.items() and .values() are core Mapping API. A user does not have to reach for an exotic
method to get wrong data — they get it from the two most ordinary iteration calls, silently, with
no exception.

Umbrella: i2mint/dol#83. Root cause: i2mint/dol#18.

Status in this package: LATENT (but of the worst kind)

To be precise about the claim: dynamodol as shipped does not trigger this. A census of the
package finds zero key-codec wrappers —

searched for	hits in dynamodol/
mk_relative_path_store	0
KeyCodecs	0
prefixless_view	0
filt_iter	0
wrap_kvs	0
PrefixRelativizationMixin	0
Store	imported at dynamodol/base.py:12, used only in the commented-out TODO at dynamodol/base.py:390

Unwrapped, every store in this package is self-consistent: DynamoDbPrefixReader.__getitem__
adds the prefix, format_get_key strips it, and dict(s) == dict(s.items()).

The bug appears the moment a user does the thing dol exists for — scoping the store down with a
key codec. That is the normal way to build a per-tenant / per-namespace view, and it is the
documented dol idiom.

Mechanism

dol wraps by has-a, not is-a: Wrap holds the leaf in self.store. Dunders
(__getitem__, __setitem__, __delitem__, __contains__, __iter__) are reimplemented on
Store and apply the codecs. Everything else is handed straight to the leaf, by one of two routes:

• Route A — instance wraps. Store.__getattr__ (dol/base.py:742) returns
  getattr(self.store, attr) — a leaf-bound method.
• Route B — class wraps. delegate_to (dol/base.py:416) installs a
  DelegatedAttribute for every name in dir(wrapped); DelegatedAttribute.__get__
  (dol/base.py:279) also returns getattr(instance.store, attr).

Confirmed on the class-wrap, these leaf attributes become DelegatedAttributes:

['extract_obj_from_data', 'filter_kwargs', 'format_get_item', 'format_get_key',
 'iter_items', 'iter_values', 'mk_db', 'partition_key', 'sort_key', 'table', ...]

The amplifier that turns "an odd method returns odd data" into ".items() lies" is
dol/base.py:723-727:

if hasattr(self.store, "ValuesView"):
    self.ValuesView = self.store.ValuesView
if hasattr(self.store, "ItemsView"):
    self.ItemsView = self.store.ItemsView

So wrapper.items() → Store.items() → self.ItemsView(self) → dynamodol's ItemsView with
_mapping = the wrapper → self._mapping.iter_items() → delegated → leaf-bound iter_items
→ raw leaf keys and raw leaf values.

Control test (same store, dol's default views substituted for dynamodol's nested ones):

CONTROL (dol default views):  dict(s.items()) = {'notes.txt': 'ALICE-SECRET'}                 # correct
dynamodol's nested views:     dict(s.items()) = {'alice/notes.txt': ..., 'bob/notes.txt': ...} # leak

That isolates dynamodol/base.py:97-109 as the trigger.

Also note the direction: format_get_key(item) takes a raw DynamoDB record, not a key — so
this is not the "handed the outer unmapped key" variant of dol#83. It is the return-side
variant: the method returns a key in the leaf's key space, which never passes through the
wrapper's _key_of_id. Same root cause, mirrored.

Affected symbols

Symbol	Location	Verdict	Severity
DynamoDbBaseReader.ValuesView / ItemsView	dynamodol/base.py:97, :104	latent	wrong-scope — the amplifier; makes .items()/.values() lie
DynamoDbBaseReader.iter_items	dynamodol/base.py:241	latent	wrong-scope
DynamoDbBaseReader.iter_values	dynamodol/base.py:247	latent	wrong-scope
DynamoDbQueryReader.iter_items / iter_values	dynamodol/partition_query.py:170, :178	latent	wrong-scope
DynamoDbBaseReader.format_get_key	dynamodol/base.py:181	latent	wrong-scope
DynamoDbPartitionReader.format_get_key	dynamodol/partition_query.py:222	latent	wrong-scope — partition scoping in a public method
DynamoDbPrefixReader.format_get_key	dynamodol/partition_query.py:259	latent	wrong-scope — prefix scoping in a public method
DynamoDbBaseReader.format_get_item	dynamodol/base.py:175	latent	value-side
DynamoDbBaseReader.extract_obj_from_data	dynamodol/base.py:166	latent	value-side

Nothing here is destructive. Every mutation path in this package is a dunder —
DynamoDbBasePersister.__setitem__ / __delitem__ (dynamodol/base.py:313, :331) and
DynamoDbPartitionPersister.__setitem__ / __delitem__ (dynamodol/partition_query.py:274,
:285) — and dol maps dunder keys correctly. There is no DynamoDbPrefixPersister and no
non-dunder write or delete method anywhere in the package. No data is destroyed by this bug.
The damage is entirely read-side: wrong and over-broad results.

The TODOs already in the source

The package knows. Verbatim, four times:

dynamodol/base.py:175-176

    def format_get_item(self, item):
        """TODO: replace with _id_of_key, etc."""

dynamodol/base.py:181-182

    def format_get_key(self, item):
        """TODO: replace with _id_of_key, etc."""

dynamodol/partition_query.py:222-223

    def format_get_key(self, item):
        """TODO: replace with _id_of_key, etc."""

dynamodol/partition_query.py:259-260

    def format_get_key(self, item):
        """TODO: replace with _id_of_key, etc."""

And dynamodol/base.py:390:

# TODO class DynamoDbStore(DynamoDbBasePersister, Store): ...

Repro

Real dynamodol classes. The only thing faked is the boto3 transport (an in-memory table),
so this runs with no AWS account and no DynamoDB Local. pip install dynamodol dol.

import botocore.exceptions
from dol import KeyCodecs, filt_iter, Pipe
from dynamodol import DynamoDbPrefixReader


class FakeTable:  # stands in for boto3's Table resource
    def __init__(self, rows): self.rows = rows
    @staticmethod
    def _m(r, key): return all(r.get(k) == v for k, v in key.items())
    def get_item(self, Key=None, **kw):
        for r in self.rows:
            if self._m(r, Key):
                return {"Item": dict(r)}
        raise KeyError(Key)
    def scan(self, **kw):
        return ({"Count": len(self.rows)} if kw.get("Select") == "COUNT"
                else {"Items": [dict(r) for r in self.rows]})
    query = scan


class FakeDb:
    def __init__(self, rows): self._t = FakeTable(rows)
    def create_table(self, **kw):
        raise botocore.exceptions.ClientError({"Error": {"Code": "InUse"}}, "CreateTable")
    def Table(self, name): return self._t


ROWS = [
    {"pk": "part1", "sk": "v1/alice/notes.txt", "value": "ALICE-SECRET"},
    {"pk": "part1", "sk": "v1/bob/notes.txt",   "value": "BOB-SECRET"},
]

def leaf(cls=DynamoDbPrefixReader):
    return cls(db=FakeDb([dict(r) for r in ROWS]), table_name="t",
               key_fields=("pk", "sk"), data_fields=("value",),
               partition="part1", prefix="v1/")

# the standard dol idiom for scoping a store down to one tenant
alice_only = Pipe(filt_iter.prefixes("alice/"), KeyCodecs.prefixed("alice/"))

def show(tag, s):
    print(tag)
    print(f"   list(s)          = {list(s)}")
    print(f"   dict(s)          = {dict(s)}")
    print(f"   dict(s.items())  = {dict(s.items())}")
    print(f"   list(s.values()) = {list(s.values())}")

show("leaf, unwrapped (correct):", leaf())
show("ROUTE A - instance wrap:  ", alice_only(leaf()))
show("ROUTE B - class wrap:     ", leaf(alice_only(DynamoDbPrefixReader)))

Output (dynamodol's own debug prints stripped — see footnote):

leaf, unwrapped (correct):
   list(s)          = ['alice/notes.txt', 'bob/notes.txt']
   dict(s)          = {'alice/notes.txt': 'ALICE-SECRET', 'bob/notes.txt': 'BOB-SECRET'}
   dict(s.items())  = {'alice/notes.txt': 'ALICE-SECRET', 'bob/notes.txt': 'BOB-SECRET'}
   list(s.values()) = ['ALICE-SECRET', 'BOB-SECRET']
ROUTE A - instance wrap:
   list(s)          = ['notes.txt']
   dict(s)          = {'notes.txt': 'ALICE-SECRET'}
   dict(s.items())  = {'alice/notes.txt': 'ALICE-SECRET', 'bob/notes.txt': 'BOB-SECRET'}   <-- LEAK
   list(s.values()) = ['ALICE-SECRET', 'BOB-SECRET']                                       <-- LEAK
ROUTE B - class wrap:
   list(s)          = ['notes.txt']
   dict(s)          = {'notes.txt': 'ALICE-SECRET'}
   dict(s.items())  = {'alice/notes.txt': 'ALICE-SECRET', 'bob/notes.txt': 'BOB-SECRET'}   <-- LEAK
   list(s.values()) = ['ALICE-SECRET', 'BOB-SECRET']                                       <-- LEAK

A value-codec version of the same thing:

from dol import wrap_kvs
wv = wrap_kvs(leaf(), obj_of_data=lambda s: {'text': s})

wv['alice/notes.txt']   # {'text': 'ALICE-SECRET'}          decoder applied
list(wv.values())       # ['ALICE-SECRET', 'BOB-SECRET']    raw str; decoder skipped
list(wv.items())        # [('alice/notes.txt', 'ALICE-SECRET'), ...]  raw

User-visible consequence

A caller who scopes a dynamodol store — per tenant, per user, per namespace, per version prefix —
gets a store where keys() and __getitem__ respect the scope but items() and values() do
not. Concretely:

• Cross-scope disclosure. dict(store.items()) returns rows belonging to every other scope
  in the partition. If the wrapper was the authorization boundary, it isn't one.
• Keys in the wrong space. The keys yielded by items() cannot be fed back into
  store[k] / del store[k] — they are inner-store keys. Round-tripping
  {k: f(v) for k, v in store.items()} back into the store either raises or writes to the wrong
  key.
• Value decoders skipped. values() / items() return the raw stored representation while
  store[k] returns the decoded object. Two paths, two answers, no error.
• dict(store) != dict(store.items()), which violates the Mapping contract that most
  downstream code (and dol's own combinators) assumes.
• Nothing is deleted or overwritten by this bug.

Suggested remediation

The real fix for this package is structural, and it is the one the source TODOs already name:
move the transforms into dol's hooks so they compose, and stop shipping views that bypass them.

1. Delete the nested ValuesView / ItemsView (dynamodol/base.py:97-109). This is the
   single highest-value change, and the control test above shows it alone fixes .items() /
   .values(). dol's default views iterate __iter__ + __getitem__, which are transform-correct
   through the whole chain. If the one-scan-instead-of-N-gets optimisation matters, expose it as
   an explicit standalone function (scan_items(store)) that callers opt into — not as .items(),
   where a wrapped store will silently hand back out-of-scope data.
   (Bonus: those two __contains__ implementations are already broken unwrapped — see footnote.)
2. format_get_key → _key_of_id, and format_get_item / extract_obj_from_data →
   _obj_of_data, keeping the record→field extraction as a private helper
   (_key_from_record). Then Store composes them instead of shadowing them.
3. Move the prefix/partition scoping out of __getitem__ into _id_of_key.
   DynamoDbPrefixReader.__getitem__ (dynamodol/partition_query.py:263-270) hand-rolls
   self.prefix + k; DynamoDbPartitionReader.__getitem__ (:226-234) hand-rolls the partition.
   As _id_of_key, those become composable and every inherited method gets them for free.
4. Land the DynamoDbStore(DynamoDbBasePersister, Store) TODO at dynamodol/base.py:390, so
   the shipped stores are real Stores with the hooks wired.

Stop-gap escape hatch (for any future public method that takes a key)

No current dynamodol method takes a user key, so this doesn't apply to today's code — but it is
the general pattern for dol#83, worth recording before someone adds a
describe_key(k) / ttl_for(k) style method:

from dol import wrapped_self       # exported from dol
from dol.dig import inner_most_key # NOT exported from dol — import from dol.dig

def describe_key(self, k):
    _id = inner_most_key(wrapped_self(self), k)
    if not isinstance(_id, str):   # mandatory: see trap 2
        _id = self._id_of_key(k)
    ...

Two traps, both real:

• It walks the WHOLE chain, including the leaf's own _id_of_key. So it replaces
  self._id_of_key(k) — never compose the two, or the key gets transformed twice.
• It returns None, silently, when no layer in the chain defines _id_of_key. Verified on an
  unwrapped DynamoDbPrefixReader: inner_most_key(leaf, 'notes.txt') → None. The
  isinstance(..., str) check is not optional.

Footnote — adjacent defects found while verifying (out of scope, but live today)

These are unrelated to the delegation bug and reproduce on a plain unwrapped store:

• ValuesView.__contains__ / ItemsView.__contains__ call methods that do not exist.
  dynamodol/base.py:99 calls self._mapping.contains_value(v) and :106 calls
  self._mapping.contains_item(item). Neither contains_value nor contains_item is defined
  anywhere in dynamodol or in dol. So 'A' in store.values() raises
  AttributeError: 'DynamoDbPrefixReader' object has no attribute 'contains_value'.
  Deleting these views (remediation step 1) also fixes this.
• Debug prints left in the library. dynamodol/base.py:50 (print(f"x: {x}"), inside
  decimal_to_float, so it fires for every value and every nested element read),
  dynamodol/base.py:178 (print(f"obj: {obj}"), every __getitem__), and
  dynamodol/partition_query.py:227 (print(f"getitem: {k}")).
• DynamoDbPartitionPersister.__delitem__ (dynamodol/partition_query.py:290) does
  getattr(e, "__name__") with no default inside an except block, which raises
  AttributeError and masks the original exception for essentially every real error.
  (DynamoDbBasePersister.__delitem__ at dynamodol/base.py:342 guards this correctly with
  hasattr; the partition subclass does not.)

---

Note on in-flight upstream fixes (added when filing)

Two dol PRs are open and change details referenced above:

• i2mint/dol#84 — inner_most_key and unravel_key
  become importable from dol directly (no more from dol.dig import ...), and
  inner_most_key now raises instead of returning None when no layer of the chain
  defines _id_of_key. If you write a local shim, the isinstance(_id, str) guard becomes
  unnecessary once that lands — but the "it replaces _id_of_key, never composes with it" trap
  still applies.
• i2mint/dol#85 — fixes dol.content_url to
  resolve the key through wrapping layers, and makes mk_relative_path_store install
  key-mapping is_valid_key/validate_key. Any finding above that is inherited from
  dol.filesys.Files is repaired by #85 with no change needed in this repo — this issue will
  be closed with verification once it merges.

Design context for why the ecosystem-wide answer is not "sprinkle wrapped_self everywhere":
i2mint/s3dol#14 and
s3dol ADR-0011.
Short version: wrapped_self is a best-effort guardrail with its own silent failure mode (it
degrades to the raw leaf when nothing holds a reference to the wrapper), so the durable fix is
to have no key-taking methods rather than to harden each one.