Appendable s3dol

[andeaseme]

We need an incremental contiguous storage for live waveforms.

S3 has multipart uploads

Sample multipart upload calls

For this example, assume that you are generating a multipart upload for a 100 GB file. In this case, you would have the following API calls for the entire process. There would be a total of 1002 API calls.

• A CreateMultipartUpload call to start the process.

• 1000 individual UploadPart calls, each uploading a part of 100 MB, for a total size of 100 GB.

• A CompleteMultipartUpload call to finish the process.

We need an interface

Minimally implement the 3 APIs as methods.

My suggestion is to use s3bucket[key] += part. It would need to CreateMultipartUpload if it empty. Keep track of the parts uploaded. A way to complete, maybe a sentinel value. Ideally, it would be an interface that will be shared between other stores but we don't have that well established yet.

Also need a means to clear out any incomplete multipart in case we want to discard (part of a large file) and if we want to keep (part of a wf) so they aren't stuck in limbo and taking up s3 space.

[thorwhalen]

Great find. Now that I see it, I remember our team using it in several occasions. Let's not forget it this time. Or... let's indeed use this in s3dol, and then forget about it, since it's not a functionality concern, but an implementation one.

Had a discussion with @valentin-feron about it. I'll let him add his comments. Here's my part:

s[k] = v for s3

We definitely want to use it when v is big.

Do we use it conditionally (conditioned on the size of v) or systematically?
I'd say systematically (if possible) since the overhead is paid for small files.
Though if possible, I'd separate, encapsulate, and make it easy to change this default choice
(e.g. though a setitem parameter, or a mixin that has a __setitem__).

The three objects you mentioned have parameters too. Make sure the user can change these.
I'd say that we encapsulate the trio into a item_setter (or setitem?) object (callable instance of a class or partial of a function). Then, I'd give (the possibility of) control to the user through a item_setter argument or a (mk_item_setter, item_setter_params) pair.

s[k] += v for s3

Before doing this, some familiarity with the general s[k] += v use cases, problems and designs should be known. Still, with that knowledge, it's completely acceptable (and even wise) to start with the s3-specific implementation, and refactoring to use the more general tools later. (Add a TODO for that in the code.)

That said, it should be noted (as @valentin-feron did) that we don't necessarily want to use s[k] += parts_of_v to implement the s[k] = v, since unless we do some extra work (that we don't want to do at this point, unless it's quick and easy), s[k] += parts_of_v won't take advantage of the asynchronous aspects of the multipart upload.

s[k] += v in general

Yes to making a well designed, dol-wide consistent interface for this.
Be well aware of the many use-cases, tools, and attempts of the past before embarking on this.

Please spend a bit of poking around dol, py2store and creek for previous thoughts (issues, discussions, wikis) and actual code regarding this. As you read stuff, please add your comments, or links to relevant resources (internal or external) for the problem

Not sure it's the best name for it, but I already have this Paging (aka chunking, batch) tools discussion started, so lets accumulate the general ideas there, and the "s3dol in particular" here.
I added this new comment about the problem, which you should definitely read.

[valentin-feron]

Yes, s[k] = v and s[k] += v are 2 different problems:

• s[k] = v is about making the operation possible, "regardless" the size of v (and maybe other aspects), and optimize it
• s[k] += v is about extending an existing value in the store.

I think we don't want to use the latter to handle the former, but do the opposite (kind of at least). Indeed, as @thorwhalen mentioned, s[k] += v is a sequential way to append data, when multipart allow us to upload parts in parallel. Now, when we do s[k] += v, we want to be able to do it, regardless the size of v, so we also want to use multipart upload here (reuse the logic of s[k] = v).

[thorwhalen]

Picking this up as part of the v1 redesign. The s[k] += v half stays deferred to dol (see below); this is about the s[k] = v half, and specifically about @thorwhalen's question of what the type of v should be when the whole point is that the bytes don't fit in memory.

Design is written up in misc/docs/decisions/0005-large-object-io.md. Summary and reasoning below.

The type question

The framing already answers half of it: if v is bytes, we've lost before we start. bytes is a value; multipart needs a producer. So the write domain must contain at least one type that is a reference to bytes rather than the bytes.

Proposal — a small, closed, explicitly-named union:

BytesSource = bytes | bytearray | BinaryIO | Filepath | Chunks | Streamable

where Filepath, Chunks and Streamable are tiny frozen dataclasses (value refs), dispatched with functools.singledispatch so the union is open for user extension and closed for modification.

On s[k] = 'filepath' — this one has a clean answer: reject str

Dispatch isn't the crime. Dispatching on str is. The test is decidability:

• Path('/a/b') — never content. Decidable. Safe.
• an open file handle — never content. Decidable. Safe.
• Chunks(gen) — explicitly named. Decidable. Safe.
• str — s['config'] = '{"a": 1}' and s['video'] = '/tmp/big.mp4' are both overwhelmingly plausible, and there is no rule separating them that isn't a latent data-corruption bug. (os.path.exists is a heuristic whose behaviour depends on the filesystem — the same code would do different things on two machines.)

So:

TypeError: A str value is ambiguous here: did you mean its utf-8 bytes, or a filepath?
  content : s[k] = v.encode()      (or wrap the store with a str codec)
  filepath: s[k] = s3dol.Filepath(v)

One word, zero ambiguity. This is the same move dol.util.write_to_file already makes when it can't tell what you meant.

(Bare os.PathLike gets rejected too — a Path is unambiguous, but accepting it while rejecting str is a confusing half-rule.)

"Is asymmetric write/read type a real problem, or just design ickiness?"

Mostly ickiness — it's the overwhelming norm. Checked in a live environment rather than from memory:

Library	write accepts	read returns	MutableMapping?
h5py	list, ndarray, scalars, bytes, SoftLink/ExternalLink reference objects	Dataset/Group (not an ndarray)	yes
configparser	a Mapping of anything	SectionProxy; values come back str	yes
fsspec FSMap	bytes, bytearray, array, anything with __array__; str rejected	bytes	yes
dol.Files	any buffer-protocol object	bytes	yes
shelve, numpy, zarr, pandas	broader than read	narrower	mostly

h5py is the one that settles it: SoftLink/ExternalLink are purpose-built objects meaning "the value is a reference to content elsewhere", assigned through an ordinary MutableMapping.__setitem__, in the most widely used scientific-data library in Python. That's exactly Filepath/Chunks.

But it's only safe under a law, and the honest version of the law is not the one you'd write first:

• N1 — canonical form. normalize: WriteDomain → bytes is total on bytes | bytearray | Filepath, and one-shot on BinaryIO | Chunks | Streamable — a stream ref is consumed by its first write. Assigning the same ref twice is a documented error, not a second copy.
• N2 — stability. For the re-readable half, s[k] = s[k] is a no-op and dst.update(src) terminates.
• N3 — the tested invariant. Not s[k] = v ⟹ s[k] == v, but s[k] = v ⟹ s[k] == normalize(v).

Where it does bite, concretely (so nobody has to rediscover it): setdefault becomes type-unstable; pop/popitem become expensive; and refs only type-check against the concrete class — through MutableMapping[str, bytes] (which is how dependents actually annotate the store) s[k] = Filepath(...) fails a type checker, because the value type is invariant. That last one is the real cost, and it's paid with an explicit update() override.

On the write-only store idea

One solution is to not have a __getitem__ in s at all… We basically can say that this special MultipartS3Store object is Iterable, Settable, and Deletable (but not Gettable)?

Answering the typing question first, since it's interesting: collections.abc offers nothing for this, and can't cheaply — MutableMapping inherits __getitem__ as abstract from Mapping, and pop/popitem/clear/setdefault are all defined in terms of it; only update survives. dol has mk_read_only/disable_setitem/disable_delitem but no disable_getitem and no mk_write_only — you can take writes away but not reads. A Protocol can express the shape, but @runtime_checkable checks method presence only, so isinstance({}, WriteOnlyStore) is True — "must NOT be gettable" needs a separate predicate.

That said — recommendation is to not split. Three reasons, in order of force:

1. It solves a problem that doesn't exist. After complete_multipart_upload the object is an ordinary S3 object; GetObject reads it back with no knowledge that it arrived in parts. Multipart is a transport concern, not a storage-model concern. There is no state in which the key is writable but not readable.
2. It conflates two different things. (a) "this blob is huge, upload it efficiently" is one __setitem__, fully solved by a strategy. (b) "this is a live stream being appended to over minutes" is a session — a lifetime, an UploadId, an ordering invariant, a completion event. (b) isn't a Mapping operation at all and shouldn't be modelled as one.
3. It costs the whole Mapping toolchain: dict(s), s.items(), filt_iter, cache_vals, kv_walk, Mapping.__eq__, and every write-then-read doctest.

There's also a use-case argument that only showed up once the full picture was mapped: filesystem→S3 migration and sync wants dst.update(src) to work between a dol.Files store and an S3 store. That's the strongest constraint against splitting, and it doesn't appear anywhere in the original framing.

How it reaches through the pure interface

Per @thorwhalen's note above — encapsulate the trio and let the user control it — the mechanism is an injected strategy, decided at construction:

BucketStore(
    bucket, connection=conn,
    writes=transfer_writes(multipart_threshold=64<<20, max_concurrency=8),
    reads=bytes_reads(),          # or stream_reads() / ranged_reads()
)

__setitem__ stays __setitem__; how it uploads is a parameter. Default is transfer_writes at boto3's 8 MiB threshold, so small writes are a single PutObject and large ones transparently go multipart — systematically, as suggested, with the overhead being one branch.

One structural note worth recording, because the obvious alternative looks tempting: a transfer strategy cannot be a dol value codec. Not for the reason you'd guess (wrap_kvs(preset=...) is handed the store and the key), but because preset's return value is still passed to the inner __setitem__ — it can transform the value, not replace the write — and because self there is the unwrapped leaf. So the strategy has to live in the leaf store, below wrap_kvs.

Also, per @valentin-feron's point: every non-bytes source routes through upload_fileobj/TransferManager rather than PutObject, so seekability is s3transfer's problem and the size threshold is never consulted for a non-seekable source (where it's undecidable anyway).

The read side

s[k] returns bytes — always, because N1 demands it. Streaming is reached in increasing explicitness: reads=stream_reads() at construction, store.handle(k) for .open()/.stream()/.read(byte_range=…), or a Filepath destination for download-to-disk. ObjectHandle is not a Mapping and is the documented escape hatch — the same role BlobHandle plays in azuredol.

s[k] += v

Stays out, and agreed it belongs in dol first. += on a Mapping is __getitem__ then __setitem__ — it reads the whole object back, which is the opposite of the point. Prior art is dol.appendable's Extender and dol#29. Worth noting azuredol reached the same conclusion from the other side: it removed append-blob-as-default because MutableMapping.__setitem__ semantically replaces the value.

---

Scope for v1 is core + large-object I/O. The related "in-flight multipart uploads as a MutableMapping where del uploads[(key, upload_id)] aborts" is deferred to v1.x — it's the single best collections.abc fit in the whole S3 API and it kills the classic orphaned-upload bill, but it's a separate interface from this issue. Tracked separately.