flush() leaves stale index copies in the file: 27x bloat when flushing often

[Ramlaoui]

What happens

Calling flush() often makes the .atompack file much bigger than it needs to be. The data inside is fine and reads back correctly — the file is just padded with copies of old indexes that nobody will ever read again.

It gets worse the more you flush, and it gets worse the bigger the dataset is. The two multiply, so it grows quadratically.

Repro

import os, tempfile, numpy as np, atompack

NAT, B = 8, 10

def build(flush_every_batches=None):
    p = os.path.join(tempfile.mkdtemp(), "t.atompack")
    db = atompack.Database(p)
    pos = np.zeros((B, NAT, 3), np.float32)
    z = np.full((B, NAT), 6, np.uint8)
    for i in range(400):                      # 4000 molecules total
        db.add_arrays_batch(pos, z, energy=np.zeros(B))
        if flush_every_batches and (i + 1) % flush_every_batches == 0:
            db.flush()
    db.flush()
    return os.path.getsize(p)

base = build(None)
print(f"single flush at end : {base:>12,}")
for fe in (100, 10, 1):
    s = build(fe)
    print(f"flush every {fe*B:>5} mols: {s:>12,}  {s/base:5.1f}x")

Same 4000 molecules every time, only the flush frequency changes:

flush cadence	file size	vs. flushing once
once at the end	612,200	1.0x
every 1000 molecules	812,232	1.3x
every 100 molecules	2,252,520	3.7x
every 10 molecules	16,655,400	27.2x

All 4000 molecules read back correctly in every case, so this is wasted space, not corruption.

The waste is accounted for exactly. For the every-10 case there are 400 flushes, so the stale index copies should total sum over k=1..399 of (8 + 20 * 10k) = 15,963,192 bytes. And 16,575,392 - 15,963,192 = 612,200 — precisely the single-flush size. Every excess byte is a stale index, nothing else.

Why

flush() writes the entire index to the end of the file every time it is called (encode_index of the full entry list, appended at End(0)).

Molecules added afterwards are appended at End(0) too — that is, past the index that was just written. So the file ends up looking like this:

[hdrA][hdrB][ rec 0..999 ][IDX 0..999][ rec 1000..1999 ][IDX 0..1999][ rec 2000..2999 ][IDX 0..2999]
                           ^^^^^^^^^^                    ^^^^^^^^^^^                    ^^^^^^^^^^^
                              dead                          dead                           live

Each old index copy is stranded in the middle of the file, and each one is bigger than the last because it holds every molecule written so far, at 20 bytes per entry.

This placement is not an accident: the two-slot alternating header is only crash-safe if the previous index still exists while the new header slot is being written. So the append is load-bearing — it just has no cleanup to go with it.

The space is never reclaimed

after writing (400 flushes)      :   16,575,392
after dropping the writer object :   16,575,392
after reopen (mmap=False)        :   16,575,392   (4000 molecules readable)
after reopen + 10 more + flush   :   16,656,910   <- keeps growing

Three reasons:

1. There is no close path — no impl Drop for AtomDatabase, no close(), no context manager on Database.
2. truncate_uncommitted_tail_if_needed only shrinks the file to committed_end, and for a cleanly flushed file committed_end is the end of the file, so it no-ops. It exists to discard an uncommitted tail after a crash, which is a different problem. It is also the only place the truncate flag is ever set, and it is only set on reopen — never between a flush and the next append in the same session.
3. Even a working truncation could not help, because the stale indexes are stranded mid-file, not at the tail. Removing them means moving all the records after them and rewriting every absolute offset in the index. There is no compaction API in the crate.

The file is not sparse either — du and du --apparent-size both report 16M, so these are really allocated blocks.

Why it matters

• 1M molecules flushing every 1k molecules adds roughly 10 GB of dead index. Flushing per molecule adds ~10 TB.
• mmap readers map the whole file, so a bloated file costs proportionally more address space and page cache on open, and much more with populate=True.
• Nothing in the docs warns about it. architecture.rst:162 describes the trailing index without mentioning the cost, and the getting-started examples call flush() freely.

Workaround for now

Flush rarely. Every ~100k molecules keeps the overhead under a few percent. Existing bloated files can only be fixed by rewriting them into a fresh file.

Possible fix

The index is append-only (IndexStorage::extend only ever pushes), so a flush does not need to rewrite it — it only needs to append the entries added since the last flush, and record the resulting chunks in a small extent table.

Conveniently, the header slot is 4096 bytes and only ~80 are used, leaving room for ~251 extents inline in a slot that is already checksummed and already crash-safe. Nothing existing would ever be overwritten, so crash safety is preserved rather than traded away. Full design to follow in a PR.