docs: write architecture document

[frrist]

What this is

The design doc set for the Ingot S3-over-Forge redesign, bundled as one PR so the architecture
and the analysis that backs it are reviewed together. No code changes — these four docs are the
shared understanding we need before building.

docs/
├── aggregation-gate.md            ← start here (5 min): why this is gated on Piri
├── architecture.md                ← the target design (the main artifact)
├── rfc-pdp-minimum-piece-size.md  ← the PDP gas/economics + Piri policy that the size knobs rest on
└── pdp-cost-calculator.html       ← interactive model behind the RFC (open in a browser)

Start here

Suggested reading order

1. aggregation-gate.md — the one-page framing. Thesis: none of the S3 design runs on Forge until Piri stops size-pooling unrelated objects into ~128 MiB pieces. Read this first; it tells you why the other two docs exist and what the two asks of the rest of the stack are.
2. architecture.md — the target architecture, layer by layer (S3 → catalog → data → Forge/chain → cross-cutting), then a system contract, topology, gaps, and a Postgres schema appendix. This is the artifact most of your review time should go to.
3. rfc-pdp-minimum-piece-size.md — open to the Decision summary (top). That page is the whole argument; §1–§10 are the supporting analysis and the Appendices (behind the "Evidence" banner) are the raw gas audit — skim unless you're checking the numbers.
4. pdp-cost-calculator.html — play with piece size / churn / base fee / sender count to pressure-test the economics yourself.

Where your input is most wanted

The docs call these out explicitly; these are the decisions we want ratified, not just read:

• Aggregation model (aggregation-gate.md, architecture.md §6, rfc §7): the proposal is no cross-object aggregation — store each object as its own piece(s) — with the size floor (min_aggregate_size, ~8 MiB) demoted to the fallback if the contracts can't take that piece volume. The open question for FilOZ / FOC: can the PDP contracts/proving take object=piece at PiB scale? (See the top-level note "Should Piri do cross-object aggregation at all?")
• Versioned key encoding (architecture.md §3 callout): composite-key vs per-key version index; and versionId as a direct locator (scan-free, but leaks version ordering) vs opaque. We picked the direct locator — weigh the trade-off.
• The two asks (aggregation-gate.md): (1) rethink Piri's aggregation — target no cross-object aggregation (object=piece), with a generational background aggregator as the relief valve and the size floor as the fallback; (2) prove small pieces are operationally viable in the PDP contracts at PiB scale. Do you agree this is the gate, and the sequencing?

Correctness — what was verified against source

Every gas/contract claim in the RFC was checked against the actual contracts (by Claude!)
(PDPVerifier.sol and FilecoinWarmStorageService.sol, re-verified against FWSS v1.3.0 / PDPVerifier v3.4.0 after an upstream contract update) and the calibnet
gas-benchmark fixtures. The load-bearing (dammit claude) findings held: no proof-slashing exists (the new 0.1 FIL cleanup deposit is refundable, not a penalty), add cost is size-independent (~120,700 gas), every on-chain hot path is bounded (no O(live pieces) loop), and the calibnet anchors matched exactly. Corrections that came out of the audit and are already baked in:

• ⚠️ The addPieces batch cap was removed upstream (FWSS v1.3.0 / PDPVerifier v3.4.0). EXTRA_DATA_MAX_SIZE / MAX_ADD_PIECES_EXTRA_DATA_SIZE are gone; batch size is now bound by the FVM PiecesAdded event-size + per-tx gas. So the registration-throughput numbers (pieces/tx, txns/PiB, sender fleet) are an open measurement, not fixed — the earlier ~13-pieces/tx / ~11M-txn / ~12-sender figures are withdrawn pending re-measurement. (Good news for "many small pieces": the contract wall that would have throttled them is gone.)
• Storage price is 2.5 USDFC/TiB/mo (not 0.9), now an immutable constant (lib/PriceListUSDFC.sol); v1.3.0 also adds a per-dataset USDFC fee layer. Doesn't move the margin conclusion.
• Citation fixes (nextPieceId now :800, standalone-submit jobqueue.go:205, challenge-window 60 mainnet / 20 calibnet / 10 devnet, etc.).

Scope / non-goals

• Docs only — no implementation in this PR. The architecture explicitly supersedes the bootstrap MVP currently in the tree.
• Catalog GC, the allocate-by-size security model, and the batch allocate/accept wire format are named as known gaps, not solved here (architecture.md §11).
• Some RFC code references point at sibling repos (piri, filecoin-services/FilOzone) by design, since the gas model is derived from them.

[bajtos]

This is a well-written, detailed proposal. Love it!

I am afraid I ran out of energy & concentration by the time I got to the middle of docs/architecture.md. I'll continue the review of section 7 and onwards tomorrow.

[bajtos]

never repacked on delete; dead bytes linger and compact periodically (or never).

We need to consider requirements for compliance with GDPR and other regulations.

In fil-one/RFC#9 (comment), we are relying on Forge to remove deleted files from PDP within 24 hours after deletion.

That means we cannot let dead bytes linger forever, we need to compact them within that 24 hour window.

[frrist]

Good point, and I think the key is to separate two obligations that this line is conflating:

1. Compliance / "the customer's data is gone" — met by cryptoshredding, not by byte removal. The committed encryption RFC is explicit that true byte-removal is a slow, future mechanism and that the deletion path is (a) remove the entry from the bucket, (b) destroy the region key:

4. Deletes should be fast. When an object is deleted, its region keys are cryptoshredded. Removing the underlying blob bytes from the network is a separate, future mechanism; this RFC is about the keys... The first step to deleting is removing the entry from the bucket. The second is deleting the region key, truly preventing access.

source (LN 28)

Cryptoshredding is realistically our only way to hit a 24h window anyway — a Piri operator can't be forced to delete bytes on demand, but we can destroy the key that makes those bytes readable.

2. Cost — compacting/removing the dead bytes (and retiring the on-chain piece) so we stop paying to store and prove them. This is the part "linger... or never" was talking about, and it's a cost concern with a looser SLA, not the compliance clock.

So the full delete path as I see it:

• Immediate: remove the key from the MST — matches S3 GET/HEAD → NoSuchKey semantics.
• Within 24h: cryptoshred the region key — the data is unreadable; this is what satisfies the compliance obligation.
• Looser, somewhat unbounded: remove the bytes from Piri's MinIO store and retire the on-chain piece — this reduces our cost (we pay for proven bytes until they're gone), not a compliance gate.

One thing I'll flag as an open cross-doc item: your link is to discussion comment asserting a hard 24h byte-removal SLA, while the committed RFC text treats byte removal as a slow non-goal. We should reconcile which is the real requirement (immediate cryptoshred vs. hard 24h byte deletion) across this doc, RFC #9, and the appliance plan — because the answer changes how aggressive the compaction process has to be. (My gut says we can't force fast compaction, so cryptoshred feels like a hard requirement, and iirc is how Auroa operates)

One more connection: if we go the way I'm proposing in the top-level note ("Should Piri do cross-object aggregation at all?" — one object = its own piece(s)), the "dead bytes linger" problem largely evaporates, because deleting an object retires its own whole piece(s) — there's no shared aggregate to compact around.

[bajtos]

I mostly agree with what you wrote.

The problem with cryptoshredding is that we want to have two copies of the content-encryption-key:

• One is stored by Ingot and easy to delete
• Another is stored in the blob via FEE. This copy is wrapped by a key owned by FilOne and has coarser granularity (per bucket or per tenant).

The consequence: As long as the blob is stored in Piri, FilOne has the means to recover the stored data.

If we want to implement proper cryptoshredding, then we need to change FilOne/Hilt to create a new per-object wrapper key instead of sharing the same per-bucket/per-tenant key.

I don't have a strong opinion about which path we should take (per-object keys in Hilt vs. removing deleted blobs within a reasonable timeframe). What's important to me is meeting the compliance requirements.

Let's add @Peeja to the loop since she is designing the encryption.

[bajtos]

Wow, that's a lot of transactions!

• Do we have an estimate of how much we will pay in gas fees to onboard PiB of 8 MiB pieces?
• How quickly can we submit these 11M transactions on the chain?

[frrist]

Good questions, and the answer splits ~cleanly: gas dollars are (probably) not the problem; transaction throughput is.

• Gas cost: registration (addPieces) is ~120,700 EVM gas/piece (size-independent), ~8× in FEVM. For a whole PiB at 8 MiB (as an example) that's on the order of a fraction of a FIL to low tens of dollars at a quiet base fee — negligible against storage revenue. And registration is deferrable: Piri gas-gates it (defer-and-retry, no deadline, no slashing for being late), so we ride cheap-gas windows and never pay spike prices for it. So I'm not worried about the fee. But there is a tradeoff for operators here, since not registering during times of high gas fees means not being paied for data stored.
• Throughput: this is the real wall. And this all depends on how many roots we can add in a single transaction, and how many of those we can send per epoch. The calculator in here attempts to model this.

So "how fast" is a function of how many addPieces transactions we can send, and how many pieces we can stuff in each transaction.

To be upfront: this throughput magnitude is exactly the assumption I'm asking FilOZ to stress-test in the top-level note ("Should Piri do cross-object aggregation at all?"). Good news since I first wrote this: FilOZ already removed the addPieces batch cap in v1.3.0, so the registration wall is softer than the early ~13-pieces/tx draft assumed — but the measured pieces/tx (and whether to shard across multiple data sets) is still the open question that decides whether we can drop aggregation entirely or have to keep a floor.

[bajtos]

Suggested change

	workload literature (IBM COS / SNIA traces); **instrument Ingot from day one** so the floor/ceiling
	become measured, not guessed.
	**instrument Ingot from day one** so the floor/ceiling become measured, not guessed.

I agree we should instrument Ingot and start collecting object sizes from day one. However, we need data from real users, not from out internal testing. I think that means this instrumentations belongs to the R1 milestone.

Thoughts?

Pull initial shapes from object-store workload literature (IBM COS / SNIA traces)

Please ask Aurora if they can find these numbers for us. Either for FilOne customers only, or for their entire customer base, whatever is easier for them. I think this will give us some good data points too!

[frrist]

Agree on both counts. The instrumentation hooks are cheap and I'd put them in the code from day one, but you're right that the tuning decision (floor/ceiling values) is only meaningful against real billable workload (which we lack w/ Forge), so it's R1-gated — internal test data would just bias us. (Hypotehtically we could punt this decision to the SP, but its a complicated tuning decision I doubt they will want to make)

On Aurora — I strongly agree we should ask, and their numbers (FilOne customers, or their whole base, whichever is easier for them) would be the best single data point we could get, since they're a real object-store operator running their own (non-Forge) stack. I don't have a direct line to them though, so I'll need a hand getting connected — who's the right person to make that intro / own the ask? Is this an ask you could take on (pretty please)?

[bajtos]

I didn't realise you are not in the shared Slack channel. I added you and started the discussion with Aurora here: https://filecoinproject.slack.com/archives/C0AGASAA4FN/p1781856109595929

[bajtos]

Data from FilOne production - last 30 days:

https://filecoinfoundation.grafana.net/explore?schemaVersion=1&panes=%7B%22wsw%22:%7B%22datasource%22:%22grafanacloud-prom%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22100%20%2A%20minio_cluster_usage_objects_size_distribution%5Cn%20%20%2F%20ignoring%28range%29%20group_left%28%29%5Cn%20%20sum%20without%28range%29%28minio_cluster_usage_objects_size_distribution%29%22,%22range%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22grafanacloud-prom%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22%7B%7Brange%7D%7D%22%7D%5D,%22range%22:%7B%22from%22:%22now-30d%22,%22to%22:%22now%22%7D,%22compact%22:false%7D%7D&orgId=1

[bajtos]

Here is a better view:

https://filecoinfoundation.grafana.net/d/mijt44q/filone-engineering?orgId=1&from=now-6h&to=now&timezone=browser&var-stage=811430801166&var-aurora_api_name=$__all&refresh=auto&editPanel=82

[bajtos]

I am wondering how you chose 8 MiB as the new floor, rather than 1, 2, 4, 16, or 32 MiB?

[frrist]

Honestly, 8 MiB is a principled starting point, not a derived optimum. It's the AWS SDK default: both boto3 and the aws CLI default multipart_threshold and multipart_chunksize to 8 MiB, so any object large enough to matter arrives already split into ~8 MiB parts by the client. Setting the floor there makes "one client part ≈ one on-chain piece" land for the dominant upload path, while smaller whole objects fall below the floor and get aggregated.

The trade across the range: a smaller floor (1/2/4) puts more objects above the floor (better delete granularity, each its own piece) but multiplies piece count and therefore registration transactions; a larger floor (16/32) means fewer pieces but more sub-floor aggregation and bigger repack units when a member is deleted. 8 sits at the natural client boundary.

That said — I'm now questioning whether we want a floor at all. The cleaner model is one object = its own piece(s) with no cross-object aggregation, which makes "8 vs 1/2/4/16/32" moot. I've written that up as a top-level note ("Should Piri do cross-object aggregation at all?"); short version: 8 MiB only matters if FilOZ tells us we must aggregate, in which case the floor is a function of the measured object-size distribution we don't have yet (the instrumentation/Aurora thread). Absent that constraint, there's no floor to pick.

[bajtos]

Honestly, 8 MiB is a principled starting point, not a derived optimum. It's the AWS SDK default: both boto3 and the aws CLI default multipart_threshold and multipart_chunksize to 8 MiB, so any object large enough to matter arrives already split into ~8 MiB parts by the client. Setting the floor there makes "one client part ≈ one on-chain piece" land for the dominant upload path, while smaller whole objects fall below the floor and get aggregated.

Thank you for the explanation; now the choice makes sense to me!

[bajtos]

Ingot is an S3 gateway over the Forge network. It runs as an embeddable Go library (a host like piri/guppy/sprue imports it in-process) and as a standalone daemon (ingot serve).

Interesting! That means we can further simplify our R0 or even R1 scope by running Ingot inside the Piri process.

@hannahhoward @alanshaw Thoughts?

[frrist]

Right, and that's by design — Ingot was always meant to be importable as a library by one of these services, with ingot serve as the standalone option. Folding it in as an optional module of Piri is a real win on a couple of fronts: it simplifies installation/deployment (one process to ship and run; one-process is a super hot topic right now 😂 ), shaves the LAN-RPC latency between Ingot and Piri, and lets Ingot's local "spool" store reuse Piri's existing object store instead of keeping a second on-disk copy.

The one thing I'd be precise about so we don't over-claim: it's a packaging/perf optimization, not an architectural or scope reduction. The appliance already co-locates ingot + piri at the provider, and the control plane still flows ingot → sprue → piri (and → Hilt in R1) regardless of whether they share a process — so it doesn't remove a network hop on the upload path or simplify the auth story.

Net: worth doing for the deploy/spool simplification, but it doesn't cut R0/R1 work in a structural way.

[bajtos]

The one thing I'd be precise about so we don't over-claim: it's a packaging/perf optimization, not an architectural or scope reduction.

💯

[bajtos]

In the R0/R1/R2 roadmap, the plan is to remove indexer from the system in the initial versions. Do we need to account for that and update this section to describe how we will be handling reads without the indexer?

[frrist]

I'm treating that roadmap as a tentative plan, and I'm not yet convinced dropping the indexer makes any of this land faster — when we remove a system we still have to build its replacement (here, a local location table) and we take on the credible-exit / external-discoverability cost of doing so. So removing the indexer reads to me as a lateral move, not a shortcut. I'd rather keep this doc indexer-based as the target and document the no-indexer R0/R1 path as a deliberate, reversible reduction, rather than rewrite the read path around its absence.

[bajtos]

Does it make sense to remove the read cache from the R0/R1 scope to allow us to ship faster?

[frrist]

I'd keep it, as I suspect the performance benefits from it will be worth it, and we'll need it any ways before a production deployment to get a reasonable TTFB. But its listed as optionally recomended here because my guts says its a few hours to implement and gut feeling are sometimes wrong. Will drop if it proves complicated, left here for posterity.

[bajtos]

Same comment as above - what is the argument for the log function in O(log)?

[frrist]

Same answer as the earlier O(log) thread: it's O(log(cumulative piece count)) = O(log(dataset_bytes / piece_size)) — the depth of the sumtree descent each proof challenge performs (floor(log2(N)) + 1). Not O(log(piece_size)), not O(log(total_size)) on its own. And the count is monotonic (nextPieceId never decrements), so under churn it's log of adds-ever, ratcheting up permanently. I'll disambiguate both occurrences in the doc.

[bajtos]

Is it worth discussing with the FilOz/FOC folks why PDP enforces this relatively low limit, and whether it's feasible to increase it to match the FWSS limits?

[bajtos]

Do we have any concerns about the one node-wide PDP data set as we grow the amount of data stored?

If a node is storing 100TiB of data, spread across 1M objects >= 8MiB:

• The data set has 1M roots, does the on-chain data structure support this?

The PDP proofs are probabilistic. I asked Claude to compare one 100TiB-sized proof set vs 100x 1TiB-sized proof sets, and it says there is meaningful difference.

Summary:

The probabilistic guarantee is "you can't cheat much for long without detection," not "every byte is checked every period." But "eventually" scales with how small the missing fraction is — and a single 100 TiB set makes "eventually" very long for small losses.

More details:

• One proof set, 100 TiB, 5 challenges/period: If the provider drops 1% of your data (1 TiB), per period: 1 − 0.99^5 ≈ 4.9% chance of catching it. They'd expect to get away with it for ~20 periods on average. Drop 0.1% (100 GiB) and per-period detection falls to ~0.5%.
• 100 proof sets, 1 TiB each, 5 challenges each = 500 challenges/period total:
  Now you have 500 independent challenges across your data every period. If the provider drops 1% spread across everything: 1 − 0.99^500 ≈ 99.3% caught per period. Even 0.1% missing gives 1 − 0.999^500 ≈ 39% per period.
• There's a localization problem beyond raw probability. With 1M roots in one set and only 5 challenges, an adversary who deletes one entire root (say one of your 1M pieces, ~100 MiB) is missing 1/1,000,000 of the leaves. Per-period detection ≈ 5/1,000,000 = 0.0005%. That root could be gone for thousands of years of proving periods before a challenge happens to land on it. The data is effectively unprotected at fine granularity.

[frrist]

Two halves to this, and I want to answer them differently.

On the security/probabilistic half (the 5-challenges-over-1M-roots dilution argument): I'd rather not pull that thread here. This doc treats the PDP/PDPVerifier contract security model as a vetted assumption, and how hard it is to cheat over a proving window is covered by the contract security analysis, not this doc.

On the operational scaling half, though — your instinct is right and it's very much live, so I've pulled it into a top-level note ("Should Piri do cross-object aggregation at all?"). Structurally, I believe ~1M roots is fine: every on-chain hot path is bounded (O(batch) / O(5) / O(log cumulative) / O(removals≤2000)). The real open question you're circling is how many proof sets a node should maintain. Today it's one data set per node for operational simplicity — but each set is a proof the node must produce every period, so sharding into N sets is N proofs/day and shifts the node's SLA and gas usaged since more proofs per day. The flip side is that a cap on pieces-per-set + sharding could help delete throughput, localization, and per-tenant rails. I don't have the answer — it's exactly the kind of thing we need FilOZ to weigh in on, and it's tied to whether we can drop cross-object aggregation entirely. Let's track it in that note.

[bajtos]

On the security/probabilistic half (the 5-challenges-over-1M-roots dilution argument): I'd rather not pull that thread here. This doc treats the PDP/PDPVerifier contract security model as a vetted assumption, and how hard it is to cheat over a proving window is covered by the contract security analysis, not this doc.

I agree to not question the PDPVerifier contract's security model.

What I am questioning: Did FilOz consider the security implications of very large ProofSets (like 100TiB+)? If they did, what is their recommendation for the maximum proofset size that still keeps PDPVerifier security promises?

I think it's important to include this point in our discussions with FilOz.

[alanshaw]

Can this please be a PR against the RFC repo?

I haven't yet reviewed rfc-pdp-minimum-piece-size.md but I have review fatigue so I'm going to stop there for today and publish the comments I have.

[alanshaw]

I'd like to not never repack. I'm unsure how we square this off, i.e. if we allow storage nodes to prove over pieces that no customer is paying for, it's basically on us to pick up the bill. Furthermore how do we distinguish a uncollected garbage piece from a completely random piece the storage node inserted to earn more $$$.

[frrist]

Agreed — "never" was too strong, and I think the cleanest way to "not never repack" is to not cross-object aggregate in the first place. I've written this up as a top-level note ("Should Piri do cross-object aggregation at all?"). If each object is its own piece(s), there's nothing to repack on delete (you retire the object's own piece), and the concerns somewhaat dissolve:

• Unpaid pieces: there's no shared aggregate accumulating dead bytes from other objects, so the proven set maps 1:1 to objects we're paying for.
• Garbage vs. fabricated — and these split on trust, which I want to be careful about: the garbage half (dead bytes from our own deletes) is honest-operator GC — non-adversarial — so our own bookkeeping tells us which pieces are no longer referenced and we schedule them for retirement on the normal delete path. The fabricated-piece half is adversarial and harder. If we, or a client ran Ingot we could reconcile on-chain roots against blob_refs. But that does not work for if the SP runs Ingot since:
  1. blob_refs is keyed by the blob's sha256, but the on-chain root is a CommP, and there's no way to verify a CommP corresponds to a given sha256 — that binding is a UCAN attestation (trust), not a computation
  2. the SP operator is the one running Ingot, so reconciling against their reverse index is letting the adversary audit themselves.

The trust-correct anchor has to be FilOne-side: Sprue issues every accept, so FilOne holds a central, SP-independent record of what it authorized (space, digest, size), and FilOne is the FilecoinPay payer, so the real backstop is reconciling the on-chain dataset size/accounting against the authorized total (which needs no per-piece CommP↔sha256 identity) and halting the rail on divergence. That covers inflate-to-earn; it doesn't by itself cover byte-substitution, but that's always been the case for this deployment configuration requiring mnual download and verification by a trusted party.

[bajtos]

From what I remember, the TX adding a new root to a PDP ProofSet must contain extraData signed by the Filecoin Pay payeer (FilOne). This extraData also contains the CommP of the added root. In other words, the SP cannot add arbitrary roots to PDP behind our back; every such addition must be signed by FilOne's wallet.

Also, since PDP is using CommPv2, we know what size we are signing over when FilOne is authorising a PDP addRoot transaction.

The point I want to make: I think we don't need to compare the total PDP proofset size against the authorised total, because we should be able to verify the size on a per-object basis.

I don't fully understand how Forge interacts with PDP, so I may be wrong.

[alanshaw]

I'm starting to wonder if we even bother with a minimum size...

While there's probably some aspect of churn related to size, my suspicion is that time is the dominant factor for churn. In other words, the longer a piece lives for, the less likely it is to be deleted.

What if Piri just put every piece on the chain as is, but had a background process to aggregate old pieces together? It allows new pieces to churn easily with minimal cost, it only aggregates pieces that are unlikely to be removed (and thus reduces re-aggregation work/cost), and it keeps the total number of proving roots low.

We can also implement this in 2 steps. Step 1 is just put every piece on chain as is. Ship it, then...Step 2 is add the generational aggregator background process.

[frrist]

This is the comment thats reshaping my thinking, and I've written the full version up as a top-level note ("Should Piri do cross-object aggregation at all?") — let me anchor the discussion there, but the short version: I think you're right, and I'd go further than "maybe no minimum" to no cross-object aggregation at all. Your "time dominates churn" prior is backed by the gas RFC (on-chain cost favors bigger pieces at every churn level, §3.2), so the only reason to aggregate is throughput/root-count, not cost — and if the contracts can take the piece volume, the cleanest answer is one object = its own piece(s), no co-tenancy, deletion always O(1).

In that world the two steps map cleanly: Step 1 is object=piece (no floor) — register everything as-is; Step 2, the generational background aggregator, becomes the optional relief valve for root-count/throughput, aggregating only old, cold pieces unlikely to be deleted (so it rarely triggers a repack).

Two caveats I'd flag rather than gloss:

1. "every piece as-is" maximizes piece count, which makes Miro's ~11M-txn / send wall worse for a heavy tail of tiny objects — so whether we can do
   this at all depends on the contract-practicality question we need to raise with FilOZ in the top-level note (and on raising the extraData cap / sharding across data sets).
2. Generational aggregation reduces the live-root count, but because nextPieceId is monotonic it doesn't undo the proving-depth ratchet, and registration txns are paid per-object up front regardless — so it's relief, not a free lunch.

Net: this is my preferred direction and I'd ship it iff FilOZ confirms the piece volume is practical; if they say it isn't, that's exactly when the size floor comes back as the fallback.

[bajtos]

I'd go further than "maybe no minimum" to no cross-object aggregation at all.

Sounds great to me! 💯

Generational aggregation reduces the live-root count, but because nextPieceId is monotonic it doesn't undo the proving-depth ratchet, and registration txns are paid per-object up front regardless — so it's relief, not a free lunch.

I am wondering if we can work around the monotonic nextPieceId by creating a new proofset for each generation aggregation run?

1. Flag the current proofset as "gc_pending".
2. Create a new proofset that will be used for new uploads from now on.
3. Wait until all TXs adding pieces to the gc_pending proofset are settled
4. Perform generational aggregation for pieces in the gc_pending proofset
   1. Create a new proofset to hold the aggregated roots
   2. Iterate over all roots in the processed proofset
      • Large root - move to the new proofset as-is
      • Aggregate small roots into bigger ones
   3. Schedule the processed proofset for deletion

Anyhow, I agree generational aggregation is post-MVP (post-M2), so we don't need to worry about it right now.

[alanshaw]

Just to clarify, my mental model of this currently is that a body digest is the hash of the entire object (single part) OR the hash of a root UnixFS node that links to all shards of the object (multi part).

Location claims are created by Piri nodes for each shard, telling us where the shard(s) for an object are.

A (sharded DAG) index is created by Ingot for the object - the index lists the shard digests that comprise the blob. The new nodes property (used only for multipart uploads) carries the root UnixFS node and it tells us the order the shards need to be served for retrieval. See RFC for details.

An object read queries the indexer for the body digest and gets back location claims and an index, allowing retrieval to take place.

What we could do, to remove the indexer would be:

1. Assume all blobs are stored at a specific Piri node URL (configurable probably).
2. Encode the (ordered) shard hashes in the object manifest.

This does however make the MST and blobs undiscoverable without Ingot. This actively prevents (or at least makes it hard) to credible exit.

[alanshaw]

Ah I see, the plan is to store these in the object manifest anyway? What do we get from this? We have to go to the indexer anyway to get the locations...

Consider calling this "shards" to align with what they are referred to in Forge.

[alanshaw]

Of course, we still need to index the MST blocks...

[alanshaw]

Um, so this is basically the reason the indexer exists. Items written to the cache are given ample time for IPNI to catch up (7 days) so this shouldn't be necessary.

That said a belt and braces approach might be a good idea. We have a library for getting notifications that an IPNI chain has "done a sync" you might want to take a look at: https://github.com/storacha/go-libstoracha/blob/main/ipnipublisher/notifier/notifier.go

[alanshaw]

Suggested change

	mints a fresh per-space location claim.
	issues a fresh per-space location commitment.

[alanshaw]

I guess so. I just don't know why you wouldn't use an index? You have to go to the indexer anyway and we should probably try to put as little information in the object manifest as possible as we don't have a good garbage collection story for the calalog.

If we ever want to access that object outside of the catalog then it is impossible - you can't ask the indexer about that object CID. 🤷‍♂️ IDK maybe not important, just seems weird to break compatibility here.

How about as a compromise we don't create indexes when the object is single part, but if there is > 1 shard then we do?

[frrist]

Should Piri do cross-object aggregation at all?

A theme runs through a lot of these comments, so I want to pull it out here and link the individual threads back to it.

The assumption I want to challenge. Piri has aggregated blobs into larger on-chain pieces since it was first written, and I think we've all just assumed that was necessary — for gas/throughput reasons — without ever actually stress-testing it. Reading this review, it struck me that

almost every hard problem in this doc traces back to one thing: cross-object aggregation.

Deletion complexity, compaction, "never repack," proving over pieces nobody's paying for, telling uncollected garbage from a fabricated piece, the GDPR compaction window, the size floor itself — they exist only because we pool blobs from different objects into a shared piece. Remove that and the whole cluster collapses.

The proposal (the simple world). Drop cross-object aggregation. Store each S3 object as its own content-addressed blob(s): one on-chain piece if it fits under a ceiling, or a handful of object-owned pieces if it's larger (split at the proving-code ceiling; per-part for multipart, which keeps us compatible with the encryption RFC's per-part envelopes). No floor, no co-tenancy — a piece belongs to exactly one object. Then:

• Deletion is ~O(1) per object, always — delete the object's piece(s), no survivors to re-hash, no compaction, no SLA window to chase.
• The garbage-vs-fabricated-piece question splits cleanly: garbage (our own dead bytes from deletes) is honest-operator GC; fabricated pieces are adversarial and can't be caught by Ingot's blob_refs (sha256↔CommP isn't verifiable — it's a UCAN attestation and the SP runs Ingot, so it can't audit itself), so that one anchors on FilOne-side accounting (Sprue's central accept log) + payment leverage. Still an open problem, but a tractable, correctly-anchored one.
• Per-tenant attribution gets cleaner, and the entire floor / "Regime B" compaction machinery in this doc just disappears.

The catch, and the real point: this hinges on a contract-practicality question we don't have the answer to. Object=piece maximizes piece count, so it only works if the PDP contracts and proving can take that volume in practice. We've never actually asked, though we meant to at the FilDev summit ( 🤦 ) The open questions for FilOZ / FOC — as the experts:

1. Is there a practical ceiling on the number of pieces in a single proof set? Where does it bite first — registration throughput (the addPieces batch is no longer contract-capped as of FWSS v1.3.0 — it's now bound by the FVM PiecesAdded event-size + per-tx gas, so what's the measured pieces/tx?), the monotonic nextPieceId proving-depth ratchet, the MAX_ENQUEUED_REMOVALS=2000/period delete cap, or the view-getter scans?
2. How many proof sets should a node maintain? Today one Piri node = one data set, for operational simplicity — but every data set is a proof the node has to produce each period, so N sets = N proofs/day and it shifts the node's SLA, proving burden, and most importantly cost via extra gas. Is a cap on pieces-per-set + sharding across multiple sets the right move (it would help delete throughput)? If so, what's the right number?
3. What's the registration-throughput ceiling now that you've removed the extraData cap? As of FWSS v1.3.0 / PDPVerifier v3.4.0 the addPieces byte cap (EXTRA_DATA_MAX_SIZE / MAX_ADD_PIECES_EXTRA_DATA_SIZE) is gone — the limit is now the FVM PiecesAdded event-size + per-tx gas. A measured pieces/tx (and whether that event-size limit is worth lifting at the protocol level) is what we need to size the sender fleet for "many small pieces."

The decision rule. I want to build around the assumption that object=piece is viable, because if it is, this gets dramatically simpler. But I'm treating it as an assumption to stress-test, not a settled fact:

• If FilOZ confirms the contracts can take it → we drop cross-object aggregation; the floor and compaction in this doc go away; Alan's generational background-aggregator becomes an optional relief valve for root-count/throughput (it only ever aggregates old, cold pieces unlikely to be deleted, so it rarely triggers a repack); and Miro's multi-dataset sharding is how we scale it.
• If FilOZ says there's a hard practical limit → we keep the size floor exactly as this doc outlines, as the documented fallback.

So the floor stays in the doc — but as the fallback, not the default. The default I want to pursue is "no cross-object aggregation," pending FilOZ telling us whether the assumption holds. This is also why I'd treat the aggregation strategy (and, relatedly, the indexer scope) as open in both this doc and the appliance plan until we have that answer.

How this relates to the S3 sharding RFC. I think these are two halves of one model, not competing proposals. The RFCs flat-file sharding strategy is the object→shard half: for each PutObject/UploadPart it shards data at 256 MB into ordered, content-addressed blobs (I think we can tune this limit as needed). This thesis is the shard→piece half: each of those shards, being its own object-owned blob, becomes its own on-chain PDP piece. The join is a single number — the RFCs 256 MB shard threshold, this thesis's blob ceiling, and Piri's MaxMemtreeSize = 256<<20 (pkg/pdp/proof/proof.go:114) are the same constant for the same reason (the commP memtree is built in RAM, for now, we will be improving this via code from Curio implementation). So under no-aggregation, shard == blob == piece, 1:1, and per-object deletion is per-shard == per-piece == O(1). The thesis is therefore the Shard RFC plus one line: each shard is its own piece. And because retrieval addresses bytes by digest, the aggregation decision is invisible to its read path entirely — the only thing the two halves must negotiate is the multi-shard ordering record (see the §8 / arch:467 thread).

[Peeja]

question: Can you say more? I didn't follow this line at all.

[frrist]

Yeah, sorry. This is a very subtle reference to the fact that Piris RemoveRoot method is incomplete.

[Peeja]

question: I didn't follow this either. I know what the nextPieceId is, but I'm not following why it's mentioned here, and I'm having trouble parsing "must not drift proving cost or the chain-reconciliation view-getters into trouble at high churn". Must not drift it into trouble?

[frrist]

Whoops, yeah I need to clear this up. What I am trying to indicate is that some methods on the PDPVerifier, and iirc a view contract over it will iterate from N->nextPieceId (where N = 0, and sometimes > 0). And since nextPieceId only ever increases, even with deletes, as the dataset churns iteration will become prohibitivly expensive over time.

[Peeja]

question: I'm not sure what that means. What's a "sender fleet"?

[frrist]

another whoops 😅 this leaked in - its a separate idea in Piri I have been playing with that fell out of this whole exploration, will provide a better definition and motivation for it in doc.

Basically its the idea of multiple blockchain accounts Piri submits transactions from in parallel, so registrations aren't all stuck single-file behind one account's queue. i.e. an account's messages must apply in strictly increasing nonce order, one stuck transaction stalls every higher nonce behind it — an underpriced tx during a gas spike, a revert, a dropped message, a bad gas estimate. At millions of registrations under variable base fee, that single-file fragility is the problem. I might be overthinking this here, but seems like a real issue that could popup at high throughput.

[Peeja]

question: What are our knobs to adjust in response to signals? Does instrumenting after launch still give us room to respond to observation?

[frrist]

Currently we have no knobs, reaction would be via an operator re-configuring their node - say for a lower floor (if we even go that route), different aggregation threshold, re-pack strats, etc. Or via an update we ship, preference for the former.

[Peeja]

suggestion: Let's reuse a noun here instead of introducing "body", to clarify what we're talking about. This is the blob digest, right?

[frrist]

👍 yes the blob digest.

[Peeja]

thought: Two things:

• The stream will also encrypt, yielding a FEE envelope of the blob on disk and a region-wrapped CEK to store in the DB. Doesn't necessarily need to be explicit here, but I want to make sure that's in your head too.
• It sounds like this is writing the whole thing out to disk before moving on, but we can upload each part blob as it's complete on disk. I had been thinking of this as bounding the space Ingot needs on disk, because we can throw out blobs after they're sent to Piri, but it turns out we want to hold onto them anyway for read-after-write and as a cache, so…that's less important. 😛

[Peeja]

question: Are these sizes in plaintext, or in FEE envelopes?

[Peeja]

question: …🦵❓

[Peeja]

question: The entire manifest is held in RAM, right? How pathological does a bucket have to be for that to be a problem?

[Peeja]

question: In other words, for credible exit, multipart manifests need to exist in the space somewhere and not just in a Postgres DB?

[hannahhoward]

This is broadly excellent.

My only hard change is I'd like to have a pretty serious conversation about my proposed alternate form for doing versioning, which I think more closely mirrors the domain model for how S3 versioning is supposed to work.

Beyond that, is just amazing well thought out.

I do want to call attention to the fact that some of the most thinking goes into questions we may not need to answer immediately:

• The first version of deletion is just crypto-shredding the regional encryption key
• The absolute top level goal in R0 is standup an end to end Forge provider in FilOne. No versioning, no real deletion, no various other things.

So my main direction is is focus on "how can I deliver this incrementally to meet R0 (which is just staging and can be completely wiped), R1 (which needs to be data format complete, but need not be feature complete), R2 (where we really want everything working)

[hannahhoward]

I want to second this and make a concreate proposal, after honestly rabbit holing an entire day of this design and how S3 versioning is supposed to work.

Before I get to what I propose, let me cover what I learned about S3 Version:

In particular: in S3 version id is different from its ordinal order:

• seq — the per-bucket monotonic ordinal (buckets.next_version_seq). Its only job is ordering (recency). Internal; never leaves the server.
• version_id — a string, the S3 client handle (x-amz-version-id). Its job is identity. It's opaque, and for suspended/unversioned writes it is the literal string "null".

Unfortunately this distinction is not just an information leaking concern.

For numbered versions they can coincide (version_id can just be seq rendered, information concerns not withstanding). They diverge for the null version: its version_id is frozen at "null" by S3, but it still has a real ordering (a seq).

Blame S3 for a wildly complicated concept.

invertedVersionId collapses the two — it's derived from the ordinal but handed back as the client version_id — and the null version is where that breaks: its id has to be "null", not an encoded ordinal, and there is only ever one null per key, replaced in place (verified against real S3 - I encourage you to make a bucket and mess around with turning on versioning, suspending and re-enabling multiple times etc). Keeping seq and version_id as separate fields makes that case fall out instead of needing a sentinel.

Ok now to the concrete proposal built on @Peeja 's idea:

1. Top MST stays keyed by the plain object key
2. Each leaf is:
   node = {seq, version_id, CatalogCID}
   leaf = { current: node, prev: MST<seq -> node >, null_seq?: seq }

current gives the fastest current path — GET / ListObjects read one leaf, no descent. prev is a per-key sub-tree of older versions keyed by seq; because it's scoped to one buckey key, the prev key is just the ordinal — no fancy error prone string math to produce an object-key + version composite; "order an integer" is all that's left. null_seq is to track if there is a null version id in the prev tree -- because of S3's very unique way of having exactly one "null" in the version history.

Write rule — depending on versioning state, push the displaced current into prev or discard it; and since S3 allows only one null per key, creating a new null also evicts any existing one:

retain = (versioning == Enabled) || (displaced.version_id != "null")
if retain: 
   prev = put(prev, displaced.seq, displaced)
   if displaced.version_id == "null": null_seq = displaced.seq
else: 
   gc_release(displaced)
if new.version_id == "null":
   if null_seq: prev = remove(prev, null_seq)
   null_seq = undefined
current = new

Mode	Displaced	retain
Enabled	numbered or null	push (S3 keeps a re-enabled null as noncurrent "null")
Suspended	numbered	push
Suspended	null	discard (replace the null in place)
Unversioned	null	discard

prev is written only when we retain history — always for Enabled, never for Unversioned, only at the numbered→null boundary for Suspended. The lingering null is often noncurrent (it lands in prev when an Enabled write displaces it), so the leaf tracks its seq (null_seq);

Optional (to speed up lists): lift is_delete + render fields (etag/size/last_modified) into the leaf and prev entries, so ListObjects/ListObjectVersions render straight from the tree without fetching manifests, still verifiable. Left out above to keep the base shape minimal.

Cost: Enabled writes do a prev insert + top resplice each time — more blocks per write than a flat splice, but scoped to keys you're actually versioning. One thing to confirm: rebuild assumes logstore keeps per-op records / op-roots that point at created manifests, not just the latest forge_root_cid.

[hannahhoward]

MinIO or "storage backend" -- I do think there's a world where some people just wanna throw ZFS at this.

[bajtos]

I finished reading the version from yesterday and have a few more comments. Next, I'll read the discussion and the updated version.

[bajtos]

sub-millisecond order

Are you sure we can perform a few Postgres writes in a sub-millisecond time?

[bajtos]

Can we use the name space_did or forge_space_did to capture the information from the code comment in the variable name? (Similarly to how we have root_cid and not root).

[bajtos]

What is bucket here?

Is it the string bucket name (referencing ingot.buckets.name? If yes, then I think bucket_name would be a better column name.

Do we have any concerns about the storage & performance implications of using text instead of integers? By using integers, I mean blob_refs.bucket_id INT as a foreign key reference to buckets.id column we would need to add.

[bajtos]

Why text and not bigint, considering that next_version_seq already has bigint type?

[bajtos]

space_did if we agree to rename the column (https://github.com/fil-forge/ingot/pull/2/changes#r3440973094)

[bajtos]

Have you considered using an enum type instead? I think that will be more efficient in the terms of storage size and query performance.

[bajtos]

What is bucket here? Is it the string bucket name (referencing ingot.buckets.name? Do we have any concerns about the storage & performance implications of using text instead of integers for this?

[bajtos]

Have you considered using an enum type instead? I think that will be more efficient in the terms of storage size and query performance.

[bajtos]

Have you considered using an enum type instead? I think that will be more efficient in the terms of storage size and query performance.

[bajtos]

What is bucket here? Is it the string bucket name (referencing ingot.buckets.name? Do we have any concerns about the storage & performance implications of using text instead of integers for this?

[pyropy]

Why do objects need to be re-hashed? Should not we only compute the new root hash instead?

[pyropy]

What happens if the objects are less than 8 MiB in size? Should they get padded or are they going to be aggregated with unrelated objects into a shared piece like they are aggregated today?

[pyropy]

Is there an ongoing discussion about this with the FWSS team? Would you mind linking that discussion in case there is one?

[pyropy]

question: Is Sprue aware if Ingot node also doubles as Piri node? Will it prioritize such node for blob slot allocation over the other nodes since there's no moving bytes across the wire?

[pyropy]

Is Ingot going to check UCAN delegations for local reads?

[bajtos]

The proposal (the simple world). Drop cross-object aggregation. Store each S3 object as its own content-addressed blob.
The catch, and the real point: this hinges on a contract-practicality question we don't have the answer to. Object=piece maximizes piece count, so it only works if the PDP contracts and proving can take that volume in practice.

FWIW, Filecoin Onchain Cloud uses exactly this design (or at least used to use it back in Q1 2026): one file = one PDP piece. So PDP & FWSS was designed with the assumption that there will be one piece per file, with files as small as ~32 bytes.

Of course, the question is whether FOC has ambitions to handle the storage scale FilOne is aiming for, or whether their design works for FOC only because they operate at a much smaller scale.