[walrus] Exact-change WAL payment + destroy_zero races against on-chain price changes (took out mainnet writes for a prod service today)

[DrVelvetFog]

Summary

@mysten/walrus pre-funds every WAL payment as exact change computed from a cached on-chain price, then asserts the payment coin is empty with 0x2::coin::destroy_zero. Any change in storage_price_per_unit_size / write_price_per_unit_size between the cache fill and execution makes the transaction abort deterministically. This took out 100% of mainnet writes for a production service earlier today (MystenLabs/MemWal#351); @ducnmm root-caused it in MystenLabs/MemWal#352 and noted it's worth hardening upstream — filing that here.

The pattern

#withWal splits exactly amount (or creates it via coinWithBalance), hands the coin to the payment call, then unconditionally appends:

tx.moveCall({
    target: '0x2::coin::destroy_zero',
    typeArguments: [walType],
    arguments: [coin],
});

amount comes from storageCost(), which reads systemState() through the SuiObjectDataLoader cache. The cache is only cleared by reset() / #retryOnPossibleEpochChange, which triggers on RetryableWalrusClientError — a MoveAbort in destroy_zero is not classified as one, so callers see a permanent-looking failure.

All three payment paths share the race: createStorage (storage cost → reserve_space), registerBlob (write cost → register_blob), extendBlob (storage cost → extend_blob).

Failure modes

• Price moves down → the contract deducts less than the split amount, the leftover trips destroy_zero (ENonZero, abort code 0, surfaces at 0x2::balance::destroy_zero via coin::destroy_zero).
• Price moves up → the split coin underfunds the payment and the register/reserve/extend call itself aborts.

Two aggravating factors from today's incident:

1. Under sponsorship the abort is masked. With Enoki the failure surfaces at the sponsor's dry-run as dry_run_failed: could not automatically determine a budget, so operators debug the sponsor before finding the stale price (that's exactly how Mainnet writes fail on both prod relayers: Enoki dry_run_failed (MoveAbort balance::destroy_zero in sponsored upload) MemWal#351 was reported).
2. It reads as permanent. Because the error isn't classified retryable, downstream job systems dead-letter every write until someone restarts the process (observed: storage_price_per_unit_size moved 71464 → 70922 and a relayer flipped from healthy to 100% failure at the price transition — see fix(relayer): recover Walrus register destroy_zero from stale WAL price (#351) MemWal#352 for the evidence).

Confirmed unchanged on latest @mysten/walrus (1.2.3).

Possible directions

Any subset of these would help; the first eliminates the downward-drift race rather than narrowing it:

1. Return the remainder instead of asserting zero. When a walCoin source is provided, tx.mergeCoins(source, [coin]) after the payment call; otherwise transfer the (usually zero) remainder to the sender. No behavior change on the happy path beyond not asserting.
2. Optional over-funding tolerance (small buffer on amount, remainder returned per 1) — also covers upward drift.
3. Classify the abort as retryable: recognize ENonZero aborts on the SDK's own appended destroy_zero command, reset() the cache, rebuild. Narrows the window (a fresh price can still drift before execution) but fixes the "permanent failure" misclassification cheaply.
4. Export an error type/classifier so integrators don't have to string-match the MoveAbort to know it's a stale-price condition (the MemWal relayer's auto-heal missed it for exactly that reason — its matcher looked for balance+split).

Happy to put up a PR for (1) (+ (3) if wanted) if a direction here is agreeable — I don't want to guess at which trade-off you'd prefer in the payment path.

[hayes-mysten]

Thanks for the report, I'll look into this next week. I think with address balances now being live, transferring 0 balances as address balances is probably more reasonable.

I'll need to dig into it more, but my initial thought on what we'd probably do is:

• Add a configurable amount to over-estimate the cost for upwards drift
• Transfer balances back as address balances (we avoided this previously because of storage costs of 0 balance coins)
• Look into cache invalidation + retry on budget failures

[DrVelvetFog]

Thanks — that plan sounds right, and address balances make it cleaner than what I sketched: returning the remainder as an address balance removes the zero-coin storage cost that made merge-back unattractive, and with the over-estimate buffer covering upward drift, both directions get handled at the transaction level rather than by retry alone.

Two small notes from the incident side:

• Whatever shape lands, exporting some classifier for the stale-price condition (an error type or a helper) would have saved the downstream relayer — its auto-heal missed this exact abort because the only thing available to match on was the MoveAbort text.
• In production the failure was masked behind Enoki's dry_run_failed budget error (sponsored PTB, dry-run before signing), which is where it was hardest to diagnose. When there's a branch, glad to test it against that path — I have the sponsored-upload setup that hit this.

No urgency from my side — the relayer is mitigated downstream.

[hayes-mysten]

@DrVelvetFog you can check out #1128 does that align with what you expect?