perf(upload): skip suite objects the bucket already holds

[qu0b]

Follow-up to #301. That PR made the suite upload succeed; this one stops it re-sending the same bytes on every run.

The waste

A suite directory is content-addressed — ComputeSuiteHash digests the pre-run and test step file contents — so a given key under suites/<hash>/ always holds the same bytes. UploadSuiteDir re-sent all of them on every completed run.

For 0d93b5bf3b970403 (jochemnet stateful) that is ~12.4 GB per run: a 9.4 GiB pre-run bundle (pre-run.request, bloatnet setup with BLOATNET_RECEIVER_CONTRACT_COUNT=100000) plus 2.96 GB of fixtures across 2733 files. That suite alone has completed 12.8 runs/day over the last two days, so roughly 158 GB/day of PUTs overwriting byte-identical objects — and every other suite pays the same tax in proportion to its size.

It went unnoticed because the bundle was failing with EntityTooLarge before it could ever be sent a second time. #301 fixed the failure, which turns the waste on.

The change

• List the destination prefix once and skip anything already there at the same size. One paginated ListObjectsV2 covers 2733 keys in 3 requests, versus a HEAD per file.
• Where the ETag is a plain MD5 — objects small enough to have gone up in a single part — verify that too. The skip is then content-based, not size-based, for every file except the one whose identity the suite hash already pins.
• summary.json is never skipped. Metadata labels can change between runs without affecting the suite hash, which is exactly why CreateSuiteOutput rewrites it every time.
• A listing failure logs a warning and uploads everything, so losing the optimisation can never lose the suite.

On ETags

Worth stating the limit plainly: an ETag is only an MD5 for single-part objects. For a multipart upload it is a digest of the part digests with a -N suffix, and reproducing it means re-reading the whole file with the same part size — the exact I/O this skip avoids, on the exact file that motivated it. So the 9.4 GiB bundle is matched on size alone, which is sound here because its key is content-addressed. Everything else gets the stronger check.

Tests

TestUploadSuiteDirSkipsUnchangedObjects uploads a suite twice against a fake S3 that serves ListObjectsV2 with real sizes and MD5 ETags:

• first pass sends all three files
• second pass sends only summary.json
• rewriting a file to different content of the same length re-sends it — the ETag catches what a size check would not

Verification

make build-core, make lint-core (0 issues), full go test ./pkg/... ./cmd/... all pass locally.

[qu0b]

Simplified — the ETag check is gone, size alone now.

It existed to catch same-size-different-bytes. With the pre-run cap in #304 every remaining suite file is under the 64 MiB part size, so that branch would have covered all of them: a full read and MD5 of the whole suite, 1–3 GB, on every run, to avoid re-sending the same 1–3 GB. And it guarded something the design already rules out — a suite hash is a digest of exactly these step files, and only benchmarkoor writes under that prefix, so a key cannot hold different bytes at the same size without the hash changing too.

The skip is now one ListObjectsV2 and a size comparison. 63 lines and a crypto import lighter.

summary.json stays exempt: it is rewritten rather than content-addressed, so a label edit that happens to preserve its length is plausible in a way it is not for the rest.

Why this is still worth having alongside #304

#304 removes the 9.4 GiB pre-run bundle, but only one of the five active suites has one. The other four re-send their full fixture payload on every run and #304 does nothing for them. Measured from payload_sizes in each suite's summary.json against the last 5 days of the runs index (591 runs, ~119/day):

suite	size	runs/day	redundant/day
state-actor-gd7-compute	1.50 GB	32.8	49.4 GB
state-actor-gd7-stateful	2.87 GB	13.2	37.9 GB
state-actor-gd7-full-compute	0.87 GB	37.2	32.4 GB
state-actor-gd7-full-stateful	0.78 GB	29.8	23.2 GB
jochemnet-gd7-stateful	2.96 GB	5.2	15.4 GB

~158 GB/day and ~273k PUTs/day — about 8.2M class-A ops a month. #304 removes ~52 GB/day on top of that, all of it from the one suite.