Summary
When a snapshot data file's torrent-hash doesn't match the local file (e.g. because the local file was rebuilt by erigon while a stale `.torrent` from a prior published version sits alongside it), the downloader's `invalidateData` silently renames the local `.kv` → `.kv.part` with no log at default verbosity. Erigon then starts up "happily" but the snapshot tier has a hole — reads for keys in that step range fall through to "not found" and return default-zero state. The bug surfaces much later at unrelated places (`nonce too high`, `wrong trie root`, etc.) with no breadcrumb pointing back at the rename.
Code: `db/downloader/downloader.go:1012` (`invalidateData` does `os.Rename(name, name+".part")`). It's reached from `addPreverifiedSnapshotForDownload` after the metainfo check fails.
Repro
- Sync erigon with downloader enabled; snapshot manifest publishes hashes for step ranges 0-N.
- Stop erigon. Wipe the data files for step range `X-Y` (e.g. `v2.0-{accounts,storage,commitment}.X-Y.kv`) but leave the `*.torrent` companions in place.
- Restart erigon and let it rebuild step `X-Y` from chaindata. Wait for it to finish (you'll see the new `.kv` written, mtime more recent than the `.torrent`).
- Restart erigon again.
- Observe: the rebuilt `.kv` is silently renamed to `.kv.part`. Smaller files (e.g. `code.kv` at ~10MB) sometimes escape if their hash check happens to land on a verified copy; large files (`commitment.kv` at ~2GB) consistently get tagged.
- State reads for keys in that step range now return default-zero. Block execution past that range fails with `nonce too high` (or wrong trie root, depending on what the read feeds into).
Hit on the in-house mainnet rig: rebuilt step ranges `9072-9080`, `9080-9084`, `9084-9086` each had their `commitment.kv` (and in 9072-9080 also accounts/storage) renamed to `.part`, while step `9086-9087` (which never had a published torrent) remained `.kv`.
Why this is severe
- Silent. No warn/info log at default verbosity when a `.kv` is renamed away from completeness.
- Surfaces far from the cause. The first user-visible symptom is a "nonce too high" at some block in the affected range, which looks like state corruption.
- Easy to hit. Anyone who has ever rebuilt their own state on top of a datadir that previously held the published snapshots is vulnerable. The natural intuition ("keep `.torrent` files so I don't have to re-download the parts I haven't changed") is the trigger.
Suggested fix
At minimum: `invalidateData` should log at `WARN` with the local path, expected hash, observed hash, and the source-of-truth metainfo it's invalidating against. That alone would have saved us several debugging cycles.
Better: detect "local file's hash matches what an old metainfo on disk says but doesn't match the live manifest" and refuse to silently rename — instead surface a hard error or quarantine path so the operator notices.
Best: when invalidating a locally-rebuilt file that the operator clearly intended to keep (e.g. mtime > the corresponding `.torrent`'s mtime), treat as operator authoritative and keep the local file, only warning about the manifest divergence.
Workaround
When wiping a step range's data files, also wipe the matching `.torrent` files in the same step range. Operators who keep them across rebuilds will eventually hit this.
Summary
When a snapshot data file's torrent-hash doesn't match the local file (e.g. because the local file was rebuilt by erigon while a stale `.torrent` from a prior published version sits alongside it), the downloader's `invalidateData` silently renames the local `.kv` → `.kv.part` with no log at default verbosity. Erigon then starts up "happily" but the snapshot tier has a hole — reads for keys in that step range fall through to "not found" and return default-zero state. The bug surfaces much later at unrelated places (`nonce too high`, `wrong trie root`, etc.) with no breadcrumb pointing back at the rename.
Code: `db/downloader/downloader.go:1012` (`invalidateData` does `os.Rename(name, name+".part")`). It's reached from `addPreverifiedSnapshotForDownload` after the metainfo check fails.
Repro
Hit on the in-house mainnet rig: rebuilt step ranges `9072-9080`, `9080-9084`, `9084-9086` each had their `commitment.kv` (and in 9072-9080 also accounts/storage) renamed to `.part`, while step `9086-9087` (which never had a published torrent) remained `.kv`.
Why this is severe
Suggested fix
At minimum: `invalidateData` should log at `WARN` with the local path, expected hash, observed hash, and the source-of-truth metainfo it's invalidating against. That alone would have saved us several debugging cycles.
Better: detect "local file's hash matches what an old metainfo on disk says but doesn't match the live manifest" and refuse to silently rename — instead surface a hard error or quarantine path so the operator notices.
Best: when invalidating a locally-rebuilt file that the operator clearly intended to keep (e.g. mtime > the corresponding `.torrent`'s mtime), treat as operator authoritative and keep the local file, only warning about the manifest divergence.
Workaround
When wiping a step range's data files, also wipe the matching `.torrent` files in the same step range. Operators who keep them across rebuilds will eventually hit this.