High sustained MDBX COW writes at mainnet tip (110-140 MiB/block)

[mbg033]

System information

Erigon version:

erigon version 3.5.4-28bc52d6

The same behavior was also observed on Erigon 3.4.2.

OS & Version:

Guest: Ubuntu 24.04.4 LTS, Linux 6.8.0-137-generic, x86_64
Host:  Proxmox VE 9.2.0, Linux 7.0.14-6-pve, x86_64
VM:    8 vCPU, 40 GiB RAM
Disk:  WD_BLACK SN850X 4 TB NVMe
Host filesystem: ext4, noatime
Guest data mount: virtiofs

There is no btrfs, ZFS, or autodefrag layer in the data path. SMART reports no
media or data integrity errors. The node has very low JSON-RPC traffic during
the measurements.

Commit hash: 28bc52d6

Erigon command (addresses shortened; all other relevant flags are unchanged):

/usr/bin/erigon \
  --chain=mainnet \
  --datadir=/var/lib/eth/erigon \
  --authrpc.addr=127.0.0.1 \
  --authrpc.port=8551 \
  --authrpc.jwtsecret=/var/lib/eth/jwt/jwt.hex \
  --http \
  --http.addr=<private-ip> \
  --http.port=8545 \
  --http.api=eth,net,web3,erigon,trace,ots \
  --port=30303 \
  --torrent.port=42069 \
  --nat=stun \
  --caplin.nat=stun \
  --metrics \
  --metrics.addr=<private-ip> \
  --metrics.port=6060 \
  --sync.loop.block.limit=10000 \
  --batchSize=1g \
  --caplin.max-peer-count=64

Consensus Layer: embedded Caplin. I also reproduced the execution-layer write
rate with Caplin disabled (--externalcl) and Lighthouse BN 8.2.1 connected to
the Engine API.

Consensus Layer command for the external-CL control:

lighthouse bn \
  --network=mainnet \
  --datadir=/var/lib/eth/lighthouse \
  --execution-endpoint=http://127.0.0.1:8551 \
  --execution-jwt=/var/lib/eth/jwt/jwt.hex \
  --checkpoint-sync-url=https://mainnet.checkpoint.sigp.io

Chain/Network: Ethereum mainnet

Expected behaviour

I expected steady-state writes at chain tip to be substantially lower than
110-140 MiB of MDBX copy-on-write pages and 160-180 MiB of physical NVMe
writes per execution block, especially while the MDBX file and total database
size remain constant.

If this write volume is expected for Erigon 3 at mainnet tip, please confirm it
and consider documenting the expected steady-state write rate or write
amplification. At the observed rate, the host writes approximately
1.2-1.4 TB/day if the workload remains constant.

Actual behaviour

Erigon stays fully synced and healthy, but continuously writes approximately
13-15 MiB/s to the physical NVMe at chain tip. The write activity continues
without meaningful RPC load and stops when execution stops processing new
blocks.

The strongest 30-minute baseline on Erigon 3.5.4 processed 150 blocks with
zero head lag and zero restarts:

Metric	Result
Physical NVMe writes	13.522 MiB/s
Physical NVMe writes per block	162.26 MiB/block
MDBX COW writes	9.166 MiB/s
MDBX COW writes per block	109.99 MiB/block
MDBX page splits	361.59/block
MDBX file size delta	0

The following controls were run on the same node and data directory:

Test	Duration	Blocks	Physical NVMe	MDBX COW	Notes
--externalcl, no CL input	300s	0	0.369 MiB/s	0	Execution head intentionally static
Embedded Caplin	300s	25	12.30 MiB/s	10.57 MiB/s	Chain tip, lag 0
Caplin, parallel execution	1800s	150	13.522 MiB/s	9.166 MiB/s	Chain tip, lag 0
Caplin, --exec.serial	1800s	149	13.358 MiB/s	10.180 MiB/s	Chain tip, lag 0
External Lighthouse	1812s	151	14.955 MiB/s	11.565 MiB/s	Chain tip, lag 0; Lighthouse was backfilling

Normalizing the two 30-minute Caplin execution modes by block:

Metric	Parallel	Serial	Serial change
Physical NVMe write/block	162.26 MiB	161.37 MiB	-0.55%
MDBX COW/block	109.99 MiB	122.97 MiB	+11.81%
MDBX page splits/block	361.59	415.05	+14.78%

Therefore --exec.serial does not reduce the write amplification. Replacing
Caplin with Lighthouse also does not reduce Erigon MDBX COW. The external
Lighthouse physical-disk result includes its checkpoint backfill, but the MDBX
COW metric is emitted by Erigon and remains high independently of that CL I/O.

During longer observation periods:

• mdbx.dat remained approximately 6.26 GiB;
• total db_size{db="chaindata"} remained constant;
• ReclaimableSpace, CommitmentVals, and ChangeSets3 changed in both
  directions instead of growing without bound;
• db_rotx_overloaded_total remained 0;
• the execution head remained at the public head;
• running Erigon with --externalcl and no CL input reduced physical writes to
  the host background level while the execution head remained static.

This looks like sustained MDBX COW/page-split churn caused by per-block
execution state and commitment updates rather than database growth, Caplin,
pruning backlog, or RPC traffic.

Steps to reproduce the behaviour

1. Sync Erigon 3.5.4 on Ethereum mainnet and wait for eth_syncing=false and
   zero public-head lag.
2. Wait at least five minutes after catch-up so catch-up execution is excluded.
3. Enable Erigon Prometheus metrics and scrape them every 15 seconds.
4. Measure an uninterrupted 30-minute chain-tip window.
5. Record the start and end execution block numbers and normalize all counters
   by the number of processed blocks.
6. Calculate MDBX COW bytes from the 16 KiB MDBX page counter:

increase(db_pgops{db="chaindata",phase="cow"}[30m]) * 16384

7. Measure physical device writes using node-exporter or Linux diskstats:

increase(node_disk_written_bytes_total{device="nvme0n1"}[30m])

8. Confirm that the physical write rate falls to the host background level
   when execution is intentionally held static with --externalcl and no CL.

For a control that isolates consensus input, start Erigon with --externalcl
and no CL. MDBX COW remains zero while the execution head is static. Then attach
a synced external CL: MDBX COW returns as execution resumes at chain tip.

Related work

This report may be an operator-visible measurement of the work already tracked
in:

• commitment: hierarchical branch cache + delayed writes + parallel trie computation #19801, which states that commitment dominates at tip and proposes delayed
  trie writes to reduce MDBX write amplification;
• [wip] [bloatnet] domain: store largeValues in own auto-increment-id table #21801, which attributes random MDBX page writes, splits, and overflow churn
  to large randomly keyed Commitment values and proposes a sequential value
  layout.

Could you confirm whether 110-140 MiB of MDBX COW per mainnet block is
expected on 3.5.4 and whether these two changes are intended to address this
steady-state write volume? I can provide raw Prometheus series, a longer test
window, selected Erigon logs, or test a specific branch/build if useful.

Backtrace

There is no crash or backtrace. Erigon remains synced, has no automatic
restarts, and logs no errors during the measurement windows.

[AskAlexSharov]

1. to reduce mdbx write io: you can reduce mdbx's pageSize: --db.pagesize (and remove rm datadir/chaindata and rm datadir/caplin)

2. in Erigon mdbx used for recent-hot-data only. It means we writing there new data, and then moving old data from mdbx to datadir/snapshots directory (to immutable mmap files). Means old data eventually deleted from mdbx. This is reason why it's not growing - and it must be << RAM (for perf)

3. our datadir has several mdbx databases: datadir/caplin/indexing, datadir/caplin/history, datadir/nodes, datadir/txpool, datadir/chaindata (maybe more). Can you isolate - which folder does painful IO? You can: --txpool.disable and --no-downloader to reduce amount of moving parts - to simplify diagnostics.

4. Also maybe some old data or db leaks - is result of old Erigon versions bugs. If this is the case - then you can just rm datadir/chaindata - it will reduce write IO for long time. Same about datadir/txpool and datadir/caplin. Erigon will get back to tip fast-enough.
   Aso reducing (use default) this flags - may reduce mdbx size:

--sync.loop.block.limit=10000 
--batchSize=1g 

you can see current mdbx pagesize by make db-tools, ./build/bin/mdbx_stat -ef datadir/chaindata

6. Erigon also does much datadir/temp writes (append-only temporary files).

7. FYI: In 3.6 we will reduce IO produced by datadir/snapshots/domain folder. But will need re-sync to new files there to get this effect (optional).

Here is example of some our 3.5 node 24 hours run (it's datadir/chaindata only stat. cow jumps: it's pruning of old data from mdbx - moving data to datadir/snapshots). So it's ~1M cow per day on chain-tip of mainnet:

on same server total write IO per 1 day (all folders) i see 2tb

[AskAlexSharov]

i see 2tb - but it's server with raid - maybe double-accounting happening in grafana

[AskAlexSharov]

FYI: tables with random key updates producing much more cow than append-only tables (even if tables are same size). For example CommitmentVals is much worse than ChangeSet3

[AskAlexSharov]

FYI: I think it will not help with write IO, but still maybe you want to try:
we have env flag MDBX_NO_FSYNC=true
(it's safe flag - it maps to mdbx's setting MDBX_SAFE_NOSYNC https://github.com/erigontech/mdbx-go/blob/155c45c282d0bf4a61840a279c7b297f8ec544d7/libmdbx/mdbx.h#L1438 )

your ext4 mount options are fine