[EL] max reorg depth and changesets may not be enough in non-finality conditions

[taratorio]

• we got stuck in Reorg requested too low, capping to the minimum unwindable block (after an unwind of 519 blocks Unwind Execution from=202491 to=201972; our max reorg depth for change sets is 512) on fusaka-devnet-3 non-finality tests (seems like they simulate several days of non finality)
• check thread with some more info here
• investigate this a bit deeper - maybe better after "[5/5 Execution] wrong trie root" on a valid block on fusaka-devnet-3 #17025 is resolved as these may be related
• check with other EL clients - what is a realistic reorg depth that we all are expected to support - where do we stand with regards to that - we may want to write some more sophisticated re-org scenario tests for our engine api

[VBulikov]

yperbasis added Imp2 as Importance

[taratorio]

@yperbasis fyi I started a discussion on ETH R&D discord to gather info from other clients to understand better what the reorg requirements are (ie depth to aim for) - https://discord.com/channels/595666850260713488/1427205979673595975

summary:

• Geth: reorg resistance up to 90k blocks by default, reorgs in the last 128 blocks are very fast, everything below that is slow, using reverse diffs
• Reth: can do reorgs up to 10k blocks, keeps history for 10k blocks when in full node mode, but they've never tested reorgs deeper than 128 blocks
• Nethermind: currently only support 128 block reorgs, but are planning on increasing that considerably by using the "finalizedHash" from FCU by pruning data up to the finalised hash header and not doing any pruning during non-finality since finalised hash won't change (this approach causes db bloat so probably not a good option for us)
• Besu: unlimited, they only prune their trie for finalised hashes (same as the approach for nethermind)
• Nimbus EL: unlimited

Reasoning behind why for example Geth aim to be reorg resistant for longer periods - "The 90k was set a long time ago because it is ~2 weeks of blocks and in the worst case we can make any decision within 2 weeks" (from Marius). I personally agree with their decision here. It is something that is important to ensure mainnet runs without outages when bad events take place.

discussed a few options on our side with @AskAlexSharov

1. quick unwinds using change sets, for anything beyond that we do slow unwinds which reverse delete files backward and prune relevant db data
2. history based unwinds (will require more work on commitment history compression and improvements)

[AskAlexSharov]

1. slow unwinds which reverse delete files backward and prune relevant db data actually this technically can be very fast. File deletes are fast on linux, prune db may mean rm -rf datadir/chaindata (which we can speedup theoretically - and it will also improve UX). And after "Unwind" - anyway need re-exec blocks (I don't think somebody storing deep copies of state for fast-switch between them - but as I remember Geth did it in the past for recent forks) - so, I guess this will be a bottleneck of re-org. Of-course rm -rf datadir/chaindata on practice can be slow - because of our "graceful shutdown" feature (wait for all read thx to finish before close db) - but maybe it's doable (also maybe "graceful shutdown" feature is not very useful - because it slows-down restart speed of Erigon). Only problem with: rm -rf datadir/chaindata - you will have to download recent blocks from P2P - which we don't do now during re-org (we store non-canonical blocks in db, but not in files): but of-course we can do rawdbreset.ResetExec() instead of rm -rf datadir/chaindata.

[taratorio]

1. slow unwinds which reverse delete files backward and prune relevant db data actually this technically can be very fast. File deletes are fast on linux, prune db may mean rm -rf datadir/chaindata (which we can speedup theoretically - and it will also improve UX). And after "Unwind" - anyway need re-exec blocks (I don't think somebody storing deep copies of state for fast-switch between them - but as I remember Geth did it in the past for recent forks) - so, I guess this will be a bottleneck of re-org. Of-course rm -rf datadir/chaindata on practice can be slow - because of our "graceful shutdown" feature (wait for all read thx to finish before close db) - but maybe it's doable (also maybe "graceful shutdown" feature is not very useful - because it slows-down restart speed of Erigon). Only problem with: rm -rf datadir/chaindata - you will have to download recent blocks from P2P - which we don't do now during re-org (we store non-canonical blocks in db, but not in files): but of-course we can do rawdbreset.ResetExec() instead of rm -rf datadir/chaindata.

When a NewPayload/FCU request is received for another fork, the blocks missing on that fork are downloaded first so I dont think we need to worry about p2p at least for now.

The very difficult part for NewPayload will be - right now it is "in-mem unwind + in-mem exec + stash shared domain changes for flushing". Because it is in-mem unwind & exec it means that technically we can't delete the "wrong fork" files here. We might need to play with visible files (filter them there) or something else. When a FCU comes after, we do all the unwinds - delete files, db, etc. flush shared domain from NewPayload or not depending on sequence of requests. Most likely not in this case.

I'm starting to get scared a bit of the complexity involved here.

Another option is: go back to large step size, keep more steps in DB, go back to 100k blocks freeze threshold and just accept a big chaindata DB and optimise on top of it.

[taratorio]

If we accept a bigger chaindata DB size and optimise on top of it we can also consider the Finalised hash approach as a guide to when to collate & prune. Best case collate every 32 blocks after the first 96 blocks. Worst case keep going for 2 weeks without finality and have a large chaindata db that we can work with.

[taratorio]

1. slow unwinds which reverse delete files backward and prune relevant db data actually this technically can be very fast. File deletes are fast on linux, prune db may mean rm -rf datadir/chaindata (which we can speedup theoretically - and it will also improve UX). And after "Unwind" - anyway need re-exec blocks (I don't think somebody storing deep copies of state for fast-switch between them - but as I remember Geth did it in the past for recent forks) - so, I guess this will be a bottleneck of re-org. Of-course rm -rf datadir/chaindata on practice can be slow - because of our "graceful shutdown" feature (wait for all read thx to finish before close db) - but maybe it's doable (also maybe "graceful shutdown" feature is not very useful - because it slows-down restart speed of Erigon). Only problem with: rm -rf datadir/chaindata - you will have to download recent blocks from P2P - which we don't do now during re-org (we store non-canonical blocks in db, but not in files): but of-course we can do rawdbreset.ResetExec() instead of rm -rf datadir/chaindata.

When a NewPayload/FCU request is received for another fork, the blocks missing on that fork are downloaded first so I dont think we need to worry about p2p at least for now.

The very difficult part for NewPayload will be - right now it is "in-mem unwind + in-mem exec + stash shared domain changes for flushing". Because it is in-mem unwind & exec it means that technically we can't delete the "wrong fork" files here. We might need to play with visible files (filter them there) or something else. When a FCU comes after, we do all the unwinds - delete files, db, etc. flush shared domain from NewPayload or not depending on sequence of requests. Most likely not in this case.

I'm starting to get scared a bit of the complexity involved here.

Another option is: go back to large step size, keep more steps in DB, go back to 100k blocks freeze threshold and just accept a big chaindata DB and optimise on top of it.

One way we can potentially avoid this complexity with NewPayload is:

• only allow unwinds that do not require deleting files for NewPayload requests
• if a NewPayload request requires an unwind beyond files - do not do it - and return SYNCING/ACCEPTED status to the CL
• that will allow the CL to eventually send a FCU for this fork that requires such a long unwind beyond files
• then when processing the FCU request - we have freedom to delete the files - and purge chaindata DB
• I've already taken a similar approach for this in the EL backward downloader when we have to download large ranges of blocks (to respect --sync.loop.block.limit) and it works quite well

[taratorio]

If we accept a bigger chaindata DB size and optimise on top of it we can also consider the Finalised hash approach as a guide to when to collate & prune. Best case collate every 32 blocks after the first 96 blocks. Worst case keep going for 2 weeks without finality and have a large chaindata db that we can work with.

About the "FinalizedHash from FCU" approach:

• in normal finality operation of the beacon chain (99% of the time) - we get a new FinalizedHash every epoch (every 32 blocks)
• the FinalizedHash is for block Tip-96
• so in 99% of the times we are safe to start collating every 32 blocks when at Tip-96 (for all steps that are ready/completed before this block)
• in the non-finality operation mode of the beacon chain (1% of the time) - we don't get a FinalizedHash - so we do not allow collation
• we may also need to keep chagesets (and not prune them) during non-finality to allow unwinds within the non-finalised blocks (or switch to history based and get rid of changesets if we are able to optimise it)
• this will lead to chaindata DB growth but maybe we can operate fine with it for this 1% scenario
• our target is ~2weeks of non-finality ability - which is around 100k blocks - maybe it will be ok

The nice thing about this is that in the 99% case of finality:

• our worst case of blocks to keep data for is 96
• 100MGas blocks have a worst case of 4,761 transactions (eth transfers 21,000 gas each) which comes to 457,142 transactions
• our new safe step size can then be 457,142 which is nearly 4x smaller than the current one (we will need to still co-ordinate it with the finalised hash before collating) and still be ok in terms of finality in the 99% case
• or we can take this even further, we can have an unlimited small step number (we will have to balance the layered-file lookup of the LSM only) and only start collation for steps that are behind the FinalizedHash - this will allow us to have a much more aggressive collate & prune -> much smaller chaindata db (literally only 96 blocks worth of data has to be in chaindata)

[Giulio2002]

this is obviously something we cannot fix and noone can fix because non-finality is infinite. so I will close this as this is just not a feasible requst. there is a flag to enable changesets for all blocks for testnets if you wished but this is not something solvable