Slow wallet loading

[pavel-kokolemin]

I'm running a testnet LWK wallet with the legacy FS store.
There are 2075 transactions in total and 4607 update files in the enc_cache folder.
Recently I noticed that the wallet loading became quite slow.
It now takes about 29 seconds to load the wallet in the release build.
I attached a profiler and saw that the bottleneck is in this line:

        let mut vec: Vec<_> = self.timestamps.iter().collect();
        vec.sort(); // This line takes most of the time, 29 seconds in my case
        vec.hash(state);

After I changed it to this (to prevent extra indirection):

let mut vec: Vec<(u32, u32)> = self.timestamps.clone().into_iter().collect();

It took about 18 seconds to load the wallet.

And after I commented out this code:

            if self.wollet_status() != update.wollet_status {
                return Err(Error::UpdateOnDifferentStatus {
                    wollet_status: self.wollet_status(),
                    update_status: update.wollet_status,
                });
            }

It took less than 2 seconds to load the wallet.

Here is my cache size at the end:
all_txs size: 2075
scripts size: 2118
heights size: 2118
unblinded size: 2118
timestamps size: 286064

Cache::hash is called 4607 times (once per an update file).
So in my case, LWK is sorting up to 286064 elements 4607 times on loading.
I think this should be fixed.

[LeoComandini]

Hi @pavel-kokolemin

Thanks for the issue.

In these months we've been working on improving LWK to better handle high-volume wallets, see here.

However we prefer to avoid breaking changes, so we're doing a series of incremental improvements.

These (ongoing) efforts also include improving the way we handle Updates files (and possibly migrating away from those entirely).
So far we have added with_merge_threshold to automatically compact Updates files once they are more than threshold.
Setting threshold to 1, might be a reasonable choice: faster restarts, however a crash when doing the merge might cause some cache data to be downloaded again.