Materialize vault deltas for balance sync

[findolor]

Summary

• add derived_vault_deltas as a pipeline-maintained read model for the existing vault_deltas view output
• refresh derived vault deltas in the apply transaction before vault_balance_changes / running_vault_balances
• switch vault balance maintenance SQL to read derived_vault_deltas instead of recomputing the view
• include the table in required schema, clear/reset paths, export/import, and bump local DB schema version from 3 to 4

This PR is standalone on main and does not depend on the derived-trades stack.

Benchmarks

Measured with nix develop on a copied Base ST0x local DB, not the live DB.

Before this PR, the same balance maintenance path measured:

• recent 10k window: ~13.38s combined vault upsert batch
• busy window: upsert running_vault_balances alone hit ~184.54s

With derived_vault_deltas backfilled on the copy:

• one-time full derived backfill: ~6.65s for 29,682 rows
• recent 10k window: insert changes ~2.19ms, update running balances ~4.53ms
• densest 10k window found: 644 derived rows, insert changes ~91.33ms, update running balances ~56.28ms

Tests

• COMMIT_SHA=local nix develop -c cargo test -p raindex_common local_db::query --lib
• COMMIT_SHA=local nix develop -c cargo test -p raindex_common local_db::pipeline::adapters::apply --lib
• COMMIT_SHA=local nix develop -c cargo test -p raindex_common local_db::export --lib
• COMMIT_SHA=local nix develop -c cargo test -p raindex_common raindex_client::local_db::pipeline::bootstrap --lib
• COMMIT_SHA=local nix develop -c cargo test -p raindex_app_settings local_db_manifest --lib

Summary by CodeRabbit

• Chores
  • Updated database schema to version 4, introducing a new derived vault deltas layer for optimized balance tracking. Existing database instances will require a schema migration.

[findolor]

• Materialize vault deltas for balance sync #2592 👈 (View in Graphite)
• main

---

How to use the Graphite Merge Queue

Add the label Raindex-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 27af4125-20e1-464c-87ba-0962f207b3b9

📥 Commits

Reviewing files that changed from the base of the PR and between 29b7822 and c16b41a.

📒 Files selected for processing (11)

• crates/common/src/local_db/pipeline/adapters/apply.rs
• crates/common/src/local_db/query/clear_raindex_data/query.sql
• crates/common/src/local_db/query/clear_tables/query.sql
• crates/common/src/local_db/query/create_tables/mod.rs
• crates/common/src/local_db/query/create_tables/query.sql
• crates/common/src/local_db/query/mod.rs
• crates/common/src/local_db/query/upsert_derived_vault_deltas/mod.rs
• crates/common/src/local_db/query/upsert_derived_vault_deltas/query.sql
• crates/common/src/local_db/query/upsert_vault_balances/insert_balance_changes.sql
• crates/common/src/local_db/query/upsert_vault_balances/insert_running_balances.sql
• crates/settings/src/local_db_manifest.rs

---

📝 Walkthrough

Walkthrough

This PR adds a new derived_vault_deltas database table and materialized read-model that the pipeline maintains from source event data, then rewires vault balance calculations to consume this derived table instead of raw event deltas, including schema migration to version 4 with comprehensive tests.

Changes

Derived Vault Deltas Read-Model

Layer / File(s)	Summary
Schema definition and version migration crates/common/src/local_db/query/create_tables/query.sql, crates/common/src/local_db/query/create_tables/mod.rs, crates/settings/src/local_db_manifest.rs	New derived_vault_deltas table with composite primary key and two indexes (idx_derived_vault_deltas_window and idx_derived_vault_deltas_balance_key) supports windowed and balance-key lookups. Database schema version bumped from 3 to 4.
Upsert batch function and integration tests crates/common/src/local_db/query/upsert_derived_vault_deltas/mod.rs, crates/common/src/local_db/query/upsert_derived_vault_deltas/query.sql, crates/common/src/local_db/query/mod.rs	upsert_derived_vault_deltas_batch() produces a two-statement batch that deletes and re-inserts derived_vault_deltas rows for a block window, sourcing from vault_deltas. Unit tests verify batch structure and SQL correctness; SQLite integration tests confirm full and incremental upsert behavior with seeded deposit/withdrawal data.
Pipeline integration and ordering crates/common/src/local_db/pipeline/adapters/apply.rs	Derived deltas upsert executes in ApplyPipeline::build_batch after decoded event inserts and before vault balance updates whenever start_block ≤ target_block. Tests verify correct statement ordering within the batch.
Downstream query consumer updates crates/common/src/local_db/query/upsert_vault_balances/insert_balance_changes.sql, crates/common/src/local_db/query/upsert_vault_balances/insert_running_balances.sql	Vault balance calculation queries now source deltas from derived_vault_deltas instead of vault_deltas in their filtered and latest-blocks CTEs, decoupling balance computation from raw event data.
Cleanup and maintenance operations crates/common/src/local_db/query/clear_raindex_data/query.sql, crates/common/src/local_db/query/clear_tables/query.sql	clear_raindex_data adds DELETE for derived_vault_deltas rows; clear_tables adds DROP TABLE for the new table.
Test manifest schema version updates crates/settings/src/local_db_manifest.rs	All test fixtures updated to reflect schema version 4 across validation, error-path, and integration test cases.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

• 0xgleb
• JuaniRios
• hardyjosh

Poem

🐰 A vault of deltas, derived and clean,
With blocks and indices in between,
The pipeline maintains with care each row,
While balances flow where derivatives go! 📊

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 62.50% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures the main change: introducing a new derived_vault_deltas table to materialize vault delta data for use in balance synchronization operations.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch arda/derived-vault-deltas-standalone

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}