lore-0049: rewrite overview to match ADR 0007 (Hetzner CH)#26
Open
karczuRF wants to merge 2 commits into
Open
Conversation
Sixteen sections touched (§0, §1.1, §1.2, §2.1, §2.3, §3, §4.5, §5.2–§5.4, §5.6, §6, §7, §8, §9, §10, §11) plus a new Revision History row. The live data sink description flips from Prices- owned RDS PostgreSQL to BE's shared Hetzner ClickHouse cluster (separate `prices` database) per ADR 0007 (accepted in PR #25). Key shape changes: - §1.1 / §1.2 diagrams: ClickHouse + SNS fan-out replace RDS + direct S3 → Lambda. - §2.1: RDS, NAT Gateway, OHLCV Rollup Lambda moved into a "no longer in the budget" sub-list; Secrets Manager line updated to mention per-env mTLS material. - §2.3: Hetzner CH data plane and mTLS CA added as shared components; VPC + NAT Gateway moved to a "no longer shared because not needed" callout. - §3: schema rewritten from PostgreSQL native range partitioning to ClickHouse — `ReplacingMergeTree(version)` per-source rows on per-granularity tables (`price_ohlcv_1m`/`_15m`/.../`_1M`) with a materialised-view rollup chain that eliminates the OHLCV Rollup Lambda. Backfill progress on `ReplacingMergeTree(updated_at)`. Cleanup via `ALTER TABLE … DROP PARTITION`. - §5.2: UPSERT → ReplacingMergeTree INSERT (version-keyed idempotency); HTTPS-mTLS to Caddy:443; warm connection reuse to amortise the ~80-130 ms cross-cloud RTT. - §5.3 / §5.4: OHLCV Rollup row + EventBridge rule removed. - §6: RDS sizing/scaling ladder removed; replaced with a short CH-on-Hetzner sizing note pointing to the sidecar-CH fallback (ADR 0007 Alternative 3) if task 0047 returns RED. - §7: mTLS section added (per-env certs, 1-year rotation, CA revocation, NotAfter alarm). - §8: sqlx → `clickhouse` Rust crate; Hetzner CH + Caddy in the stack table. - §10: $12/mo RDS line removed; ~$1-2/env/mo Hetzner CH cost-share added per task 0046's empirical ~0.45 GB/yr footprint. Steady- state per env: ~$108/mo. The RDS escalation ladder (db.r6g.large + Multi-AZ + read replica + Proxy) eliminated. - §11: sharing table refreshed — Hetzner CH and mTLS CA added; agreement record from task 0045 referenced in §11.1, §11.2, and §11.4 risk table. Preserved unchanged: - §5.5 L1/L2/L3 VWAP layering callout (added by task 0048). - Local backfill sections (Stream 1 ADR 0001, Stream 2 ADR 0005) — only the cloud-push targets shift RDS → Hetzner CH `prices.*`. Final grep confirms no live-state RDS / NAT Gateway / OHLCV Rollup / sqlx references remain in the document body — only historical context in the Revision History and explicit "removed" callouts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
karczuRF
force-pushed
the
lore-0049/overview-rewrite-adr-0007
branch
from
d1ee0e5 to
20bb744
Compare
Companion to the main overview rewrite (commit 20bb744). Brings docs/database-schema/ in line with ADR 0007 (live data sink on shared Hetzner ClickHouse): - amm-trades-schema.md: added a clear SUPERSEDED banner pointing at ADR 0001, ADR 0007, and BE ADRs 0044/0045. The pre-ADR-0001 design (custom AMM trades table on BE's RDS) is no longer pursued; the file is retained for historical context only. - database-schema-overview.md (1540 → 1906 lines, +366 net): full rewrite across all 13 sections + both Appendices. - Revision History row added. - §1.1 / §1.2 diagrams: ClickHouse + SNS fan-out + mTLS Caddy edge replace RDS + direct S3. - §2 Tech Stack: PostgreSQL 16 → ClickHouse on Hetzner; sqlx → `clickhouse` Rust crate; MV-chain rollups; mTLS material in Secrets Manager. - §3 Schema: full rewrite to ClickHouse DDL — `ReplacingMergeTree(version)` per-source rows on per-granularity `price_ohlcv_*` tables, MV chain sketch for the 1m → 15m → ... → 1M rollup, `prices.assets` / `current_prices` / `oracle_prices` / `backfill_progress` on appropriate engines. - §4 Retention: `ALTER TABLE ... DROP PARTITION` per per-granularity table (no row DELETEs). - §5 reframed as "Sort Keys & Query Patterns" — CH sort key semantics, projections-not-B-tree-indexes. - §6 Workers: OHLCV Rollup Lambda row removed; MV chain noted as the replacement; `ohlcv-rollup` EventBridge rule gone. - §7 Backfill: both streams' diagrams + metric tables reflect local CLIs + Hetzner CH push targets; `task_healthy` / `last_heartbeat` swapped for `last_push_at` everywhere; §7.6 example response uses the new field shape. - §8 reframed as "Sizing, Performance, Scaling (Hetzner ClickHouse)" — RDS escalation ladder removed, sidecar-CH fallback documented, cost lines updated to ~$1-2/env/mo cost-share per task 0046. - §9 Security: mTLS sub-bullets (per-env certs, 1-year rotation, CA revocation, NotAfter alarm); no-VPC framing; Borg RPO trade-off. - §10 Cross-Service Dependency: rewritten around Hetzner shared tenancy (ADR 0007); mermaid block shows the co-tenancy boundary. - §11 What Is Not Shared: RDS instance row replaced with `prices.*` schema + onboarding portal rows. - §12 Tranche-1 acceptance: flipped to ClickHouse / no-VPC / SNS / mTLS / push-cadence. - §13 Quick Reference: engine + sort-key + partition columns per per-granularity table. - Appendix A (ER diagram): full rewrite to CH types, engines, MV-chain edges, no SQL FKs. - Appendix B (full system diagram): full rewrite — SNS bucket fan-out, mTLS edge, Hetzner box with shared CH (default.* + prices.*), workstation backfill subgraph (BE backfill-runner, local CH, local Postgres, cloud-push tools), MV chain dotted edges, push-freshness alarm. clickhouse-prod-schema.sql left as-is (BE's `default.*` production reference, not prices-api's schema). Task 0049 closed (expanded scope; reopened earlier today as this work was identified). Final grep confirms no live-state RDS / sqlx / PostgreSQL 16 / OHLCV Rollup / ECS Fargate references remain in current-state text. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rewrite
docs/prices-api-general-overview.mdto match ADR 0007 (live data sink on shared Hetzner ClickHouse), accepted in PR #25. Sixteen sections touched plus a new Revision History row.Note on stacking. This branch is stacked on top of
lore-0045/close-be-agreementso the ADR 0007 accepted state is in the working base. Until #25 merges, this PR's diff will appear to include both sets of changes. GitHub will rebase the diff once #25 lands.Section-by-section
ReplacingMergeTree(version)per-source, per-granularity tables, MV chain, monthly partitions,ALTER TABLE DROP PARTITIONcleanup/backfill/statussqlx→clickhousecrate; PostgreSQL 16 → ClickHouse + CaddyVerification
Final grep on the doc confirms:
sqlxorPostgreSQL 16outside the historical Revision Historydb.t4g.*/db.m6g.*/db.r6g.*/ RDS Proxy / Multi-AZ / RDS read replica in live-state textohlcv-rollup/ "OHLCV Rollup Lambda" except in explicit "removed" calloutsRDSmentions are confined to: Revision History rows, explicit "removed" / "no longer" callouts, and the §2.3 paragraph documenting obsolete table rowsNet change: +788 / -384 lines.
Test plan
docs/prices-api-general-overview.mdreferences in other lore tasks (0011/0038/0039/0040) still resolveFollow-ups (out of scope)
docs/database-schema/clickhouse-prod-schema.sql— the full schema reference referenced from §3.2 still needs to be writtenlore/3-wiki/🤖 Generated with Claude Code