From 0fe41ff2e6f91c2b8d831c57d342ef165c3559af Mon Sep 17 00:00:00 2001 From: Younggi Choi <74581798+choiyounggi@users.noreply.github.com> Date: Thu, 3 Sep 2026 11:04:36 +0900 Subject: [PATCH] =?UTF-8?q?feat(wiki):=20add=20databases/selection=20categ?= =?UTF-8?q?ory=20=E2=80=94=20datastore=20choice=20by=20workload?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four researched pages on which database type to choose when designing an architecture: overall datastore-by-workload gate (polyglot persistence), JSONB vs document store, pgvector vs dedicated vector engine, and relational CTEs vs graph database. Domain/root indexes routed, two-way related links added, log entry appended. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01K7rjymauWGk7kPdNc8iKSa --- INDEX.md | 2 +- log.md | 1 + wiki/databases/index.md | 12 ++- .../partial-and-expression-indexes.md | 2 +- .../schema-design/column-data-types.md | 2 +- .../foreign-keys-and-referential-actions.md | 2 +- .../schema-design/requirements-to-tables.md | 2 +- .../choosing-a-datastore-by-workload.md | 80 +++++++++++++++++++ .../graph-workloads-relational-vs-graph-db.md | 70 ++++++++++++++++ .../relational-jsonb-vs-document-store.md | 68 ++++++++++++++++ .../vector-search-engine-selection.md | 74 +++++++++++++++++ 11 files changed, 309 insertions(+), 6 deletions(-) create mode 100644 wiki/databases/selection/choosing-a-datastore-by-workload.md create mode 100644 wiki/databases/selection/graph-workloads-relational-vs-graph-db.md create mode 100644 wiki/databases/selection/relational-jsonb-vs-document-store.md create mode 100644 wiki/databases/selection/vector-search-engine-selection.md diff --git a/INDEX.md b/INDEX.md index 788910d..d644e55 100644 --- a/INDEX.md +++ b/INDEX.md @@ -11,7 +11,7 @@ follow the cross-pointers in their index or take the next matching seeded domain | Domain | Status | Route here when | |--------|--------|-----------------| -| [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior, surveying live data to derive a rule, verifying additive migrations | +| [databases](wiki/databases/index.md) | **seeded** | Choosing a datastore/database type for a workload (relational vs document vs vector vs graph), designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior, surveying live data to derive a rule, verifying additive migrations | | [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, call-site enumeration before a contract change, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, consuming LLM APIs (completion validation, context budgeting), authoring agent-facing artifacts (binding instruction text, agent tool-surface granularity/parity), MAPE-aligned point-prediction calibration, consuming external-API responses, externally-owned defaults, object-storage references, sync-vs-async integration choice, WebSocket/SSE connection lifecycle) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps, packaging data files with `importlib.resources`) | | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility, agent-facing tool surfaces (WebMCP) | | [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting), per-environment/path-valued config, multi-agent orchestration (worker liveness signals, shared run state, tmux pane delivery, completion gates, worktree-isolated workers, autonomous ask-vs-rule decisions, session context/token budgeting) | diff --git a/log.md b/log.md index 0eb14bc..3f8265b 100644 --- a/log.md +++ b/log.md @@ -102,3 +102,4 @@ Append-only. Format: `## [YYYY-MM-DD] `, expression index on extracted fields for equality/range), and validate + required keys with `CHECK` constraints so "flexible" stays "known shapes". +4. If a document store is chosen: enable its schema validation for required + fields, and model references vs embedding by access pattern — flexibility is + a modeling budget, not an excuse to skip modeling. + +## Edge cases + +| Case | Then | +|------|------| +| "Schema flexibility" is wanted only to defer schema design | The schema still exists — it moves into application code, unversioned. Design the stable core as columns now; keep only true variance in JSON | +| JSONB fields need per-field statistics/selectivity for the planner | Extracted expression indexes + `CREATE STATISTICS`; if most queries extract the same fields, promote them to real columns | +| Deep-nested JSONB updated concurrently at different paths | Row-level locking serializes whole-row JSONB updates; high-contention fine-grained updates favor a document store's field-level update operators | +| Team already runs both PostgreSQL and MongoDB | Route by the table above per dataset; keep each fact's system of record single → [databases-selection-choosing-a-datastore-by-workload] | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Adopt MongoDB because "the data is JSON" | Apply step 1's split; JSON at the API boundary says nothing about storage | Most "JSON data" has a stable queried core plus small variance — the core wants columns and constraints | +| Put everything in one JSONB column ("schemaless Postgres") | Typed columns for queried fields, JSONB for the remainder | All-JSONB gives up types, constraints, FKs, and planner statistics — the reasons the relational DB was chosen | + +## Sources + +- https://www.postgresql.org/docs/current/datatype-json.html — JSONB semantics, GIN indexing, when JSON types fit +- https://dev.to/mongodb/postgresql-with-jsonb-and-mongodb-with-schema-2nh0 — both stores cover both models; choose by data- vs application-centric gravity +- https://designgurus.substack.com/p/the-end-of-the-nosql-era-understanding — JSONB write amplification; joins/analytics → PostgreSQL, self-contained hierarchy → MongoDB diff --git a/wiki/databases/selection/vector-search-engine-selection.md b/wiki/databases/selection/vector-search-engine-selection.md new file mode 100644 index 0000000..728cb81 --- /dev/null +++ b/wiki/databases/selection/vector-search-engine-selection.md @@ -0,0 +1,74 @@ +--- +id: databases-selection-vector-search-engine-selection +domain: databases +category: selection +applies_to: [postgresql, pgvector] +confidence: verified +sources: + - https://github.com/pgvector/pgvector + - https://nisai.dev/guides/vector-databases-compared-2026/ + - https://qdrant.tech/blog/pgvector-tradeoffs/ + - https://tensoria.fr/en/blog/vector-database-comparison +last_verified: 2026-09-03 +related: [databases-selection-choosing-a-datastore-by-workload, databases-selection-graph-workloads-relational-vs-graph-db] +--- + +# Choosing Where Embedding/Similarity Search Lives + +## When this applies + +Adding semantic search, RAG retrieval, or recommendation over embeddings, and +deciding between a vector extension in the existing relational database +(pgvector) and a dedicated vector engine (Qdrant, Weaviate, Pinecone, Milvus); +or an existing pgvector setup is hitting limits and you are judging when to +move. + +## Do this + +1. Start with the vector capability of the database you already run (pgvector + with an HNSW index in PostgreSQL). Up to roughly the single-digit millions of + vectors on adequately sized RAM, benchmarks show it matching dedicated + engines, and it keeps embeddings transactionally consistent with their source + rows — no sync pipeline. +2. Filtered search is the common real workload — prefilter with SQL `WHERE` on + the same row's columns (tenant, ACL, date) combined with the vector index; + this joint filtering in one system is pgvector's main structural advantage. +3. Move to a dedicated vector engine when a **specific limit is measured**, not + preemptively: + +| Measured limit | Move indicator | +|----------------|----------------| +| Vector count beyond what one instance's RAM holds (HNSW for 10M × 1536-dim float32 ≈ 60–70 GB) | Dedicated engine with quantization/disk-backed indexes, or dimensionality/quantization reduction first | +| Index rebuild/insert throughput stalls bulk re-embedding | Dedicated engine with faster index maintenance | +| Vector query load starves OLTP (shared buffers/CPU contention) | Separate the search layer so it scales independently | +| Need native horizontal scaling / multi-tenant namespace isolation at hundreds of millions of vectors | Managed dedicated engine | +| Hybrid dense+sparse (BM25 + vector) ranking as a first-class feature | Engine with built-in hybrid search | + +4. When you do split, treat the vector store as a **derived index**: embeddings + are re-derivable projections of source rows, synced one-way (CDC/outbox) — + the general polyglot rule → [databases-selection-choosing-a-datastore-by-workload]. + Practitioners report source↔vector-store sync as the top operational pain of + dedicated stores; budget it as a feature, not glue. + +## Edge cases + +| Case | Then | +|------|------| +| Recall drops after heavy row churn (HNSW graph degrades on deletes) | Scheduled `REINDEX`/index rebuild; if rebuild windows are unacceptable, that is a genuine move indicator | +| Highly selective metadata filter + HNSW returns too few results | Tune `hnsw.ef_search` up, or use iterative index scans (pgvector ≥ 0.8) / partial indexes per hot filter | +| Embeddings change model/dimension | Version the embedding column/collection and re-embed offline; either store choice must support dual-version rollover | +| "We might reach 100M vectors" with no current traffic | Record the trigger metric and stay consolidated — dedicated infra for hypothetical scale is the anti-pattern in the parent page's gate | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Spin up a dedicated vector DB for a first RAG prototype | pgvector in the DB you run (or SQLite-vec/an in-process index for toys) | Prototype scale never exercises what dedicated engines are for; you pay the sync+ops tax immediately | +| Store embeddings only in the vector engine | Keep source text + embedding-version in the system of record; engine holds a projection | Lost/corrupt index becomes a re-embed job instead of data loss | + +## Sources + +- https://github.com/pgvector/pgvector — HNSW/IVFFlat options, filtering, iterative scans +- https://nisai.dev/guides/vector-databases-compared-2026/ — "move only on a specific limit: rebuild time, hybrid search, single-instance scale" +- https://qdrant.tech/blog/pgvector-tradeoffs/ — dedicated-store advantages and the sync pain (from the vendor arguing for moving) +- https://tensoria.fr/en/blog/vector-database-comparison — 1M-scale parity benchmarks; 10M × 1536-dim HNSW RAM footprint