eloqdata · liunyl · Jun 13, 2026 · Jun 12, 2026 · Jun 13, 2026 · Jun 13, 2026
diff --git a/.gitignore b/.gitignore
@@ -12,6 +12,9 @@
 
 AGENTS.md
 
+# Superpowers scratch (plans/specs), not part of the repo
+docs/superpowers/
+
 # Prerequisites
 *.d
 

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -0,0 +1,76 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What This Repo Is
+
+EloqKV is a distributed, Redis/Valkey-compatible database with ACID transactions. This repo contains the **Redis API layer** only; the transaction/storage engine lives in the `data_substrate` submodule (GitHub repo `eloqdata/tx_service`), which has its own CLAUDE.md. Most non-protocol work (concurrency control, transactions, storage, WAL) happens in the submodule.
+
+After cloning: `git submodule update --init --recursive`.
+
+## Technical Docs — Read These First
+
+`docs/` contains module-by-module design documentation (index: `docs/README.md`). **Before working on an unfamiliar module, read its doc**: `01` overview/bootstrap, `02` command processing & transactions, `03` data model (objects/commands/catalog), `04` Lua/pub-sub/blocking commands, `05` namespaces, `06` vector search, `07` RDB-AOF interop & tools. Engine internals are documented in `data_substrate/docs/`.
+
+**Maintenance rule: when a code change alters behavior described in `docs/`, update the corresponding doc in the same change.** Each doc lists the source files it covers.
+
+## Common Commands
+
+```bash
+# Configure + build (out-of-source; bld/ and install/ are gitignored output dirs)
+mkdir -p bld && cd bld
+cmake .. -DCMAKE_BUILD_TYPE=Debug \
+    -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
+    -DWITH_DATA_STORE=ELOQDSS_ROCKSDB \
+    -DWITH_LOG_STATE=ROCKSDB \
+    -DOPEN_LOG_SERVICE=OFF \
+    -DCMAKE_INSTALL_PREFIX=../install
+cmake --build . --parallel 16
+cmake --install . --prefix ../install
+
+# Run a single-node server (port 6379, foreground)
+./install/bin/eloqkv --config=eloqkv.ini
+
+# Run a single TCL test file (e.g. tests/unit/eloq/hash.tcl)
+# against the running server; loop over tests/unit/eloq/*.tcl for the full suite
+tclsh tests/test_helper.tcl --host 127.0.0.1 --port 6379 \
+    --tags -needs:repl --tags -needs:config-maxmemory --tags -needs:debug \
+    --tags -needs:redis_config --tags -needs:redis_expire --tags -needs:slow_test \
+    --tags -needs:support_cmd_later --tags -needs:cluster_mode \
+    --single /unit/eloq/hash
+
+# Format changed C/C++ files (clang-format-18; repeat inside data_substrate/)
+git diff --name-only --diff-filter=ACMR \
+    | grep -E '\.(cpp|cc|cxx|c|hpp|hh|hxx|h)$' | xargs -r clang-format-18 -i
+```
+
+Verify behavior manually with `redis-cli -h 127.0.0.1 -p 6379`.
+
+Notes:
+- Dependencies: use the `eloqdata/eloq-dev-ci-ubuntu2404` Docker image or run `scripts/install_dependency_ubuntu2404.sh` (ubuntu2404 preferred).
+- Debug builds enable fault injection (`WITH_FAULT_INJECT`); add `--tags -needs:fault_inject` to the TCL command on non-Debug builds.
+- Key CMake options: `WITH_DATA_STORE` (storage backend: `ELOQDSS_ELOQSTORE` default, `ELOQDSS_ROCKSDB` common for local dev, also DynamoDB/BigTable/RocksDB-Cloud variants), `WITH_LOG_STATE`, `WITH_LOG_SERVICE`, `OPEN_LOG_SERVICE`, `BUILD_ELOQKV_AS_LIBRARY` (build as library for a converged binary instead of the `eloqkv` executable).
+
+## Architecture
+
+Command flow, top to bottom:
+
+1. **Entry**: `src/redis_server.cpp` (`eloqkv` executable) starts a brpc server with `EloqKV::RedisServiceImpl` (`src/redis_service.cpp`, ~the largest file in the repo) as the Redis-protocol service. Config comes from gflags / an ini file (`eloqkv.ini` template at repo root).
+2. **Command parsing**: RESP requests are parsed into command objects (`include/redis_command.h`, `src/redis_command.cpp`). Replies are written through `redis_replier.h` / `output_handler.h`.
+3. **Data objects**: each Redis type is a `txservice::TxObject` subclass — `redis_string_object`, `redis_hash_object`, `redis_list_object`, `redis_set_object`, `redis_zset_object` (see `include/redis_object.h` for the base). Commands are applied to these objects inside the tx service, not in the protocol layer.
+4. **Into the engine**: `RedisServiceImpl` wraps commands in `txservice` requests (`ObjectCommandTxRequest`, `MultiObjectCommandTxRequest`, defined in `data_substrate/tx_service/include/tx_request.h`) and drives them through a `TransactionExecution` state machine. Keys are `EloqKey` (`include/eloqkv_key.h`); the schema/table mapping is `eloqkv_catalog_factory.h`.
+5. **Engine**: the `data_substrate` submodule shards data into CcShards, each owned by a TxProcessor pinned to a core, and handles concurrency control, WAL (log service), checkpointing, and cold-data storage. See `data_substrate/CLAUDE.md` and the module design docs in `data_substrate/docs/`.
+
+Auxiliary subsystems in this layer:
+- **Lua/scripting**: `include/lua_interpreter.h`, vendored `lua/`.
+- **Pub/Sub**: `include/pub_sub_manager.h`.
+- **Namespaces** (multi-tenancy): `include/namespace/`, `src/namespace/`.
+- **Vector search**: `include/vector/`, `src/vector/`.
+- **Tools**: `src/tools/eloqkv2aof`, `src/tools/eloqkv2rdb` (export to Redis AOF/RDB formats).
+- Vendored Redis C sources (`src/redis/`), `crcspeed/`, `fpconv/`.
+
+Threading model to keep in mind: request handlers run on brpc **bthreads** (coroutines on a custom brpc fork), while CC request `Execute()` runs on shard/TxProcessor context. Do not introduce `bthread::Mutex`/`ConditionVariable` shared between shard-side `Execute()` and bthread waiters — this can permanently deadlock a brpc worker. Use `std::atomic` state + `bthread_usleep` backoff polling instead (see data_substrate's CLAUDE.md for details).
+
+## Code Style
+
+Google C++ style, enforced by clang-format-18 (`.clang-format` at repo root). C++20. Naming: functions/classes `MyName`, locals `my_name`, members `my_name_`, enumerators `kEnumName`. Full project conventions: `data_substrate/style_guide.md`.
diff --git a/docs/01-architecture-overview.md b/docs/01-architecture-overview.md
@@ -0,0 +1,69 @@
+# Architecture Overview
+
+EloqKV is a Redis/Valkey-compatible distributed database. This repo implements the **Redis protocol layer**: a brpc-based RESP server (`RedisServiceImpl`) that parses commands into command objects, applies them to Redis-type data objects (`TxObject` subclasses), and drives everything through the transaction engine in the `data_substrate/` submodule (documented separately under `data_substrate/docs/`). The layer registers itself with the engine as `TableEngine::EloqKv`, supplying a `CatalogFactory` so the engine can create EloqKV's concrete keys, records, and cc maps without knowing anything about Redis.
+
+## Component Map
+
+| Area | Key files | Doc |
+|---|---|---|
+| Bootstrap / main | `src/redis_server.cpp` | this file |
+| Service & dispatch | `src/redis_service.cpp` (~6.5k lines), `include/redis_service.h`, `redis_handler.*`, `redis_connection_context.*` | [02](02-command-processing.md) |
+| Output / errors / stats | `redis_replier.*`, `output_handler.h`, `redis_errors.*`, `redis_stats.*` | [02](02-command-processing.md) |
+| Commands | `include/redis_command.h` (~8k lines), `src/redis_command.cpp` (~21k lines) | [03](03-data-model.md) |
+| Data objects | `redis_object.h`, `redis_{string,hash,list,set,zset}_object.*` | [03](03-data-model.md) |
+| Engine plug-in | `eloqkv_key.*` (EloqKey = TxKey impl), `eloqkv_catalog_factory.*` | [03](03-data-model.md) |
+| Lua scripting | `lua_interpreter.*`, `lua_output_handler.h`, vendored `lua/` | [04](04-scripting-pubsub-blocking.md) |
+| Pub/Sub | `pub_sub_manager.*` | [04](04-scripting-pubsub-blocking.md) |
+| Namespaces (multi-tenancy) | `include/namespace/`, `src/namespace/` | [05](05-namespaces.md) |
+| Vector search | `include/vector/`, `src/vector/` | [06](06-vector-search.md) |
+| RDB/AOF interop & tools | `redis_rdb_restore.*`, `src/tools/eloqkv2{rdb,aof}/`, `crcspeed/`, `fpconv/` | [07](07-persistence-and-tools.md) |
+| Vendored Redis C utils | `src/redis/` (dict, sha1, siphash, zmalloc, commands…) | — |
+| Engine | `data_substrate/` submodule | `data_substrate/docs/` |
+
+## Process Bootstrap (`src/redis_server.cpp`)
+
+`main()` wires the protocol layer into the engine in four ordered steps (mirroring `DataSubstrate`'s lifecycle, see `data_substrate/docs/01-architecture-overview.md`):
+
+1. **`DataSubstrate::Instance().Init(config_file)`** — after `ConvertEloqkvFlagsToTxFlags()` translates eloqkv-named flags (`ip`/`port`/`ip_port_list`/standby/voter lists) into the engine's `tx_*` flags. gflags override ini values throughout (`CheckCommandLineFlagIsDefault` pattern).
+2. **`RedisServiceImpl::Init(server)`** (`src/redis_service.cpp:219`) — builds the prebuilt table list and calls `DataSubstrate::RegisterEngine(TableEngine::EloqKv, &catalog_factory, nullptr, prebuilt_tables, engine_metrics, eloqkv_publish_func)`:
+   - one engine table per Redis database: `data_table_0` … `data_table_<databases-1>` (`databases` from config, default 16), each `TableType::Primary` + `TableEngine::EloqKv`;
+   - two namespace system tables: `__ns_0` (namespace registry) and `ns_data_0` (shared namespace data) — see [05-namespaces.md](05-namespaces.md);
+   - per-command duration/total metrics and read/write aggregates;
+   - `eloqkv_publish_func` so the engine can deliver cross-node PUBLISH messages back into this layer ([04](04-scripting-pubsub-blocking.md)).
+3. **`DataSubstrate::Instance().Start()`** — boots log service, storage handler, and the tx service (TxProcessors, Sharder, checkpointer…).
+4. **`RedisServiceImpl::Start(server)`** (`src/redis_service.cpp:635`) then brpc server start — the service object is installed as `brpc::ServerOptions::redis_service` (the brpc Redis protocol entry point; brpc owns and deletes it), `num_threads` is pinned to the engine's `bthread_concurrency` (= `core_number`), builtin brpc services are disabled, and TLS is configured when enabled. The server listens on `eloqkv_port` (default 6379).
+
+Shutdown is the reverse: stop accepting, `RedisServiceImpl::Stop()`, `DataSubstrate::Shutdown()`.
+
+## How a command becomes engine work (one paragraph; details in [02](02-command-processing.md)/[03](03-data-model.md))
+
+brpc parses RESP on a bthread and calls into `RedisServiceImpl`; the command name is looked up in the dispatch table; a command object is constructed (parse/validate args); the service obtains a `TransactionExecution` from the engine (`NewTxm`, pinned to the current core's shard) and submits an `ObjectCommandTxRequest` / `MultiObjectCommandTxRequest`; the engine routes the command to the owner shard (local or remote) where it executes against the in-memory `TxObject`; results flow back through the request's `TxResult`, and the bthread renders the RESP reply via the replier. Single commands auto-commit; `MULTI/EXEC` and the session-style `BEGIN/COMMIT/ROLLBACK` map onto one engine transaction.
+
+Because connection bthreads both *submit* transactions and *drive* the engine (brpc workers double as tx processors — `data_substrate/docs/02-threading-model.md`), this layer must follow the engine's threading contract: never block a bthread on a synchronization primitive shared with shard-side `Execute()` code; use the yield/resume functors on `TxRequest` or atomic + `bthread_usleep` patterns.
+
+## Configuration Surface (layer-specific)
+
+Defined in `src/redis_server.cpp` / `src/redis_service.cpp`; engine flags are listed in `data_substrate/docs/01-architecture-overview.md`.
+
+| Flag | Default | Meaning |
+|---|---|---|
+| `config` | "" | ini file path (same file is handed to the engine) |
+| `eloqkv_port` | 6379 | RESP listen port |
+| `ip` / `port` / `ip_port_list` / `standby_ip_port_list` / `voter_ip_port_list` | — | translated to engine `tx_*` flags at startup |
+| `databases` (ini `[local]`) | 16 | number of Redis databases = number of `data_table_<n>` tables |
+| `requirepass` (ini) | "" | AUTH password |
+| `cluster_mode` | — | strict Redis Cluster compatibility behavior |
+| `txn_isolation_level` / `protocol` / `isolation_level` | — | engine isolation/cc-protocol selection for txs ([02](02-command-processing.md)) |
+| `retry_on_occ_error` | — | auto-retry policy for OCC conflicts |
+| `enable_tls` / `tls_cert_file` / `tls_key_file` | off | TLS on the RESP port |
+| `slow_log_threshold` / `slow_log_max_length` | — | SLOWLOG |
+| `enable_redis_stats`, `enable_cmd_sort` | — | INFO/stats behavior |
+| `cc_notify` | — | notify-based (vs polling) wakeup between layer and engine |
+| `vector_index_worker_num` | — | vector index worker threads ([06](06-vector-search.md)) |
+| `maxclients`, `enable_io_uring`, … | — | engine flags commonly set from eloqkv configs |
+
+## Build shapes
+
+- Default: `eloqkv` executable (`src/redis_server.cpp` + the `RESELOQ` library).
+- `BUILD_ELOQKV_AS_LIBRARY=ON`: builds `eloqkv_lib` for converged multi-engine binaries (the engine's `EnableEngine`/`WaitForEnabledEnginesRegistered` flow exists for this).
+- Offline tools `eloqkv_to_rdb` / `eloqkv_to_aof` ([07](07-persistence-and-tools.md)); storage backend selected by `WITH_DATA_STORE` (see repo `CLAUDE.md`).