Remove DynamoDB and Bigtable storage backends

[thweetkomputer]

Context

DynamoDB and Bigtable are no longer supported storage backends, but EloqKV still advertised both WITH_DATA_STORE values and retained backend-specific catalog branches. The enterprise build helper also passed the already-removed OPEN_LOG_SERVICE option.

This PR intentionally points data_substrate at the companion PR branch commit so EloqKV CI validates the combined change:

• Remove DynamoDB and Bigtable storage backends tx_service#536
• data_substrate commit: 550fe3db8495b060f2e2a74bbf756be620e4ad6a

Behavior before and after

Before, top-level CMake advertised and accepted DYNAMODB and BIGTABLE; OPEN_LOG_SERVICE was passed despite having no consumer.

After, the supported WITH_DATA_STORE values are ROCKSDB, ELOQDSS_ROCKSDB, ELOQDSS_ROCKSDB_CLOUD_S3, ELOQDSS_ROCKSDB_CLOUD_GCS, and ELOQDSS_ELOQSTORE. Removed values fail explicitly with Unknown WITH_DATA_STORE.

Implementation

• Remove DynamoDB and Bigtable from the top-level CMake cache choices and compile-definition branches.
• Remove DynamoDB-specific catalog construction and no-op Redis service branches.
• Remove the stale OPEN_LOG_SERVICE argument from .github/scripts/common.sh.
• Update build/backend documentation.
• Advance the data_substrate pointer from 82cea28 to the companion PR commit 550fe3d.

Design decisions and alternatives

The parent PR carries the unmerged companion commit instead of waiting for the submodule PR to land, allowing the full EloqKV CI matrix to exercise both repositories together. The parent pointer must be refreshed to the merged data_substrate/main commit before final merge if the companion PR is squash-merged to a different SHA.

Test plan

• Unit/TCL tests
• Integration or manual validation
• Formatting/build checks
• Compatibility or performance validation, when relevant

Commands and results:

cmake --build bld-rocksdb --parallel 16                    # passed
cmake --build bld-eloqstore-local --parallel 16            # passed
cmake -S . -B /tmp/eloqkv-invalid-dynamodb... -DWITH_DATA_STORE=DYNAMODB ...  # failed as expected
cmake -S . -B /tmp/eloqkv-invalid-bigtable... -DWITH_DATA_STORE=BIGTABLE ...  # failed as expected
bash -n .github/scripts/common.sh data_substrate/scripts/third_party/build-ubuntu2404.sh  # passed
git diff --check                                           # passed in both repositories

clang-format-18 was unavailable in the workspace. No runtime/TCL tests were run locally; this PR carries the companion branch commit specifically to run the repository CI matrix.

Risk assessment

This intentionally breaks configurations that still select DynamoDB or Bigtable. Supported RocksDB and EloqStore builds passed locally. The principal integration risk is the temporary submodule pointer to an unmerged commit; reviewers should verify it is replaced with the final reachable data_substrate/main commit before merging EloqKV.

Rollback plan

Revert this PR and the companion data_substrate PR.

Reviewer guide

Review the top-level CMakeLists.txt, src/eloqkv_catalog_factory.cpp, and the data_substrate pointer first. The full backend implementation deletion is in tx_service PR #536.

Follow-up work

After tx_service PR #536 is merged, update this PR's submodule pointer to the resulting commit on data_substrate/main and rerun CI before merge.

Summary by CodeRabbit

• Configuration
  • Simplified supported data-store options to RocksDB and embedded RocksDB variants, removing DynamoDB/Bigtable paths.
  • Unsupported values now fail fast with an explicit “unknown WITH_DATA_STORE” error.
• Improvements
  • Enhanced Cassandra-backed catalog schema handling for pre-created table creation, truncation, and flush operations.
  • Updated build settings to enable transaction processing, tests, and the log service.
• Documentation
  • Refined architecture and CMake documentation to reflect the current persistence and available storage options.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ca46cf77-f510-4236-9d42-50df2efa6ab7

📥 Commits

Reviewing files that changed from the base of the PR and between 001331c and 5a93942.

📒 Files selected for processing (1)

• data_substrate

🚧 Files skipped from review as they are similar to previous changes (1)

• data_substrate

---

Walkthrough

The change removes DynamoDB and Bigtable from CMake data-store selection, updates RocksDB catalog handling, serializes Cassandra schema images during catalog operations, changes build flags, refreshes the data substrate revision, and updates related documentation.

Changes

Storage configuration and catalog integration

Layer / File(s)	Summary
Build and storage configuration .github/scripts/common.sh, CMakeLists.txt, CLAUDE.md, README.md	Build flags, supported WITH_DATA_STORE values, unsupported-value errors, and storage documentation are updated for RocksDB and EloqStore configurations.
RocksDB catalog selection src/eloqkv_catalog_factory.cpp, data_substrate	Catalog factory compilation and Redis table schema initialization now use RocksDB catalog information instead of DynamoDB-specific handling, alongside a data_substrate revision update.
Cassandra schema image serialization src/redis_service.cpp	Cassandra initialization, flush-db, and flush-all paths now serialize CassCatalogInfo schema images before catalog operations.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• eloqdata/eloqkv#512: Updates build configuration to remove the OPEN_LOG_SERVICE flag and use WITH_LOG_SERVICE.
• eloqdata/eloqkv#383: Updates the tracked data_substrate submodule reference.
• eloqdata/eloqkv#482: Updates the data_substrate submodule reference.

Suggested labels: trigger-ci

Poem

A rabbit hops through RocksDB’s door,
While Dynamo paths are seen no more.
Cassandra’s images neatly align,
Build flags glow in a tidy line.
“Hop hop!” says the hare—the catalogs shine!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly matches the main change: removing DynamoDB and Bigtable storage backends.
Description check	✅ Passed	The description follows the template and covers context, behavior, implementation, testing, risks, rollback, reviewer guidance, and follow-up work.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/remove-dynamodb-bigtable-backends

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@CMakeLists.txt`:
- Around line 168-170: Complete RocksDB schema-image support before enabling the
ROCKSDB build definition: update the prebuilt table registration path using
GenKvTableName to generate and register the expected RocksDB catalog metadata
image, and replace the TODO(lokax) ImageDataStore branches in catalog reads and
writes with the corresponding schema-image handling. Only expose
DATA_STORE_TYPE_ROCKSDB after these paths produce and consume valid, consistent
catalog metadata.

In `@data_substrate`:
- Line 1: Update the data_substrate gitlink to the final reachable commit on the
eloqdata/tx_service main branch after the submodule PR is squash-merged,
replacing the outdated parent reference while following the repository’s
submodule update policy.

In `@README.md`:
- Line 175: Move the architecture paragraph describing TxService, Log Service,
and RocksDB/EloqStore storage outside the surrounding HTML comment in README.md
so it renders as visible user-facing documentation; preserve the paragraph text
and surrounding documentation structure.

In `@src/eloqkv_catalog_factory.cpp`:
- Line 64: Update the catalog-info construction in the RocksDB path to use the
RocksDB-specific concrete KVCatalogInfo type, ensuring its declaration is
included and the type is properly qualified or imported within namespace EloqKV.
Preserve the existing kv_info_ ownership and initialization behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 4f24dcd5-a580-4aca-8eee-bf911f87c21d

📥 Commits

Reviewing files that changed from the base of the PR and between 52ad831 and 001331c.

📒 Files selected for processing (7)

• .github/scripts/common.sh
• CLAUDE.md
• CMakeLists.txt
• README.md
• data_substrate
• src/eloqkv_catalog_factory.cpp
• src/redis_service.cpp

💤 Files with no reviewable changes (2)

• .github/scripts/common.sh
• src/redis_service.cpp