Skip to content

feat: add SQLite backend support and migrations#513

Open
niteshpurohit wants to merge 1 commit into
mainfrom
feat/implement-sqlite-backend
Open

feat: add SQLite backend support and migrations#513
niteshpurohit wants to merge 1 commit into
mainfrom
feat/implement-sqlite-backend

Conversation

@niteshpurohit
Copy link
Copy Markdown
Member

  • Introduced a new SQLite backend for Karya, allowing users to configure and use SQLite as a job queue store.
  • Added tests for the SQLite backend to ensure functionality and persistence of job states across instances.
  • Implemented migration generation for SQLite, enabling users to create necessary database tables easily.
  • Updated documentation to include instructions for setting up the SQLite backend.
  • Enhanced existing test frameworks to support SQLite integration testing.

closes: #126
closes: #398

- Introduced a new SQLite backend for Karya, allowing users to configure and use SQLite as a job queue store.
- Added tests for the SQLite backend to ensure functionality and persistence of job states across instances.
- Implemented migration generation for SQLite, enabling users to create necessary database tables easily.
- Updated documentation to include instructions for setting up the SQLite backend.
- Enhanced existing test frameworks to support SQLite integration testing.

closes: #126
closes: #398
@niteshpurohit niteshpurohit self-assigned this May 20, 2026
Copilot AI review requested due to automatic review settings May 20, 2026 04:12
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new local durable SQLite backend to Karya (queue-store + backend wrapper), plus migration generation (ActiveRecord + Sequel), framework-level delegation helpers, and integration/e2e coverage to validate end-to-end behavior and persistence guarantees.

Changes:

  • Introduces Karya::QueueStore::SQLite and Karya::Backend::SQLite, with a shared SQLite schema catalog and durable state snapshot persistence.
  • Adds SQLite migration installers for Karya::ActiveRecord and Karya::Sequel, and exposes delegated install_sqlite_migration helpers in Rails/Roda/Sinatra/Hanami integrations.
  • Expands docs and test/support infrastructure to exercise SQLite in unit, integration, and e2e flows.

Reviewed changes

Copilot reviewed 48 out of 52 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
spec/support/sqlite_e2e_support.rb Adds tmp SQLite DB helper and Sequel URL adapter for e2e tests.
spec/support/framework_sql_backend_support.rb Adds with_sqlite_backend helper for framework integration specs.
gems/karya-sinatra/spec/karya/sinatra_sqlite_e2e_spec.rb Adds Sinatra e2e coverage for SQLite backend + migration application.
gems/karya-sinatra/spec/karya/sinatra_spec.rb Adds unit spec asserting SQLite migration delegation.
gems/karya-sinatra/sig/karya/sinatra.rbs Types new install_sqlite_migration API surface.
gems/karya-sinatra/lib/karya/sinatra.rb Delegates SQLite migration installation to Karya::Sequel.
gems/karya-sinatra/Gemfile.lock Adds sqlite3 dependency resolution for Sinatra gem tests.
gems/karya-sinatra/Gemfile Adds sqlite3 gem dependency.
gems/karya-roda/spec/karya/roda_sqlite_e2e_spec.rb Adds Roda e2e coverage for SQLite backend + migrations.
gems/karya-roda/spec/karya/roda_spec.rb Adds unit spec asserting SQLite migration delegation.
gems/karya-roda/sig/karya/roda.rbs Types new install_sqlite_migration API surface.
gems/karya-roda/lib/karya/roda.rb Delegates SQLite migration installation to Karya::Sequel.
gems/karya-roda/Gemfile.lock Adds sqlite3 dependency resolution for Roda gem tests.
gems/karya-roda/Gemfile Adds sqlite3 gem dependency.
gems/karya-rails/spec/karya/rails_sqlite_e2e_spec.rb Adds Rails e2e coverage for SQLite backend + AR migration execution.
gems/karya-rails/spec/karya/rails_spec.rb Adds unit spec asserting SQLite migration delegation.
gems/karya-rails/sig/karya/rails.rbs Types new install_sqlite_migration API surface.
gems/karya-rails/lib/karya/rails.rb Delegates SQLite migration installation to Karya::ActiveRecord.
gems/karya-hanami/spec/karya/hanami_sqlite_e2e_spec.rb Adds Hanami e2e coverage for SQLite backend + migrations.
gems/karya-hanami/spec/karya/hanami_integration_spec.rb Adds unit/integration spec asserting SQLite migration delegation.
gems/karya-hanami/sig/karya/hanami.rbs Types new install_sqlite_migration API surface.
gems/karya-hanami/lib/karya/hanami.rb Delegates SQLite migration installation to Karya::Sequel.
gems/karya-hanami/Gemfile.lock Adds sqlite3 dependency resolution for Hanami gem tests.
gems/karya-hanami/Gemfile Adds sqlite3 gem dependency.
docs/pages/backends.md Documents how to add sqlite3 gem and configure the SQLite backend.
core/karya/spec/karya/sequel_spec.rb Adds coverage for Sequel SQLite migration generation and default name fallback.
core/karya/spec/karya/queue_store/sq_lite_spec.rb Adds unit coverage for SQLite queue-store behavior, persistence, and failure handling.
core/karya/spec/karya/backend/sq_lite_spec.rb Adds backend wrapper tests including dependency load errors and option validation.
core/karya/spec/karya/active_record_spec.rb Adds coverage for ActiveRecord SQLite migration generation and default name fallback.
core/karya/spec/e2e/karya/cli_worker_sqlite_spec.rb Adds CLI worker e2e coverage for SQLite backend boot + execution.
core/karya/sig/karya/sequel.rbs Types new install_sqlite_migration on Karya::Sequel.
core/karya/sig/karya/queue_store/sqlite/internal/state_codec.rbs Types SQLite state codec snapshot encode/decode surface.
core/karya/sig/karya/queue_store/sqlite/internal/persistence_mutex.rbs Types SQLite persistence mutex and synchronization boundary.
core/karya/sig/karya/queue_store/sqlite/internal/dependency_loader.rbs Types SQLite dependency loader (require_sqlite3!).
core/karya/sig/karya/queue_store/sqlite.rbs Adds RBS for SQLite queue-store public surface (needs completion for nested constants).
core/karya/sig/karya/internal/sqlite_schema_catalog.rbs Types shared SQLite schema catalog and migration renderers.
core/karya/sig/karya/backend/sqlite.rbs Types Karya::Backend::SQLite.
core/karya/sig/karya/backend.rbs Extends backend union to include SQLite.
core/karya/sig/karya/active_record.rbs Types new install_sqlite_migration on Karya::ActiveRecord.
core/karya/lib/karya/sequel.rb Implements Sequel SQLite migration generator using shared schema catalog.
core/karya/lib/karya/queue_store/sqlite/internal/state_codec.rb Implements SQLite durable state encoding/decoding.
core/karya/lib/karya/queue_store/sqlite/internal/persistence_mutex.rb Implements SQLite transactional synchronization + schema bootstrap.
core/karya/lib/karya/queue_store/sqlite/internal/dependency_loader.rb Implements sqlite3 gem loading with actionable LoadError messaging.
core/karya/lib/karya/queue_store/sqlite/internal.rb Wires SQLite queue-store internal components.
core/karya/lib/karya/queue_store/sqlite.rb Implements SQLite queue store (URL parsing, connection mgmt, persistence, fork handling).
core/karya/lib/karya/queue_store.rb Autoloads Karya::QueueStore::SQLite.
core/karya/lib/karya/internal/sqlite_schema_catalog.rb Adds shared SQLite schema DDL + migration template rendering.
core/karya/lib/karya/backend/sqlite.rb Implements SQLite backend wrapper that builds the SQLite queue store.
core/karya/lib/karya/backend.rb Autoloads Karya::Backend::SQLite.
core/karya/lib/karya/active_record.rb Implements ActiveRecord SQLite migration generator using shared schema catalog.
core/karya/Gemfile.lock Adds sqlite3 dependency resolution for core package tests.
core/karya/Gemfile Adds sqlite3 gem dependency.

host = normalize_string(uri.host)
uri_path = uri.path.to_s
raw_path = if host && host != 'localhost'
File.join(host, uri_path)
Comment on lines +210 to +214
it 'falls back to the bootstrap connection when the owner cannot provide one' do
fallback_connection = instance_double(SQLite3::Database)
owner = instance_double(described_class)
allow(owner).to receive(:connection).and_raise(StandardError, 'owner unavailable')
mutex = internal_namespace.const_get(:PersistenceMutex).new(connection: fallback_connection, owner:)
class ConnectionPath
def self.build: (url: String, present_string_class: singleton(PresentString)) -> String
end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Epic: Backend Implementations Implement the SQLite backend

2 participants