feat: add SQLite backend support and migrations#513
Open
niteshpurohit wants to merge 1 commit into
Open
Conversation
- Introduced a new SQLite backend for Karya, allowing users to configure and use SQLite as a job queue store. - Added tests for the SQLite backend to ensure functionality and persistence of job states across instances. - Implemented migration generation for SQLite, enabling users to create necessary database tables easily. - Updated documentation to include instructions for setting up the SQLite backend. - Enhanced existing test frameworks to support SQLite integration testing. closes: #126 closes: #398
There was a problem hiding this comment.
Pull request overview
Adds a new local durable SQLite backend to Karya (queue-store + backend wrapper), plus migration generation (ActiveRecord + Sequel), framework-level delegation helpers, and integration/e2e coverage to validate end-to-end behavior and persistence guarantees.
Changes:
- Introduces
Karya::QueueStore::SQLiteandKarya::Backend::SQLite, with a shared SQLite schema catalog and durable state snapshot persistence. - Adds SQLite migration installers for
Karya::ActiveRecordandKarya::Sequel, and exposes delegatedinstall_sqlite_migrationhelpers in Rails/Roda/Sinatra/Hanami integrations. - Expands docs and test/support infrastructure to exercise SQLite in unit, integration, and e2e flows.
Reviewed changes
Copilot reviewed 48 out of 52 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| spec/support/sqlite_e2e_support.rb | Adds tmp SQLite DB helper and Sequel URL adapter for e2e tests. |
| spec/support/framework_sql_backend_support.rb | Adds with_sqlite_backend helper for framework integration specs. |
| gems/karya-sinatra/spec/karya/sinatra_sqlite_e2e_spec.rb | Adds Sinatra e2e coverage for SQLite backend + migration application. |
| gems/karya-sinatra/spec/karya/sinatra_spec.rb | Adds unit spec asserting SQLite migration delegation. |
| gems/karya-sinatra/sig/karya/sinatra.rbs | Types new install_sqlite_migration API surface. |
| gems/karya-sinatra/lib/karya/sinatra.rb | Delegates SQLite migration installation to Karya::Sequel. |
| gems/karya-sinatra/Gemfile.lock | Adds sqlite3 dependency resolution for Sinatra gem tests. |
| gems/karya-sinatra/Gemfile | Adds sqlite3 gem dependency. |
| gems/karya-roda/spec/karya/roda_sqlite_e2e_spec.rb | Adds Roda e2e coverage for SQLite backend + migrations. |
| gems/karya-roda/spec/karya/roda_spec.rb | Adds unit spec asserting SQLite migration delegation. |
| gems/karya-roda/sig/karya/roda.rbs | Types new install_sqlite_migration API surface. |
| gems/karya-roda/lib/karya/roda.rb | Delegates SQLite migration installation to Karya::Sequel. |
| gems/karya-roda/Gemfile.lock | Adds sqlite3 dependency resolution for Roda gem tests. |
| gems/karya-roda/Gemfile | Adds sqlite3 gem dependency. |
| gems/karya-rails/spec/karya/rails_sqlite_e2e_spec.rb | Adds Rails e2e coverage for SQLite backend + AR migration execution. |
| gems/karya-rails/spec/karya/rails_spec.rb | Adds unit spec asserting SQLite migration delegation. |
| gems/karya-rails/sig/karya/rails.rbs | Types new install_sqlite_migration API surface. |
| gems/karya-rails/lib/karya/rails.rb | Delegates SQLite migration installation to Karya::ActiveRecord. |
| gems/karya-hanami/spec/karya/hanami_sqlite_e2e_spec.rb | Adds Hanami e2e coverage for SQLite backend + migrations. |
| gems/karya-hanami/spec/karya/hanami_integration_spec.rb | Adds unit/integration spec asserting SQLite migration delegation. |
| gems/karya-hanami/sig/karya/hanami.rbs | Types new install_sqlite_migration API surface. |
| gems/karya-hanami/lib/karya/hanami.rb | Delegates SQLite migration installation to Karya::Sequel. |
| gems/karya-hanami/Gemfile.lock | Adds sqlite3 dependency resolution for Hanami gem tests. |
| gems/karya-hanami/Gemfile | Adds sqlite3 gem dependency. |
| docs/pages/backends.md | Documents how to add sqlite3 gem and configure the SQLite backend. |
| core/karya/spec/karya/sequel_spec.rb | Adds coverage for Sequel SQLite migration generation and default name fallback. |
| core/karya/spec/karya/queue_store/sq_lite_spec.rb | Adds unit coverage for SQLite queue-store behavior, persistence, and failure handling. |
| core/karya/spec/karya/backend/sq_lite_spec.rb | Adds backend wrapper tests including dependency load errors and option validation. |
| core/karya/spec/karya/active_record_spec.rb | Adds coverage for ActiveRecord SQLite migration generation and default name fallback. |
| core/karya/spec/e2e/karya/cli_worker_sqlite_spec.rb | Adds CLI worker e2e coverage for SQLite backend boot + execution. |
| core/karya/sig/karya/sequel.rbs | Types new install_sqlite_migration on Karya::Sequel. |
| core/karya/sig/karya/queue_store/sqlite/internal/state_codec.rbs | Types SQLite state codec snapshot encode/decode surface. |
| core/karya/sig/karya/queue_store/sqlite/internal/persistence_mutex.rbs | Types SQLite persistence mutex and synchronization boundary. |
| core/karya/sig/karya/queue_store/sqlite/internal/dependency_loader.rbs | Types SQLite dependency loader (require_sqlite3!). |
| core/karya/sig/karya/queue_store/sqlite.rbs | Adds RBS for SQLite queue-store public surface (needs completion for nested constants). |
| core/karya/sig/karya/internal/sqlite_schema_catalog.rbs | Types shared SQLite schema catalog and migration renderers. |
| core/karya/sig/karya/backend/sqlite.rbs | Types Karya::Backend::SQLite. |
| core/karya/sig/karya/backend.rbs | Extends backend union to include SQLite. |
| core/karya/sig/karya/active_record.rbs | Types new install_sqlite_migration on Karya::ActiveRecord. |
| core/karya/lib/karya/sequel.rb | Implements Sequel SQLite migration generator using shared schema catalog. |
| core/karya/lib/karya/queue_store/sqlite/internal/state_codec.rb | Implements SQLite durable state encoding/decoding. |
| core/karya/lib/karya/queue_store/sqlite/internal/persistence_mutex.rb | Implements SQLite transactional synchronization + schema bootstrap. |
| core/karya/lib/karya/queue_store/sqlite/internal/dependency_loader.rb | Implements sqlite3 gem loading with actionable LoadError messaging. |
| core/karya/lib/karya/queue_store/sqlite/internal.rb | Wires SQLite queue-store internal components. |
| core/karya/lib/karya/queue_store/sqlite.rb | Implements SQLite queue store (URL parsing, connection mgmt, persistence, fork handling). |
| core/karya/lib/karya/queue_store.rb | Autoloads Karya::QueueStore::SQLite. |
| core/karya/lib/karya/internal/sqlite_schema_catalog.rb | Adds shared SQLite schema DDL + migration template rendering. |
| core/karya/lib/karya/backend/sqlite.rb | Implements SQLite backend wrapper that builds the SQLite queue store. |
| core/karya/lib/karya/backend.rb | Autoloads Karya::Backend::SQLite. |
| core/karya/lib/karya/active_record.rb | Implements ActiveRecord SQLite migration generator using shared schema catalog. |
| core/karya/Gemfile.lock | Adds sqlite3 dependency resolution for core package tests. |
| core/karya/Gemfile | Adds sqlite3 gem dependency. |
| host = normalize_string(uri.host) | ||
| uri_path = uri.path.to_s | ||
| raw_path = if host && host != 'localhost' | ||
| File.join(host, uri_path) |
Comment on lines
+210
to
+214
| it 'falls back to the bootstrap connection when the owner cannot provide one' do | ||
| fallback_connection = instance_double(SQLite3::Database) | ||
| owner = instance_double(described_class) | ||
| allow(owner).to receive(:connection).and_raise(StandardError, 'owner unavailable') | ||
| mutex = internal_namespace.const_get(:PersistenceMutex).new(connection: fallback_connection, owner:) |
| class ConnectionPath | ||
| def self.build: (url: String, present_string_class: singleton(PresentString)) -> String | ||
| end | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes: #126
closes: #398