refactor: rewrite backend core — SQLite persistence, on-demand statistics, incremental updates#59
Merged
Conversation
Define the initial library.sqlite projection tables and indexes plus schema-version checks so upcoming sync and query work can build on a stable, test-verified foundation.
Add shared file fingerprint and reconcile classification types so startup and watcher sync can consistently choose targeted reparse scopes.
Replace rusqlite with sqlx for all SQLite access, gaining unified async queries and compile-time embedded migrations via `sqlx::migrate!`.
Only the library DB currently uses resolved_data_dir; stats DB copy and cover cache do not.
Add LibraryBuildPipeline that populates library.sqlite from scanned source files at startup, supporting full and incremental build modes. Incremental mode uses file fingerprint reconciliation to detect added/changed/removed items. Canonical-ID collision detection selects deterministic winners by earliest file path and records diagnostics.
Replace the in-memory ContractSnapshot data source for /api/items and
/api/items/{id} with direct queries against library.sqlite via
LibraryRepository.
Key changes:
- Wire LibraryRepository into ServerState and WebServer
- Rewrite LibraryService to query the DB (async) instead of snapshot
- Add projections module (LibraryItemRow → contract type mapping)
- Add domain-level ItemSort/SortOrder with scope/sort/order query params
- Return 404 not_found for missing /api/items/{id}
- Add InvalidQuery error code with custom message support
- Update frontend API client to use scope= instead of content_type=
- Detail endpoint returns highlights/bookmarks from DB, statistics
empty during transition (stats DB integration comes with includes)
Implement strict include token parsing for /api/items/{id} with
highlights, bookmarks, statistics, completions, and all tokens.
Omitting include returns base payload only; unknown tokens return
400 invalid_query. Align contract duration fields to _sec naming
convention. Statistics and completions data remain nullable pending
stats DB query service integration.
Frontend updated to pass include=all on detail requests and use
the new optional response shape and renamed duration fields.
After each debounced snapshot refresh, the file watcher now also runs an incremental library build against library.sqlite using the existing fingerprint-based reconciliation pipeline. This keeps the SQLite-backed item data in sync with file system changes during serve and watch-static modes.
…ading endpoints Define the response DTOs, domain query types, and shared parameter validation infrastructure that the five /api/reading/* endpoints will build on. All purely additive — no routes or behavior changes yet. New files: - contracts/reading.rs: response data types for summary, metrics, available-periods, calendar, and completions endpoints - domain/reading/queries.rs: validated domain query types (DateRange, ReadingMetric, MetricsGroupBy, PeriodSource, PeriodGroupBy, CompletionsSelector, CompletionsIncludeSet, composed query structs) Updated files: - contracts/common.rs: generic ApiResponse<T> envelope + ResponseMeta - contracts/mod.rs: register reading module - domain/reading/mod.rs: register queries module - server/api/shared.rs: Axum query param structs and parsing functions for all reading endpoints with full validation coverage
Add reading data infrastructure that preserves processed StatisticsData alongside the snapshot, enabling on-demand query computation for new reading endpoints. The ReadingDataStore follows the same thread-safe atomic-replace pattern as SnapshotStore. The summary endpoint computes overview metrics, streaks, and heatmap config from raw stats data at request time, applying scope, date-range, and timezone filters dynamically.
Extract reusable helpers (resolve_time_config, filter_and_resolve_range, filter_stats_by_scope, count_completions_in_range) into shared.rs and move summary computation + compute_streaks into summary.rs. service.rs becomes a thin delegation layer. No behavior change.
Rename legacy metrics.rs to activity.rs (serves /api/activity/* routes) and add new metrics.rs for the /api/reading/metrics endpoint. Supports 6 metric types (reading_time_sec, pages_read, sessions, completions, average_session_duration_sec, longest_session_duration_sec) with day/week/month bucketing. Produces continuous zero-filled time series over the resolved date range. Bucket key helpers (bucket_key_day, bucket_key_week, bucket_key_month) are placed in shared.rs for reuse by upcoming available-periods endpoint.
…d computation Add period discovery endpoint for week/month/year pickers used by Statistics, Calendar, and Recap routes. Supports two selector sources: - source=reading_data: periods with reading activity (week/month/year) - source=completions: periods with completed items (month/year only) Each period entry includes start/end dates and lightweight stats (reading_time_sec, pages_read, completions) for badge rendering.
…tion Add the calendar endpoint that returns month-scoped reading events, item references, and per-scope stats computed on demand from StatisticsData. Consecutive reading days are merged into event spans. - calendar.rs: reading_calendar() with helpers for month range parsing, stats-by-scope computation, event building, and day merging - service.rs: add ReadingService::calendar() dispatch - reading.rs: add reading_calendar async handler - shared.rs: activate calendar params and parser (remove dead_code) - web.rs: register /api/reading/calendar route - 9 unit tests covering date ranges, event merging, scope filtering, and month boundary exclusion
…tcher invalidation Replace legacy snapshot_updated SSE event with data_changed carrying revision_epoch, per-domain revision counters, affected domains list, and generated_at timestamp. The watcher now classifies file events by domain (Library, Metadata, Stats, Assets) and accumulates affected domains across the debounce window for targeted invalidation instead of bumping all domains on every change.
Replace snapshot-based static export with new-contract data files generated
from LibraryService and ReadingService. Exports site.json, items/index.json,
items/{id}.json, reading/summary.json, reading/periods.json,
reading/metrics/{YYYY-MM}.json, reading/calendar/{YYYY-MM}.json, and
reading/completions/{YYYY}.json. Runs alongside legacy snapshot export during
the transition period.
…ates The watcher in WatchStatic mode now re-exports all new-contract data files (site.json, items/, reading/) after each incremental library DB update. Reading data from the snapshot refresh is preserved across the rebuild loop for use by the exporter.
Replace has_activity/has_completions with single has_reading_data capability key across backend contracts and frontend. Update SSE handling from snapshot_updated to data_changed event with domain-aware revision tracking. Merge navigation capability checks into unified has_reading_data guard.
Rewrite shared/contracts.ts with all new reading API response types (ApiResponse<T>, ReadingSummaryData, ReadingMetricsData, ReadingAvailablePeriodsData, ReadingCalendarData, ReadingCompletionsData) and static export aggregate types for client-side scope extraction. Rewrite shared/api.ts to replace activity.* and completions.* namespaces with a unified reading.* namespace backed by /api/reading/* endpoints. Add dual-mode support (serve vs static) with static JSON caching, scope-aware data extraction, and client-side metrics assembly.
Rewrite statistics-data.ts to compose data from reading/summary, reading/metrics, and reading/available-periods endpoints. Align all field names (read_time→reading_time_sec, completed_count→completions, max_scale_seconds→max_scale_sec, etc.) across model, hooks, route, and section components.
Rewrite calendar-data.ts to use reading/available-periods and reading/calendar endpoints. Simplify from 3 parallel per-scope calls to a single call with stats_by_scope. Align field names across all calendar components (item_id→item_ref, time_read→reading_time_sec, total_read_time→reading_time_sec, days_read_pct→active_days_percentage).
Rewrite recap-data.ts to use reading/available-periods (completions, year) and reading/completions endpoints. Replace old domain types with contract types (CompletionsSummary, CompletionItem, CompletionGroup, CompletionsShareAssets). Update RecapSummarySection to compute hours/minutes/days from raw seconds. Align all field names across components (month_key→key, read_time→reading_time_sec, reading_time→reading_time_sec, longest_streak→longest_streak_days).
…ticApiClient implementations Split the monolithic api.ts into a proper interface-based architecture: - ApiClient interface defines the contract for all data access - HttpApiClient handles serve-mode requests to /api/* endpoints - StaticApiClient handles static-mode file loading with client-side scope filtering, metrics assembly, and completions filtering - Lazy client creation ensures mode detection runs after entry point initialization - Fixed static cache not being cleared on site version change in static-mode polling
… detail endpoint Move ReadingData struct from runtime to models so both library and reading domain services can depend on it without violating dependency rules. The library detail service now resolves statistics and completions on demand via partial_md5_checksum linkage into the in-memory reading data when include=statistics or include=completions is requested. Static export also passes reading data through for per-item detail files.
- Add TOML config file loading with `--config` / `-c` flag - Default path `./koshelf.toml` auto-loaded when present - CLI flags override config file values (precedence: CLI > config > defaults) - Add `koshelf.example.toml` with all available settings - Rename CLI `--data-dir` to `--data-path` (kept `--data-dir` as alias) - Restructure app/config into submodules: cli, file, site
…DataPath to DataPath Move config merging orchestration out of cli.rs into mod.rs where it belongs — cli.rs now only defines the CLI struct and validation, while mod.rs mediates between Cli and FileConfig. Also rename CliDataPath to DataPath so the log source label is accurate regardless of whether the value came from CLI or the config file.
When using a persistent DB and switching output directories (e.g. serve mode → --output, or deleting the assets folder), covers were not regenerated because fingerprint reconciliation marked items as unchanged. Now the existing cover_needs_generation check is applied during reconciliation so items with missing covers are automatically re-ingested.
Prevents 404 errors in static export mode by loading the available months list before prefetching, so only months with data are fetched.
…call Add ActiveDays as a new ReadingMetric that counts distinct reading days per bucket. Use it to replace the yearly section's two API calls (summary + 365-point day metrics) with a single 12-point monthly metrics call, moving aggregation server-side and removing aggregateMonthlyStats.
StaticApiClient.getReadingCalendar was ignoring the scope parameter, so calendar events were never filtered by content type in static mode. Add two tests that fail when a new API route is added without a corresponding static export: one counts .route() calls in source and compares to route_paths(), the other checks every route is listed in the export coverage registry.
Harden locate_metadata_path by replacing unwrap() calls on file_stem/parent with the ? operator for safe early return. Unify the rating field to Option<i32> across all API responses to match what sqlx returns from SQLite, removing unnecessary u32 casts.
Move from semver to YYYY.M.patch format and remove the "v" prefix from the sidebar version display.
@event-calendar/build 5.4.1→5.4.2, @event-calendar/core 5.4.1→5.4.2, @types/node 25.3.5→25.5.0, typescript-eslint 8.56.1→8.57.0, vitest 4.0.18→4.1.0
- Replace react-router-dom with react-router (v7 consolidated package) - Remove v7 future flags from HashRouter (now default behavior) - Upgrade @vitejs/plugin-react 4→6 (required by vite 8) - Update all imports across 15 source files
StaticApiClient.getReadingSummary now respects from/to params by computing ranged summaries client-side from daily metric files. Export session duration metrics (longest/average) to support this. Write scope-filtered items files (books.json, comics.json) so the frontend fetches only the needed scope.
…sion Write a .version marker (version + content hash) to the output directory after syncing the embedded frontend. On subsequent runs, skip the copy if the marker matches — avoids redundant file writes in watch mode and still ensures a new binary with updated frontend assets triggers a re-sync.
The version marker check alone isn't enough — if a user deletes frontend files the marker still matches and nothing gets recopied. Now also verify all embedded files exist before skipping the sync.
Move all frontend embedding, versioning, and sync logic out of media.rs into its own frontend.rs module. media.rs now only handles directories and cover generation.
CBR cover extraction used raw archive entry filenames in temp path construction, allowing ../ sequences to write outside the temp dir. EPUB path resolution joined XML-sourced paths without stripping .. components before zip lookups.
- Replace Lua::new() with Lua::new_with(StdLib::NONE) to prevent arbitrary code execution via malicious metadata files (os, io, ffi) - Reject Lua bytecode with ChunkMode::Text to prevent VM exploitation - Remove CorsLayer::permissive() — embedded frontend is same-origin - Add X-Content-Type-Options, X-Frame-Options, Referrer-Policy headers
CSS-first configuration: replace tailwind.config.cjs with @theme block in app.css, @custom-variant dark, @plugin directives, and @source for index.html detection. Convert @layer utilities/components to @Utility blocks. Delete JS config file. PostCSS: swap tailwindcss plugin for @tailwindcss/postcss. Remove @tailwindcss/container-queries (built into v4 core). calendar.css: convert theme('calendar.*') to CSS custom properties, theme('colors.*') to var(--color-*), add @reference "tailwindcss" with minimal @theme subset for @apply support, use text-[0.65rem]/[1rem] arbitrary value for event font sizing. Template class renames (automated): bg-gradient-to-* → bg-linear-to-*, shadow-sm → shadow-xs, shadow → shadow-sm, backdrop-blur-sm → backdrop-blur-xs, flex-shrink-0 → shrink-0, rounded → rounded-sm, focus:outline-none → focus:outline-hidden, !important → postfix ! Manual fixes for v4 behavioral changes: - aspect-book: @theme --aspect-ratio-* doesn't generate utilities in v4, replaced with explicit @Utility block - calendar shadow: rename shadow-sm → shadow-xs in calendar.css @apply (not processed by upgrade tool) - calendar border: remove .ec-main 3-sided border that became visible due to v4 border-color compatibility layer - scope selector: replace space-x-* with gap + explicit padding to fix v4 margin direction change with hidden siblings - leading-none leak: v4's --tw-leading variable persists across responsive text variants; use text-{size}/none on base only so responsive variants (md:text-3xl etc.) use their default line-heights - tsconfig.node.json: add target ES2022 for rolldown compatibility
paviro
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The `fs` module is only used in the non-Windows CBR parsing path, causing an unused import warning on Windows GNU builds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Major internal rewrite of KoShelf's API and data layer:
/api/reading/*with query parameters for scope, date range, and timezone. Old/api/activity/*and/api/completions/*endpoints removed.koshelf.toml(seekoshelf.example.toml). CLI flags still override.StaticApiClientin the frontend.docs/API.md).YYYY.M.patch(this release:2026.3.0).Security
X-Content-Type-Options,X-Frame-Options, andReferrer-PolicyheadersDependency upgrades
Code structure
src/source/— external data sources (KOReader, parsers, scanner, fingerprinting)src/store/— persistence (SQLite + in-memory state stores)src/shelf/— business logic & types (library service, reading statistics)src/server/— HTTP server (api/handlers, response DTOs, embedded frontend)src/pipeline/— orchestration (ingest, rebuild, export, watcher, media, recap)src/app/— bootstrap (CLI, config, entry point)