ADFA-4949 feat(kolibri): seeding groundwork — credential store, session, readiness gate - #302
Merged
Merged
Conversation
…on, readiness gate
Foundation for seeding Kolibri content from the app over REST. Adds a fourth
JobType ('kolibri') to the durable job engine (ADFA-4838); the runner calls
Kolibri's REST API on 127.0.0.1:8009 and does not spawn kolibri manage.
Kolibri's CLI reports no parseable progress (click.progressbar with no label:
a full import emits one blank line on a non-TTY pipe), and db.sqlite3 is no
better since `available` is set in one pass at the end. The task API exposes
percentage, transferred_file_size and total_resources.
New:
sockets/credentials.ts store: env -> persisted override -> default
sockets/kolibri.session.ts login, CSRF, ensureContentOrigin, readiness
sockets/kolibri.map.ts pure helpers, no native deps (CI runs with
--ignore-scripts, so tests must not pull in
jobs.ts -> better-sqlite3)
sockets/kolibri.exec.ts runner: enqueue + poll
sockets/kolibri.query.ts catalog, tree, estimate, installed channels
sockets/kolibri.session.test.ts 25 unit tests
tools/kolibri-preflight.sh POSIX diagnostics, system-stopped path
Routes added before the generic /:type/* handlers: /kolibri/{ready,preflight,
channels,catalog,resolve,tree,estimate,delete} and /credentials/:service.
Credential update validates against Kolibri before persisting: 401 rejected,
403 authenticates without can_manage_content, 503 unreachable. GET omits the
password and reports isDefault.
Content origin: importcontent always calls lookup_channel_listing_status(),
which reaches NetworkClient.discover_from_address(). Without a matching
NetworkLocation it falls back to ifaddr.get_adapters(), which fails under
proot on Android 13+. known_location_for_address() does not filter by
location_type and StaticNetworkLocationViewSet is a full ModelViewSet, so
ensureContentOrigin() creates a static row over REST. Idempotent, and also
satisfied by the reserved rows IIAB seeds (ADFA-4695).
Guards for the three failure modes Kolibri does not surface: job stuck in
QUEUED (workers not running), RUNNING with no byte progress (unbounded retry
loop), and COMPLETED with nothing transferred (non-existent node_ids).
Verified off-device: tsc --noEmit clean, build OK, 41/41 tests. Runner
exercised against a mock Kolibri across ten scenarios. Not run against a real
Kolibri — see the acceptance criteria on ADFA-4949.
…the read path Review follow-up on the groundwork commit. Four fixes, no new surface. preflight() no longer walks content/storage. contentBytesOnDisk() is a synchronous readdirSync/statSync recursion over tens of thousands of files on a populated device, and Node is single-threaded, so serving it from a GET blocked every other endpoint — the kiwix/maps/books jobs included. Summing bytesAvailable from SQLite gives the same number. The function stays exported for the progress-without-the-API case, with a comment saying not to call it from a handler. checkReadiness() no longer mutates. It called ensureContentOrigin(), so GET /kolibri/ready and /kolibri/preflight created a NetworkLocation. Split into hasContentOrigin() (read) and ensureContentOrigin() (write); the runner still guarantees the row before enqueuing, which is the right place. contentOrigin in the readiness payload is now boolean|null instead of the write-result string, and it is no longer a blocker: its absence does not stop startup. Long imports survive session expiry. loginForContent() ran once and the poll could last hours; when the Django session died, apiJson started failing and we aborted 15 minutes later with "el polling falló de forma sostenida", losing a job Kolibri was likely finishing. apiJson now throws KolibriApiError carrying the status, and the poll re-authenticates on 401/403 (403 is what Django returns once the CSRF token is stale too), bounded to MAX_REAUTH so a revoked credential cannot loop. The session moved into a holder so the poll can replace it. listInstalledChannels() is one pass instead of four correlated subqueries per channel. A CTE reduces content_file x content_contentnode to DISTINCT (channel_id, local_file_id) pairs, then aggregates once. Verified equivalent to the old query across 11 SQLite fixtures — including a LocalFile hanging off two nodes of the same channel, which is the double-count the DISTINCT exists to prevent, plus shared-across-channels, orphan rows, NULL available and NULL file_size, and a random 1200-row fixture. 2.5x faster at 4 channels / 20k nodes / 45k file rows. Also: allThumbnails now defaults to true only for partial selections, since a full channel already brings its topic thumbnails; and failed/canceled Kolibri jobs get cleared too, not just successful ones. Verified: tsc --noEmit clean, npm test 48/48 (41 + 7 new for matchesOrigin and KolibriApiError). Runner re-exercised against the mock, plus a new scenario that revokes the session mid-poll: it re-logs in and completes instead of stalling.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seed Kolibri content from the app over REST, so a device does not reach the user with an empty Kolibri. Adds a fourth JobType ('kolibri') to the durable job engine (ADFA-4838). Groundwork only: runner, credential store and diagnostics; the Android wishlist/UI is a follow-up.
New in static/dashboard/sockets/: credentials.ts, kolibri.session.ts, kolibri.map.ts, kolibri.exec.ts, kolibri.query.ts, kolibri.session.test.ts. Plus tools/kolibri-preflight.sh.
Routes before the generic /:type/* handlers: /kolibri/{ready,preflight,channels,catalog,resolve,tree,estimate,delete} and /credentials/:service. 'kolibri' added to JobType and VALID_TYPES.
Verified: tsc --noEmit clean, npm test 41/41. Not device-tested — acceptance criteria on ADFA-4949.
Follow-ups (out of scope): the Android side and the Courses screen, the catalog asset generator, migrating books.* onto the shared credential store.