Browser RAG UI, GPU/CPU tuning, tests, CI, and LAN serving - #1
Merged
Conversation
The chat server on :8081 answers from model weights alone; retrieval only happened inside rag-ask.py, so anything else pointed at the phone -- curl, a browser, any OpenAI client -- silently got answers that had never seen the notes. This moves retrieval behind an HTTP surface of its own. rag-web.py serves a chat page and an OpenAI-compatible /v1/chat/completions on :8083, both applying retrieval first, and holds the index in memory instead of re-reading it per question. It binds loopback by default and is reached over adb forward, so it is never exposed to the campus network; a bearer token is required if it is ever bound to a routable address. ragcore.py now holds the chunking, scoring and prompt assembly that rag-index, rag-ask and rag-web all share, so the three cannot drift apart. Vectors are kept as pre-normalized float32 arrays, which cuts the resident index from about 38 MB to 4.8 MB -- worth doing on a phone already near its RAM ceiling -- and turns cosine similarity into a plain dot product. Retrieved context is now bounded by a character budget rather than a fixed chunk count. Time to first token is prompt-eval bound at roughly 20 tokens per second, so prompt length is the wait; a fixed top_k let that wait vary with whatever the chunker happened to produce. Tests cover the whole path against stand-in embedding and chat servers, so they run on a laptop and in CI with no phone and no model weights.
Prompt processing, not generation, is what you wait on before an answer starts, and it was running at 17.9 tokens per second. Two things were leaving the phone idle. The GPU was never used. The handoff recorded "no GPU backend" as a property of the phone, but it is a property of the installed build: the Adreno 830 is reachable from unrooted Termux through Mesa's turnip Vulkan driver, and llama.cpp ships a matching backend as a Termux package. Prompt eval goes to 70.2 tokens per second, so a 605-token RAG prompt starts answering after 8.9 seconds instead of 36. The vendor OpenCL path is a dead end and worth recording as one: the Android linker refuses an absolute path into /vendor/lib64 from an app namespace, and the bare soname resolves to Termux's own ICD loader, so it can never find a platform. Vulkan sidesteps this because turnip runs entirely in userspace. The CPU is 6 performance cores plus 2 prime cores, and spreading 8 threads over all of them is slower than pinning 6 threads to the matched ones -- generation goes 9.1 to 12.2 tokens per second. The threads landing on the prime cores finish early and idle, and the two cores left free absorb the OS and the GPU driver. The chat server now takes cpu0-5 and the embedding server cpu6-7, which also ends the contention of the two asking for 12 threads on an 8-core phone. GGML_BACKEND_PATH is left deliberately unset, with a comment saying why: pointing it at a single .so restricts ggml to that one backend and silently drops the GPU, which cost an hour of chasing during this work. --prio is dropped: raising thread priority needs root and only logged "Operation not permitted" once per thread. bench/RESULTS.md records the full matrix and bench/probe.py reproduces the headline numbers against a running server.
…sured ones The README claimed 15 tokens/sec generation and 33 prompt, both estimates that turned out to be wrong in opposite directions once measured. It now carries the numbers from bench/RESULTS.md and says which half runs on the GPU and why. rag/README gains the part that was easiest to get wrong: :8081 answers from the model's weights alone and :8083 is the one that applies retrieval, so anything pointed at the chat server directly silently gets answers that never saw the notes.
…neral audience Two changes that landed together. Serving: the browser UI and OpenAI-compatible endpoint were loopback-only, reached over adb forward. They now bind LAN-wide at boot so other devices on the same network can use them without a cable, which means every request except /health must carry the bearer token. The page reads whether auth is required from /health, prompts for the token when it is, remembers it per browser, sends it on every request, and re-prompts on a 401. This is a deliberate exposure: the endpoint reads out of a private notes corpus and the token travels in clear text over HTTP, so RAG_WEB_HOST=127.0.0.1 returns it to loopback-only on an untrusted network. Docs: the public documentation no longer addresses a specific person or narrates their plans, and no longer prints a concrete LAN address. Replaced the real IP with a <phone-ip> placeholder, generalised the network context, corrected the key-generation command to one that exists on Termux, and removed the claim that a subnet size implies who can reach the service. Deployment-specific and planning notes are kept out of the repository entirely.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
main removed GUIDE.md and updated README; this branch had edited GUIDE and the README/rag-README. Resolved by dropping GUIDE.md as main intended, removing the now-dangling links to it, and keeping the neutral, non-personal wording on both sides of the rag-README conflict.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Browser UI + OpenAI-compatible RAG endpoint (
rag/bin/rag-web.py, port 8083).The chat server on :8081 answers from model weights alone; this applies retrieval
first and serves both a chat page and a
/v1/chat/completionsendpoint that adds asourcesfield. Bound LAN-wide behind the bearer token so other devices can use it;set
RAG_WEB_HOST=127.0.0.1for loopback-only.Shared core + tests (
rag/bin/ragcore.py,tests/). Chunking, scoring, promptassembly extracted so the CLI and web paths agree. 59 tests run on a laptop / in CI
against stand-in model servers — no phone, no GGUF weights. GitHub Actions runs pytest
(py3.11–3.13), ruff, and shellcheck.
Performance — prompt eval 18 → 70 tok/s, generation 10 → 12 tok/s. Prompt
processing now runs on the Adreno GPU via Mesa's turnip Vulkan driver; generation runs
on the six pinned performance cores. A 605-token retrieval prompt starts answering in
8.9 s instead of 36 s. Full matrix and reproduction in
bench/RESULTS.md.Docs rewritten for a general audience, with measured numbers replacing estimates
and no concrete local network addresses.
Verification
python3 -m pytest tests/ -q→ 59 passedruff check .→ cleanLAN auth verified over Wi-Fi.