Hidden Solver

A stealthy screen-capture QA tool powered by LLMs. It captures a selected screen region, extracts the question/options, optionally augments with a local knowledge base, and returns a compact answer.

中文说明请见 README.zh-CN.md.

Table of Contents

• Features
• Requirements
• Quick Start
• Hotkeys
• Runtime Pipeline
• Glossary
• Configuration Loading
• Configuration Reference
• RAG Document Formats & Cleaning
• Example Configurations
• Advanced Usage
• Best Practices
• Performance & Cost
• Security & Privacy
• Troubleshooting
• FAQ
• Project Structure
• License

Features

• Screen region capture (Wayland and X11)
• Fixed and dynamic selection modes
• Visual or silent overlay
• OpenAI-compatible local backend or Cherry Studio API backend
• Optional local KB (RAG via LlamaIndex)
• Separate text-solve mode (vision for extraction, text-only for final answer)
• Debug logs for inspection

Requirements

• Python >= 3.10
• Wayland: grim is required for screenshots
• X11: no extra system tool required

Quick Start

1. Install dependencies:

   cd hidden_solver
   uv sync

2. Create your .env:

   cp .env.example .env

3. Fill in API keys and model settings in .env.

4. Run the app:

   cd hidden_solver
   uv run solve

Wayland note: keyboard hooks and screenshots may require elevated privileges:

sudo uv run solve

Or add your user to the input group (reboot required):

sudo usermod -aG input $USER

Hotkeys

Default hotkeys (customizable via .env):

• HOTKEY_SETUP (<home>): enter setup mode to select regions
• HOTKEY_TRIGGER_A (<page_up>): solve with prompt A (single-choice)
• HOTKEY_TRIGGER_B (<page_down>): solve with prompt B (multi-choice)
• HOTKEY_TOGGLE_ANSWER (<end>): show/hide answer overlay
• HOTKEY_QUIT (<ctrl>+<alt>+]): quit application

Runtime Pipeline

1. Setup: select the question region (and answer overlay region for fixed mode).
2. Capture: take a screenshot of the question region.
3. Extract (optional):
   • If SOLVE_TEXT_SEPARATE=true or KB_QUERY_MODE=llm, the app extracts text using the vision backend and EXTRACT_PROMPT.
4. RAG (optional):
   • If KB_ENABLED=true, the app retrieves related snippets (RAG) or injects the full KB.
5. Solve:
   • If SOLVE_TEXT_SEPARATE=true, solve with a text-only backend using extracted text.
   • Otherwise solve with the vision backend using the screenshot.
6. Post-process: filter output to allowed option letters.

Glossary

• Vision backend: the model used for screenshot understanding (local OpenAI-compatible or Cherry).
• Text backend: the model used for final solving in separate text mode.
• Extraction: turning screenshot into question/options text via EXTRACT_PROMPT.
• RAG: retrieving snippets from a local KB using embeddings.
• Full injection: attaching the entire KB to the prompt (no retrieval).

Configuration Loading

• The app loads .env from the current working directory.
• Shell environment variables override .env.
• Recommended: run from hidden_solver so .env is discovered.
• Boolean values: true/false, 1/0, yes/no, on/off are accepted.

Configuration Reference

Defaults shown below come from hidden_solver/src/hidden_solver/config.py.

Core Controls

• DEBUG (default: false): print debug logs to terminal.
• VISION_BACKEND (default: local): local or cherry.
• SOLVE_TEXT_SEPARATE (default: false): enable text-only solve after extraction.
• SOLVE_TEXT_BACKEND (default: local): local or cherry (empty => fallback to VISION_BACKEND).

Local OpenAI-Compatible (Vision/Extraction)

• OPENAI_API_KEY (default: empty): API key.
• OPENAI_BASE_URL (default: https://api.siliconflow.cn/v1): OpenAI-compatible base URL.
• OPENAI_MODEL (default: Qwen/Qwen3-VL-235B-A22B-Thinking): vision model.
• OPENAI_TEMPERATURE (default: 0.0)
• OPENAI_TOP_P (default: 1.0)
• OPENAI_MAX_TOKENS (default: 3000)
• OPENAI_TIMEOUT (default: 60)
• OPENAI_THINKING_TYPE (default: empty): thinking switch for Volcengine Ark / Doubao and compatible OpenAI-style backends. Supports enabled | disabled | auto; empty means the field is not sent.

Cherry Studio API (Vision/Extraction)

• CHERRY_BASE_URL (default: http://127.0.0.1:23333)
• CHERRY_API_KEY (default: empty)
• CHERRY_AUTH_MODE (default: bearer): bearer | x-api-key | none
• CHERRY_AUTH_HEADER (default: empty): custom auth header name (overrides CHERRY_AUTH_MODE)
• CHERRY_MODEL (default: empty): model id as model field (copy from /v1/models)
• CHERRY_EXTRA_HEADERS (default: empty): JSON or Key=Value;Key2=Value2
• CHERRY_TEMPERATURE (default: 0.0)
• CHERRY_TOP_P (default: 1.0)
• CHERRY_MAX_TOKENS (default: 3000)
• CHERRY_TIMEOUT (default: 60)

Text Solve (Separate Mode)

OpenAI-compatible:

• SOLVE_TEXT_OPENAI_API_KEY (default: empty)
• SOLVE_TEXT_OPENAI_BASE_URL (default: empty)
• SOLVE_TEXT_OPENAI_MODEL (default: empty)
• SOLVE_TEXT_OPENAI_TEMPERATURE (default: empty, fallback to OPENAI_TEMPERATURE)
• SOLVE_TEXT_OPENAI_TOP_P (default: empty, fallback to OPENAI_TOP_P)
• SOLVE_TEXT_OPENAI_MAX_TOKENS (default: empty, fallback to OPENAI_MAX_TOKENS)
• SOLVE_TEXT_OPENAI_TIMEOUT (default: empty, fallback to OPENAI_TIMEOUT)
• SOLVE_TEXT_OPENAI_THINKING_TYPE (default: empty, fallback to OPENAI_THINKING_TYPE)

Cherry:

• SOLVE_TEXT_CHERRY_BASE_URL (default: empty)
• SOLVE_TEXT_CHERRY_API_KEY (default: empty)
• SOLVE_TEXT_CHERRY_AUTH_MODE (default: empty)
• SOLVE_TEXT_CHERRY_AUTH_HEADER (default: empty)
• SOLVE_TEXT_CHERRY_MODEL (default: empty)
• SOLVE_TEXT_CHERRY_EXTRA_HEADERS (default: empty)
• SOLVE_TEXT_CHERRY_TEMPERATURE (default: 0.0)
• SOLVE_TEXT_CHERRY_TOP_P (default: 1.0)
• SOLVE_TEXT_CHERRY_MAX_TOKENS (default: 3000)
• SOLVE_TEXT_CHERRY_TIMEOUT (default: 60)

Prompts

• PROMPT_A: single-choice prompt.
• PROMPT_B: multi-choice prompt.
• EXTRACT_PROMPT: extraction prompt (used when text extraction is needed).

Output Filter

• ANSWER_ALLOWED_OPTIONS (default: ABCDEF)
• ANSWER_NO_MATCH (default: ?)

App Modes

• APP_MODE (default: fixed): fixed | dynamic
• VISUAL_MODE (default: visible): visible | silent
• AUTO_POLL_ENABLED (default: false)
• AUTO_POLL_INTERVAL (default: 60)
• QUESTION_ARCHIVE_ENABLED (default: false): save the exact question-region image sent to the LLM.
• QUESTION_ARCHIVE_DIR (default: question_archive): archive directory; relative paths are resolved from the current working directory.

Knowledge Base (RAG)

• KB_ENABLED (default: false)
• KB_REBUILD_ON_START (default: false)
• KB_MODE (default: rag): rag | full
• KB_DOCS_DIR (default: kb)
• KB_INDEX_DIR (default: .kb_index)
• KB_TOP_K (default: 4)
• KB_MAX_CHARS (default: 4000)
• KB_FULL_MAX_CHARS (default: KB_MAX_CHARS)
• KB_QUERY_MODE (default: llm): llm | prompt | none
• KB_EMBED_BASE_URL (default: empty)
• KB_EMBED_API_KEY (default: empty)
• KB_EMBED_MODEL (default: empty)
• KB_EMBED_TIMEOUT (default: empty, fallback to OPENAI_TIMEOUT)
• KB_EMBED_REBUILD_TIMEOUT (default: empty, used when KB_REBUILD_ON_START=true)

RAG rules:

• KB_MODE=full does not use embeddings.
• KB_MODE=rag requires KB_EMBED_*.
• KB_REBUILD_ON_START=true deletes KB_INDEX_DIR once on the first RAG call.
• When rebuilding, the embed timeout prefers KB_EMBED_REBUILD_TIMEOUT if set.

RAG Document Formats & Cleaning

Supported formats (via LlamaIndex SimpleDirectoryReader):

• Plain text (.txt) and Markdown (.md) are the most reliable.
• PDFs often work, but scanned PDFs may produce empty text. Use OCR before indexing.
• For other formats (docx/html), convert to .txt or .md if parsing is unreliable.

Cleaning strategies (recommended before indexing):

• Remove headers, footers, page numbers, and navigation noise.
• Normalize whitespace and remove duplicates.
• Split large documents by topic (1k-5k characters per file is a good starting point).
• Convert tables into bullet lists or key-value lines for better retrieval.
• Use meaningful file names (they appear in retrieval snippets as the source).
• Keep a stable directory structure to avoid accidental re-index churn.

Example KB layout:

kb/
  math/
    algebra_basics.md
  policy/
    onboarding.txt

Example Configurations

1) Local Vision Only (no RAG)

VISION_BACKEND=local
OPENAI_BASE_URL=https://api.siliconflow.cn/v1
OPENAI_MODEL=Qwen/Qwen3-VL-235B-A22B-Instruct
OPENAI_API_KEY=sk-xxxx
SOLVE_TEXT_SEPARATE=false
KB_ENABLED=false

2) Cherry Vision + Local Text Solve + Local RAG

VISION_BACKEND=cherry
CHERRY_BASE_URL=http://127.0.0.1:23333
CHERRY_API_KEY=cs-sk-xxxx
CHERRY_AUTH_MODE=bearer
CHERRY_MODEL=my-openai:gpt-4

SOLVE_TEXT_SEPARATE=true
SOLVE_TEXT_BACKEND=local
SOLVE_TEXT_OPENAI_BASE_URL=https://api.siliconflow.cn/v1
SOLVE_TEXT_OPENAI_MODEL=deepseek-ai/DeepSeek-V3.1-Terminus
SOLVE_TEXT_OPENAI_API_KEY=sk-xxxx
SOLVE_TEXT_OPENAI_MAX_TOKENS=30

KB_ENABLED=true
KB_MODE=rag
KB_QUERY_MODE=llm
KB_EMBED_BASE_URL=https://api.siliconflow.cn/v1
KB_EMBED_MODEL=Qwen/Qwen3-Embedding-8B
KB_EMBED_API_KEY=sk-xxxx

3) Small KB Full Injection (no retrieval)

KB_ENABLED=true
KB_MODE=full
KB_FULL_MAX_CHARS=20000

4) Vision + Prompt-based Retrieval

KB_ENABLED=true
KB_MODE=rag
KB_QUERY_MODE=prompt

5) Minimal Text Solve Only (no vision for final answer)

VISION_BACKEND=local
SOLVE_TEXT_SEPARATE=true
SOLVE_TEXT_BACKEND=local
SOLVE_TEXT_OPENAI_MODEL=deepseek-ai/DeepSeek-V3.1-Terminus
SOLVE_TEXT_OPENAI_MAX_TOKENS=20

Advanced Usage

• Vision-only with strict output: use low temperature and low max tokens.
• Text solve for cheaper inference: keep extraction on vision model, solve with a smaller text model.
• RAG for domain data: store PDFs/markdown in KB_DOCS_DIR, set KB_MODE=rag.
• Full injection for tiny KB: KB_MODE=full gives stable prompt prefixes for caching.
• Use prompt-based retrieval: KB_QUERY_MODE=prompt if extraction is unreliable.

Best Practices

• Keep extraction deterministic: EXTRACT_PROMPT + temperature=0.
• Limit output length for text solve: SOLVE_TEXT_OPENAI_MAX_TOKENS=10~50.
• Use ANSWER_ALLOWED_OPTIONS to reduce noise.
• For small KBs, KB_MODE=full improves caching and avoids embedding costs.
• For large KBs, keep KB_TOP_K low and tune KB_MAX_CHARS.
• Prefer running from hidden_solver to ensure .env is loaded.
• Ignore .kb_index in git (index is regenerable).

Performance & Cost

• Use smaller capture regions to reduce token usage and latency.
• Keep OPENAI_MAX_TOKENS small when you only need a letter.
• Prefer SOLVE_TEXT_SEPARATE=true with a lightweight text model for cheaper inference.
• Reduce RAG overhead with smaller KB_TOP_K and KB_MAX_CHARS.
• Disable DEBUG in normal use to avoid I/O overhead.

Security & Privacy

• Do not commit .env or real API keys. Use .env.example for sharing.
• Rotate keys if they were exposed.
• Be cautious when sending sensitive content to third-party APIs.

Troubleshooting

• RAG says embedding not configured: ensure KB_EMBED_* is set and reachable.
• RAG still uses old docs: delete KB_INDEX_DIR or set KB_REBUILD_ON_START=true.
• Wayland screenshots are black: check grim and permissions.
• Hotkeys not working: try running with sudo or add user to input group.
• Cherry API 401/403: verify CHERRY_API_KEY and auth headers.
• Multi-monitor offset: ensure primary monitor scaling is correct.
• Embedding request fails: check KB_EMBED_BASE_URL and model name support.

FAQ

Q: Do I need to re-embed the entire KB on every query? A: No. The KB is embedded once and stored in KB_INDEX_DIR. Later queries only embed the query text.

Q: What is the difference between KB_QUERY_MODE=llm and prompt? A: llm uses extracted text from the screenshot as the retrieval query. prompt uses the raw prompt string.

Q: Why is the answer only a letter? A: Output is filtered by ANSWER_ALLOWED_OPTIONS to keep results compact and reliable.

Q: Can I use Cherry for vision and local embeddings for RAG? A: Yes. RAG uses KB_EMBED_* settings and is independent of VISION_BACKEND.

Q: How do I reset the KB index? A: Delete KB_INDEX_DIR or set KB_REBUILD_ON_START=true before the next run.

Project Structure

• src/hidden_solver/
  • core/: app controller and worker threads
  • services/: LLM, KB, screenshot, hotkey
  • ui/: overlay widgets
  • config.py: configuration loader
  • main.py: entry point

License

MIT