feat(lucebox): docker stack + CLI + bench/profile + harness + luce-bench in-tree

[easel]

This PR turns Lucebox into a one-command local inference deployment and ships the two tools that operate it: lucebox (the host CLI that runs and tunes the server) and luce-bench (the benchmark + grading framework that measures it). All three ship together so a fresh box goes from nothing to a tuned, benchmarked server with a single install.

The three pieces, what each is, and how to use it:

---

1. Docker — the server image

A CUDA 12.8 image (ghcr.io/luce-org/lucebox-hub:cuda12) that builds the dflash server and bundles server/, lucebox/, harness/, and luce-bench/. The entrypoint dispatches serve (default), benchmark, any lucebox subcommand, or shell. An in-container autotune fallback picks VRAM-tiered defaults and resolves the draft GGUF by target architecture (gemma4 → gemma drafter, qwen3.6 → dflash-draft-3.6).

Use it directly:

docker run --rm --gpus all -p 8080:8080 \
  -v ~/.local/share/lucebox/models:/opt/lucebox-hub/server/models \
  ghcr.io/luce-org/lucebox-hub:cuda12
# OpenAI + Anthropic-compatible API on :8080
curl -s http://localhost:8080/v1/models

Image tags: :cuda12, :vX.Y.Z-cuda12, :X.Y-cuda12, :sha-<short>-cuda12. Built and pushed by .github/workflows/docker.yml; docker-bake.hcl has a cuda13 slot ready.

2. lucebox — the host CLI

lucebox.sh is the host-side wrapper (deps: docker + nvidia-smi only). It probes the host, writes a tuned config.toml, runs the container as a user-systemd service, and delegates provisioning/workloads to the in-container Python CLI (models, autotune, profile, smoke, config, the client drivers).

Stand a server up:

lucebox check            # driver / docker / NVIDIA Container Toolkit / VRAM / systemd / WSL2 probe
lucebox pull             # docker pull the cuda12 image
lucebox models download  # pull target + DFlash draft GGUFs  (verbs: list, download)
lucebox autotune         # VRAM-tiered DFLASH_* defaults → ~/.lucebox/config.toml  (autotune --sweep picks a winner empirically)
lucebox install          # install the user-systemd unit
lucebox start            # bring it up   (enable = start at every login)
lucebox status           # unit state + the server's startup banner
lucebox logs             # follow the journal
lucebox smoke            # props/tools/http/1-token health check

Tune it to the GPU:

lucebox profile          # level1/2/3 sweep over DFLASH_MAX_CTX × DFLASH_BUDGET ×
                         # {KV type, pFlash mode, lazy-draft, prefix-cache slots},
                         # gated on capability + ds4-eval/agentic validation before
                         # the winner merges into config.toml

The running config is observable at GET /props (schema 4), which now reports a host block — kernel, OS, WSL vs native, driver, CPU, RAM, GPU — so a server self-describes its real config and host.

3. luce-bench — the benchmark + grading framework

In-tree workspace member (luce-bench/, 0.2.7.dev0) that scores any OpenAI/Anthropic-compatible endpoint and writes versioned, comparable result files. Areas: smoke, ds4-eval (92 reasoning items), gsm8k, truthfulqa-mc1, hellaswag, code, longctx, agent, agent_recorded, forge. Every result stamps a per-area grader_version and a host block (from /props.host, or a clearly-marked client-side fallback for servers without /props).

Run it:

uvx --from 'git+https://github.com/easel/lucebox-hub@feat/lucebox-docker#subdirectory=luce-bench' \
  luce-bench --base-url http://localhost:8080 --model dflash --areas all --no-think

Thinking control is portable. Each request carries three control shapes (chat_template_kwargs.enable_thinking, Anthropic thinking:{type}, reasoning_effort). For servers that ignore the API flags (e.g. OpenRouter), --prompt-thinking-control {auto,on,off} (default auto) injects the model family's in-band token (/no_think, /think); auto fires only when /props shows no server-side enforcement. A post-run verifier records thinking_control_honored so a nothink run that secretly reasoned is flagged, not silently mislabeled.

Comparing results: runs from one grader version are comparable as written. For older snapshots graded by a different version, luce-bench regrade <dirs> re-scores stored outputs at the current pinned grader and refuses to place mismatched-version (or mismatched-host) runs in the same row. report / snapshot / submit-baseline round out the reporting surface.

---

Also in this PR

• harness/ — drives real clients (claude_code, codex, opencode, hermes, pi, openclaw) against a running server; lucebox profile delegates bench runs here.
• Model-card sidecars — share/model_cards/{qwen3.6-27b,gemma-4-26b-a4b-it,gemma-4-31b-it,laguna-xs.2}.json + _schema.json, so the server resolves sampler defaults, thinking budgets, and the force-close hint per model.
• Workspace — pyproject.toml declares all members (server, lucebox, luce-bench, harness, optimizations/{megakernel,pflash}); [tool.uv.sources] luce-bench = { workspace = true } replaces the prior git-tag pin. release-luce-bench.yml publishes to PyPI on luce-bench-v* tags.
• Docs — README quick start + hardware/env reference; server/docs/ benchmark-snapshot spec and experiment write-ups.
• Removes the obsolete server/scripts/bench_*.py (their work now lives in luce-bench).

Out of scope / follow-ups

• Gemma 4 31B backend wiring beyond what its model card ships (validated empirically @ 24 GB, AR-only).
• gemma4 MoE expert split.
• Multi-Token Prediction (upstream, draft).

Validation

• uv sync clean on the workspace; luce-bench test suite passes.
• Full --areas all sweeps run end-to-end against bragi (RTX 5090 Laptop), sindri (RTX 3090 Ti), vidar (M2 Ultra / MLX), and OpenRouter, think and nothink, all on one grader version.
• /props.host confirmed populated on lucebox servers (bragi + sindri report WSL2); OpenRouter nothink confirmed honored via client-side /no_think injection.

[easel]

Some commands to test this... copied from the readme.

Install the lucebox wrapper:

curl -fsSL https://raw.githubusercontent.com/easel/lucebox-hub/feat/lucebox-docker/lucebox.sh \
       -o ~/.local/bin/lucebox.sh && chmod +x ~/.local/bin/lucebox.sh

Run lucebox using the docker image

# Override the container image to the temporary build:
export LUCEBOX_IMAGE=ghcr.io/easel/lucebox-hub

# Check your machine for lucebox compatibility
lucebox check

# Start the lucebox server
lucebox serve

Run benchmarks against a local server:

uvx --refresh --from "git+https://github.com/easel/lucebox-hub@feat/lucebox-docker#subdirectory=luce-bench" lucebench --url http://localhost:1236

Run benchmarks against open router

uvx --refresh --from "git+https://github.com/easel/lucebox-hub@feat/lucebox-docker#subdirectory=luce-bench" lucebench --base-url https://openrouter.ai/api --model qwen/qwen3.6-27b --auth-env OPENROUTER_API_KEY

[cubic-dev-ai]

17 issues found across 188 files

<sub>Note: This PR contains a large number of files. cubic only reviews up to 100 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

3 issues found across 12 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="luce-bench/src/lucebench/cli.py">

<violation number="1" location="luce-bench/src/lucebench/cli.py:692">
P2: `--prompt-thinking-control=on` can be silently disabled by the new card-capability gate, so explicit user override no longer works on unresolved/non-capable cards.</violation>
</file>

<file name="luce-bench/src/lucebench/runner.py">

<violation number="1" location="luce-bench/src/lucebench/runner.py:682">
P2: Continuation request can unintentionally re-enable thinking because `extra_body` overwrites forced disable flags.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: --prompt-thinking-control=on can be silently disabled by the new card-capability gate, so explicit user override no longer works on unresolved/non-capable cards.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At luce-bench/src/lucebench/cli.py, line 692:

<comment>`--prompt-thinking-control=on` can be silently disabled by the new card-capability gate, so explicit user override no longer works on unresolved/non-capable cards.</comment>

<file context>
@@ -665,6 +685,13 @@ def _run_standard_area_to_dir(
+    # tokens for a thinking-capable card; otherwise force the flag off so
+    # neither the card nor the family-map fallback injects.
+    effective_thinking_control = (
+        prompt_thinking_control if card_is_thinking_capable(model_card) else "off"
+    )
+
</file context>

[cubic-dev-ai]

P2: Continuation request can unintentionally re-enable thinking because extra_body overwrites forced disable flags.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At luce-bench/src/lucebench/runner.py, line 682:

<comment>Continuation request can unintentionally re-enable thinking because `extra_body` overwrites forced disable flags.</comment>

<file context>
@@ -384,7 +618,88 @@ def run_case(
+    if top_k is not None and top_k > 0:
+        body["top_k"] = int(top_k)
+    if extra_body:
+        body.update(extra_body)
+
+    headers = {"Content-Type": "application/json"}
</file context>

Suggested change

	body.update(extra_body)
	body.update(
	{
	k: v
	for k, v in extra_body.items()
	if k
	not in {
	"messages",
	"max_tokens",
	"stream",
	"chat_template_kwargs",
	"thinking",
	"reasoning_effort",
	}
	}
	)

[easel]

Superseded by the 8-PR split (#334, #335, #336, #337, #338, #339, #340, #341). All core feature content is captured across those 8 PRs; the umbrella diff additionally contained docs/experiments and dev-script baggage that was intentionally dropped during the polish pass. Closing in favor of reviewing the splits.

🤖 Generated with Claude Code