diff --git a/README.md b/README.md index d96b49592..bf328444c 100644 --- a/README.md +++ b/README.md @@ -105,7 +105,7 @@ Reference target: **RTX 3090 (Ampere sm_86)** — all headline numbers. Other NV [`harness/`](harness/) contains RTX 3090 client launchers and regression tests for Lucebox server compatibility. Run Lucebox inside Claude Code, Codex, -OpenCode, Hermes, Pi, OpenClaw, or Open WebUI, or check if a server change +OpenCode, Hermes, OMP, Pi, OpenClaw, or Open WebUI, or check if a server change still works with those clients. @@ -123,6 +123,7 @@ still works with those clients. | Codex | [`run_codex.sh`](harness/clients/run_codex.sh) | | OpenCode | [`run_opencode.sh`](harness/clients/run_opencode.sh) | | Hermes | [`run_hermes.sh`](harness/clients/run_hermes.sh) | +| [OMP](https://omp.sh/) | [`run_omp.sh`](harness/clients/run_omp.sh) | | Pi | [`run_pi.sh`](harness/clients/run_pi.sh) | | OpenClaw | [`run_openclaw.sh`](harness/clients/run_openclaw.sh) | | Open WebUI | [`run_openwebui.sh`](harness/clients/run_openwebui.sh) | @@ -149,7 +150,7 @@ Launcher scripts install missing real-client CLIs automatically under `.harness-work/`. To preinstall them yourself: ```bash -python3 harness/client_test_runner.py install --clients codex,hermes,openwebui +python3 harness/client_test_runner.py install --clients codex,hermes,omp,openwebui ``` For direct TPS/TTFT numbers against a running server: diff --git a/harness/README.md b/harness/README.md index 5a1adb86c..01a050daa 100644 --- a/harness/README.md +++ b/harness/README.md @@ -10,7 +10,7 @@

Client launchers and regression tests for Lucebox server compatibility.
- Run Lucebox from Claude Code, Codex, OpenCode, Hermes, Pi, OpenClaw, or Open WebUI.
+ Run Lucebox from Claude Code, Codex, OpenCode, Hermes, OMP, Pi, OpenClaw, or Open WebUI.
RTX 3090 / 24 GB defaults are included for each client.

@@ -31,6 +31,7 @@ cd lucebox-hub harness/clients/run_codex.sh harness/clients/run_claude_code.sh harness/clients/run_opencode.sh +harness/clients/run_omp.sh ``` The launchers default to the current repo, install/use client packages under @@ -93,7 +94,7 @@ If you already have `dflash_server` running, use `probe`: ```bash python3 harness/client_test_runner.py probe \ --url http://127.0.0.1:8000 \ - --clients claude_code,codex,opencode,openwebui,pi \ + --clients claude_code,codex,opencode,omp,openwebui,pi \ --json-out /tmp/lucebox_harness_probe.json ``` @@ -103,7 +104,7 @@ client packages. Without it, the HTTP protocol probes still run. To preinstall real-client CLIs yourself: ```bash -python3 harness/client_test_runner.py install --clients codex,hermes,openwebui +python3 harness/client_test_runner.py install --clients codex,hermes,omp,openwebui ``` For a GPU sweep, let the runner start Lucebox for each profile: diff --git a/harness/benchmarks/README.md b/harness/benchmarks/README.md index 80b90c68e..68b7d0cb5 100644 --- a/harness/benchmarks/README.md +++ b/harness/benchmarks/README.md @@ -6,7 +6,7 @@ deterministic prompts. Use this when you want to know whether a server change affects output quality or decode speed. Use `harness/clients/` when you want to know whether Codex, -OpenCode, Open WebUI, Pi, and the other clients still work. +OpenCode, OMP, Open WebUI, Pi, and the other clients still work. ## Bench suites (HumanEval, GSM8K, Math500, Agent) diff --git a/harness/client_test_runner.py b/harness/client_test_runner.py index 7763ae909..f5fae318f 100755 --- a/harness/client_test_runner.py +++ b/harness/client_test_runner.py @@ -101,6 +101,14 @@ class ClientSpec: binary="opencode", protocol="openai_chat", ), + "omp": ClientSpec( + name="omp", + install="omp", + package="https://raw.githubusercontent.com/can1357/oh-my-pi/main/scripts/install.sh", + binary="omp", + protocol="responses", + notes="OMP uses a custom openai-responses provider pointed at Lucebox.", + ), "pi": ClientSpec( name="pi", install="npm", @@ -281,6 +289,8 @@ def client_bin(work_dir: Path, spec: ClientSpec) -> Path: return pip_venv(work_dir, spec.name) / "bin" / spec.binary if spec.install == "hermes": return hermes_home(work_dir) / ".local" / "bin" / spec.binary + if spec.install == "omp": + return work_dir / "clients" / spec.name / "bin" / spec.binary raise HarnessError(f"unknown installer {spec.install}") @@ -339,6 +349,26 @@ def install_client(work_dir: Path, spec: ClientSpec) -> dict[str, Any]: timeout=1800, stream=True, ) + elif spec.install == "omp": + # Install the latest release, matching the repo's unpinned npm/pip + # client policy. The --version smoke below records the exact binary + # in the install report for reproducibility. + root = work_dir / "clients" / spec.name + bin_dir = root / "bin" + root.mkdir(parents=True, exist_ok=True) + bin_dir.mkdir(parents=True, exist_ok=True) + script_path = root / "install.sh" + with urllib.request.urlopen(spec.package, timeout=60) as response: + with script_path.open("wb") as script_file: + shutil.copyfileobj(response, script_file) + env = os.environ.copy() + env["PI_INSTALL_DIR"] = str(bin_dir) + result = run_cmd( + ["sh", str(script_path), "--binary"], + env=env, + timeout=900, + stream=True, + ) else: raise HarnessError(f"unknown installer {spec.install}") diff --git a/harness/clients/README.md b/harness/clients/README.md index 73004b261..ccaafefa6 100644 --- a/harness/clients/README.md +++ b/harness/clients/README.md @@ -20,7 +20,7 @@ If a client CLI is missing, the launcher installs it automatically. Set To preinstall real-client CLIs yourself: ```bash -python3 harness/client_test_runner.py install --clients codex,hermes,openwebui +python3 harness/client_test_runner.py install --clients codex,hermes,omp,openwebui ``` The launcher will start `server/build/dflash_server` by default, or the path in @@ -52,7 +52,8 @@ harness/clients/run_codex.sh The C++ server is expected to handle the same client protocol shapes covered by these launchers and probes: OpenAI Chat Completions, streaming chunks, tool -metadata, OpenAI Responses for Codex, Anthropic Messages for Claude Code, and +metadata, OpenAI Responses for Codex and OMP, Anthropic Messages for Claude +Code, and Open WebUI model metadata. ## Defaults @@ -66,6 +67,7 @@ The defaults below are the current RTX 3090 starting points for | Codex | `run_codex.sh` | `MAX_CTX=32768 BUDGET=22 VERIFY_MODE=ddtree EXTRA_SERVER_ARGS=--lazy-draft` | | OpenCode | `run_opencode.sh` | `MAX_CTX=86016 BUDGET=22 VERIFY_MODE=ddtree EXTRA_SERVER_ARGS=--lazy-draft` | | Hermes Agent | `run_hermes.sh` | `MAX_CTX=98304 BUDGET=22 VERIFY_MODE=ddtree EXTRA_SERVER_ARGS=--lazy-draft` | +| OMP | `run_omp.sh` | `MAX_CTX=65536 BUDGET=22 VERIFY_MODE=ddtree EXTRA_SERVER_ARGS=--lazy-draft OMP_TIMEOUT=3600` | | Pi | `run_pi.sh` | `MAX_CTX=65536 BUDGET=22 VERIFY_MODE=ddtree EXTRA_SERVER_ARGS=--lazy-draft PI_TIMEOUT=3600` | | OpenClaw | `run_openclaw.sh` | `MAX_CTX=204800 BUDGET=22 VERIFY_MODE=ddtree EXTRA_SERVER_ARGS=--lazy-draft` | | Open WebUI chat | `run_openwebui.sh` | `MAX_CTX=262144 BUDGET=22 VERIFY_MODE=ddtree EXTRA_SERVER_ARGS=--lazy-draft` | @@ -79,6 +81,17 @@ PROMPT='Explain the repo and end with lucebox-client-ok' harness/clients/run_ope PROMPT_FILE=harness/clients/prompts/repo_inspection.txt harness/clients/run_hermes.sh ``` +OMP uses a generated `models.yml` with Lucebox as a keyless custom +`openai-responses` provider. `OMP_TIMEOUT` controls the launcher's wall-clock +deadline, while `OMP_STREAM_IDLE_TIMEOUT_MS` defaults to one hour so long +prefills are not interrupted by OMP's stream watchdog; set it to `0` to +disable that watchdog. Override either when needed: + +```bash +OMP_TIMEOUT=0 OMP_STREAM_IDLE_TIMEOUT_MS=7200000 \ + harness/clients/run_omp.sh +``` + `PI_TIMEOUT` is Pi's total wall-clock limit in seconds. Its one-hour default allows long-context prefill and long generations to finish; set `PI_TIMEOUT=0` to run without a launcher deadline. The launcher also disables @@ -92,8 +105,8 @@ put the same setting in `~/.pi/agent/settings.json`: The other real-client launchers also allow one hour by default. Override their deadlines with `CLAUDE_TIMEOUT`, `CODEX_TIMEOUT`, `OPENCODE_TIMEOUT`, -`HERMES_TIMEOUT`, `OPENCLAW_TIMEOUT`, or (for Open WebUI's curl probe) -`CURL_MAX_TIME`. The CLI launcher timeouts accept `0` to disable the outer +`OMP_TIMEOUT`, `HERMES_TIMEOUT`, `OPENCLAW_TIMEOUT`, or (for Open WebUI's +curl probe) `CURL_MAX_TIME`. The CLI launcher timeouts accept `0` to disable the outer deadline. OpenCode's provider-level request and chunk deadlines default to one hour too and can be changed with `OPENCODE_REQUEST_TIMEOUT_MS` and `OPENCODE_CHUNK_TIMEOUT_MS`. The server independently sends SSE heartbeat @@ -119,9 +132,9 @@ CLIENT=opencode PROMPT_FILE=harness/clients/prompts/repo_inspection.txt \ harness/clients/run_backend_pair.sh ``` -OpenAI Chat Completions clients can call llama.cpp directly. Claude Code and -Codex use `llamacpp_compat_proxy.py` so their real Anthropic Messages and -Responses requests can be compared too. +OpenAI Chat Completions clients and OMP's Responses client can call llama.cpp +directly. Claude Code and Codex use `llamacpp_compat_proxy.py` so their real +Anthropic Messages and Responses requests can be compared too. ## Notes diff --git a/harness/clients/run_backend_pair.sh b/harness/clients/run_backend_pair.sh index 23df52775..364507340 100755 --- a/harness/clients/run_backend_pair.sh +++ b/harness/clients/run_backend_pair.sh @@ -18,6 +18,7 @@ case "$CLIENT" in openclaw) CLIENT_SCRIPT="$SCRIPT_DIR/run_openclaw.sh" ;; openwebui) CLIENT_SCRIPT="$SCRIPT_DIR/run_openwebui.sh" ;; openwebui_tools) CLIENT_SCRIPT="$SCRIPT_DIR/run_openwebui_tools.sh" ;; + omp) CLIENT_SCRIPT="$SCRIPT_DIR/run_omp.sh" ;; pi) CLIENT_SCRIPT="$SCRIPT_DIR/run_pi.sh" ;; *) echo "unknown CLIENT=$CLIENT" >&2 diff --git a/harness/clients/run_omp.sh b/harness/clients/run_omp.sh new file mode 100755 index 000000000..68ec9622c --- /dev/null +++ b/harness/clients/run_omp.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +: "${MAX_CTX:=65536}" +: "${BUDGET:=22}" +: "${VERIFY_MODE:=ddtree}" +: "${EXTRA_SERVER_ARGS:=--lazy-draft}" +: "${OMP_TOOLS:=read,grep,glob}" +: "${OMP_TIMEOUT:=3600}" +: "${OMP_STREAM_IDLE_TIMEOUT_MS:=3600000}" +if [[ ! "$OMP_TIMEOUT" =~ ^[0-9]+$ ]]; then + echo "OMP_TIMEOUT must be a non-negative integer (seconds; 0 disables it)" >&2 + exit 2 +fi +if [[ ! "$OMP_STREAM_IDLE_TIMEOUT_MS" =~ ^[0-9]+$ ]]; then + echo "OMP_STREAM_IDLE_TIMEOUT_MS must be a non-negative integer (0 disables OMP's stream watchdog)" >&2 + exit 2 +fi +source "$SCRIPT_DIR/common.sh" + +CLIENT_OUT="$LOG_DIR/omp.out" +OMP_BIN="${OMP_BIN:-$CLIENT_WORK_DIR/clients/omp/bin/omp}" +require_client_binary "OMP" "$OMP_BIN" "omp" "OMP_BIN" +HOME_DIR="$LOG_DIR/omp-home" +AGENT_DIR="$HOME_DIR/.omp/agent" +mkdir -p "$AGENT_DIR" "$HOME_DIR/sessions" + +cat > "$AGENT_DIR/models.yml" < "$CLIENT_OUT" 2>&1 +RC=$? +set -e + +finish_report "$CLIENT_OUT" "$RC" +exit "$RC" diff --git a/harness/clients/summarize_backend_pair.py b/harness/clients/summarize_backend_pair.py index 5062cac77..88886d443 100755 --- a/harness/clients/summarize_backend_pair.py +++ b/harness/clients/summarize_backend_pair.py @@ -28,6 +28,7 @@ "openclaw.out", "openwebui.out", "openwebui-tools.out", + "omp.out", "pi.out", } MARKERS = ("OK_DONE", "lucebox-client-ok", "OPENWEBUI_TOOL_OK") @@ -129,6 +130,40 @@ def first_json_value(text: str): return value +def extract_omp_json_text(text: str) -> str | None: + """Pull assistant text out of OMP's `--mode json` event stream. + + OMP echoes the user prompt and tool results as message events too, so only + `role == "assistant"` message content counts as generated output. + """ + parts: list[str] = [] + handled_assistant_event = False + for line in text.splitlines(): + line = line.strip() + if not line.startswith("{"): + continue + try: + event = json.loads(line) + except json.JSONDecodeError: + continue + if not isinstance(event, dict) or event.get("type") != "message_end": + continue + message = event.get("message") + if not isinstance(message, dict) or message.get("role") != "assistant": + continue + handled_assistant_event = True + content = message.get("content") + if isinstance(content, str): + parts.append(content) + elif isinstance(content, list): + for item in content: + if isinstance(item, dict) and isinstance(item.get("text"), str): + parts.append(item["text"]) + if handled_assistant_event: + return "\n".join(part for part in parts if part) + return None + + def extract_generated_text(text: str) -> str: parts: list[str] = [] value = first_json_value(text) @@ -164,6 +199,9 @@ def extract_generated_text(text: str) -> str: return "" if not parts: + omp_text = extract_omp_json_text(text) + if omp_text is not None: + return omp_text for line in text.splitlines(): line = line.strip() if not line: @@ -205,6 +243,31 @@ def tool_call_ok(text: str) -> bool: return "tool_call:" in text or '"tool_calls"' in text +def omp_tool_call_ok(text: str) -> bool: + for line in text.splitlines(): + line = line.strip() + if not line.startswith("{"): + continue + try: + event = json.loads(line) + except json.JSONDecodeError: + continue + if not isinstance(event, dict): + continue + if event.get("type") == "tool_execution_start": + return True + message = event.get("message") + if not isinstance(message, dict): + continue + content = message.get("content") + if isinstance(content, list) and any( + isinstance(item, dict) and item.get("type") == "toolCall" + for item in content + ): + return True + return False + + def preview(text: str, limit: int = 180) -> str: compact = re.sub(r"\s+", " ", text).strip() return compact[:limit] @@ -230,7 +293,11 @@ def summarize_backend(pair_dir: Path, backend: str) -> dict: m = re.search(r"^rc=(\d+)$", backend_out, flags=re.M) if m: rc = m.group(1) - observed_tool_call = tool_call_ok(generated_text) or any(call.get("finish") == "tool_calls" for call in calls) + observed_tool_call = ( + tool_call_ok(generated_text) + or omp_tool_call_ok(client_text) + or any(call.get("finish") == "tool_calls" for call in calls) + ) return { "backend": backend, "run_dir": str(run_dir), diff --git a/harness/tests/test_client_launcher_timeouts.sh b/harness/tests/test_client_launcher_timeouts.sh index 712cfcf55..45ef5459e 100644 --- a/harness/tests/test_client_launcher_timeouts.sh +++ b/harness/tests/test_client_launcher_timeouts.sh @@ -117,6 +117,7 @@ launcher_cases=( 'run_hermes.sh|HERMES_BIN|HERMES_TIMEOUT|1' 'run_openclaw.sh|OPENCLAW_BIN|OPENCLAW_TIMEOUT|2' 'run_opencode.sh|OPENCODE_BIN|OPENCODE_TIMEOUT|1' + 'run_omp.sh|OMP_BIN|OMP_TIMEOUT|1' ) for launcher_case in "${launcher_cases[@]}"; do IFS='|' read -r script client_var timeout_var expected_calls <<<"$launcher_case" @@ -159,6 +160,8 @@ grep -Fq ': "${HERMES_TIMEOUT:=3600}"' "$CLIENTS/run_hermes.sh" grep -Fq 'run_with_timeout "$HERMES_TIMEOUT"' "$CLIENTS/run_hermes.sh" grep -Fq ': "${CLAUDE_TIMEOUT:=3600}"' "$CLIENTS/run_claude_code.sh" grep -Fq 'run_with_timeout "$CLAUDE_TIMEOUT"' "$CLIENTS/run_claude_code.sh" +grep -Fq ': "${OMP_TIMEOUT:=3600}"' "$CLIENTS/run_omp.sh" +grep -Fq 'run_with_timeout "$OMP_TIMEOUT"' "$CLIENTS/run_omp.sh" grep -Fq ': "${OPENCLAW_TIMEOUT:=3600}"' "$CLIENTS/run_openclaw.sh" grep -Fq 'run_with_timeout "$OPENCLAW_TIMEOUT"' "$CLIENTS/run_openclaw.sh" grep -Fq 'openclaw_cmd+=(--timeout "$OPENCLAW_TIMEOUT")' "$CLIENTS/run_openclaw.sh" diff --git a/harness/tests/test_run_omp_config.sh b/harness/tests/test_run_omp_config.sh new file mode 100755 index 000000000..395ba469a --- /dev/null +++ b/harness/tests/test_run_omp_config.sh @@ -0,0 +1,133 @@ +#!/usr/bin/env bash + +set -euo pipefail + +RUN_OMP="${1:?usage: test_run_omp_config.sh }" +TMP_DIR="$(mktemp -d)" +trap 'rm -rf "$TMP_DIR"' EXIT + +FAKE_BIN="$TMP_DIR/bin" +FAKE_TARGET="$TMP_DIR/model.gguf" +FAKE_DRAFT="$TMP_DIR/draft.gguf" +FAKE_SERVER="$TMP_DIR/dflash_server" +FAKE_OMP="$TMP_DIR/omp" +mkdir -p "$FAKE_BIN" +touch "$FAKE_TARGET" "$FAKE_DRAFT" + +cat >"$FAKE_SERVER" <<'EOF' +#!/usr/bin/env bash +exec sleep 600 +EOF + +cat >"$FAKE_OMP" <<'EOF' +#!/usr/bin/env bash +printf '%s\n' "$*" >"${OMP_ARGS_CAPTURE:?}" +printf 'HOME=%s\n' "$HOME" >>"${OMP_ARGS_CAPTURE}" +printf 'PI_CODING_AGENT_DIR=%s\n' "$PI_CODING_AGENT_DIR" >>"${OMP_ARGS_CAPTURE}" +printf 'OMP_PROFILE=%s\n' "$OMP_PROFILE" >>"${OMP_ARGS_CAPTURE}" +printf 'PI_PROFILE=%s\n' "$PI_PROFILE" >>"${OMP_ARGS_CAPTURE}" +EOF + +cat >"$FAKE_BIN/curl" <<'EOF' +#!/usr/bin/env bash +exit 0 +EOF + +cat >"$FAKE_BIN/nvidia-smi" <<'EOF' +#!/usr/bin/env bash +exit 0 +EOF + +cat >"$FAKE_BIN/timeout" <<'EOF' +#!/usr/bin/env bash +printf '%s\n' "$1" >"${TIMEOUT_CAPTURE:?}" +shift +exec "$@" +EOF + +chmod +x "$FAKE_SERVER" "$FAKE_OMP" "$FAKE_BIN"/* + +ARGS_CAPTURE="$TMP_DIR/omp-args" +TIMEOUT_CAPTURE="$TMP_DIR/omp-timeout" +env \ + PATH="$FAKE_BIN:$PATH" \ + RUN_DIR="$TMP_DIR/runs" \ + STAMP=omp-contract \ + TARGET="$FAKE_TARGET" \ + DRAFT="$FAKE_DRAFT" \ + FA_WINDOW=1 \ + DFLASH_SERVER_BIN="$FAKE_SERVER" \ + OMP_BIN="$FAKE_OMP" \ + AUTO_INSTALL_CLIENTS=0 \ + OMP_ARGS_CAPTURE="$ARGS_CAPTURE" \ + TIMEOUT_CAPTURE="$TIMEOUT_CAPTURE" \ + MODEL_ID=omp-test-model \ + MAX_CTX=32768 \ + MAX_TOKENS=768 \ + OMP_STREAM_IDLE_TIMEOUT_MS=720000 \ + OMP_PROFILE=inherited-profile \ + PI_PROFILE=inherited-profile \ + PROMPT=omp-contract-prompt \ + bash "$RUN_OMP" >/dev/null + +MODELS="$TMP_DIR/runs/omp-contract/omp-home/.omp/agent/models.yml" +grep -Fq 'baseUrl: "http://127.0.0.1:18080/v1"' "$MODELS" +grep -Fq 'auth: none' "$MODELS" +grep -Fq 'api: openai-responses' "$MODELS" +grep -Fq 'supportsDeveloperRole: false' "$MODELS" +grep -Fq 'supportsReasoningEffort: false' "$MODELS" +grep -Fq 'maxTokensField: max_tokens' "$MODELS" +grep -Fq 'streamIdleTimeoutMs: 720000' "$MODELS" +grep -Fq 'id: "omp-test-model"' "$MODELS" +grep -Fq 'contextWindow: 32768' "$MODELS" +grep -Fq 'maxTokens: 768' "$MODELS" + +grep -Fq -- '--model lucebox/omp-test-model' "$ARGS_CAPTURE" +grep -Fq -- '--print --mode json' "$ARGS_CAPTURE" +grep -Fq -- '--tools read,grep,glob' "$ARGS_CAPTURE" +grep -Fq -- '--no-session --no-extensions --no-skills --no-rules --no-title' "$ARGS_CAPTURE" +grep -Fq -- 'omp-contract-prompt' "$ARGS_CAPTURE" +grep -Fq "PI_CODING_AGENT_DIR=$TMP_DIR/runs/omp-contract/omp-home/.omp/agent" "$ARGS_CAPTURE" +grep -Fxq 'OMP_PROFILE=' "$ARGS_CAPTURE" +grep -Fxq 'PI_PROFILE=' "$ARGS_CAPTURE" +grep -Fxq '3600s' "$TIMEOUT_CAPTURE" + +if [[ -n "${REAL_OMP_BIN:-}" ]]; then + real_models="$( + env \ + HOME="$TMP_DIR/runs/omp-contract/omp-home" \ + PI_CODING_AGENT_DIR="$TMP_DIR/runs/omp-contract/omp-home/.omp/agent" \ + PI_CODING_AGENT_SESSION_DIR="$TMP_DIR/runs/omp-contract/omp-home/sessions" \ + "$REAL_OMP_BIN" models + )" + grep -Fq 'lucebox (1)' <<<"$real_models" + grep -Fq 'omp-test-model' <<<"$real_models" +fi + +set +e +invalid_output="$( + OMP_BIN="$FAKE_OMP" OMP_STREAM_IDLE_TIMEOUT_MS=invalid \ + AUTO_INSTALL_CLIENTS=0 bash "$RUN_OMP" 2>&1 +)" +invalid_rc=$? +set -e +if [[ "$invalid_rc" -ne 2 ]] || + ! grep -Fq 'OMP_STREAM_IDLE_TIMEOUT_MS must be a non-negative integer (0 disables OMP'\''s stream watchdog)' <<<"$invalid_output"; then + echo "invalid OMP stream timeout must fail with a useful error" >&2 + exit 1 +fi + +set +e +invalid_output="$( + OMP_BIN="$FAKE_OMP" OMP_TIMEOUT=invalid \ + AUTO_INSTALL_CLIENTS=0 bash "$RUN_OMP" 2>&1 +)" +invalid_rc=$? +set -e +if [[ "$invalid_rc" -ne 2 ]] || + ! grep -Fq 'OMP_TIMEOUT must be a non-negative integer (seconds; 0 disables it)' <<<"$invalid_output"; then + echo "invalid OMP_TIMEOUT must fail with a useful error before startup" >&2 + exit 1 +fi + +echo "OMP launcher configuration: PASS" diff --git a/harness/tests/test_summarize_backend_pair.py b/harness/tests/test_summarize_backend_pair.py new file mode 100644 index 000000000..d902700b4 --- /dev/null +++ b/harness/tests/test_summarize_backend_pair.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""Regression tests for backend-pair client output parsing.""" + +from __future__ import annotations + +import importlib.util +import json +import unittest +from pathlib import Path + +SCRIPT = Path(__file__).resolve().parents[1] / "clients" / "summarize_backend_pair.py" +SPEC = importlib.util.spec_from_file_location("summarize_backend_pair", SCRIPT) +if SPEC is None or SPEC.loader is None: + raise RuntimeError(f"could not load {SCRIPT}") +SUMMARY = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(SUMMARY) + + +def event(payload: dict) -> str: + return json.dumps(payload, separators=(",", ":")) + + +class OmpToolCallTests(unittest.TestCase): + def test_tool_call_only_assistant_does_not_echo_other_messages(self) -> None: + output = "\n".join(( + event({ + "type": "message_end", + "message": { + "role": "user", + "content": [{"type": "text", "text": "echoed user prompt"}], + }, + }), + event({ + "type": "message_end", + "message": { + "role": "assistant", + "content": [{"type": "toolCall", "name": "read", "arguments": {}}], + "stopReason": "toolUse", + }, + }), + event({ + "type": "message_end", + "message": { + "role": "toolResult", + "content": [{"type": "text", "text": "echoed tool result"}], + }, + }), + )) + + self.assertEqual(SUMMARY.extract_generated_text(output), "") + + def test_tool_use_stop_without_call_is_not_success(self) -> None: + output = event({ + "type": "message_end", + "message": { + "role": "assistant", + "content": [], + "stopReason": "toolUse", + }, + }) + + self.assertFalse(SUMMARY.omp_tool_call_ok(output)) + + def test_tool_call_text_is_not_success(self) -> None: + output = event({ + "type": "message_end", + "message": { + "role": "assistant", + "content": [{"type": "text", "text": '{"type":"toolCall"}'}], + "stopReason": "stop", + }, + }) + + self.assertFalse(SUMMARY.omp_tool_call_ok(output)) + + def test_tool_call_content_is_success(self) -> None: + output = event({ + "type": "message_end", + "message": { + "role": "assistant", + "content": [{"type": "toolCall", "name": "read", "arguments": {}}], + "stopReason": "toolUse", + }, + }) + + self.assertTrue(SUMMARY.omp_tool_call_ok(output)) + + def test_tool_execution_start_is_success(self) -> None: + output = "\n".join(( + "OMP diagnostic", + event({"type": "tool_execution_start", "toolName": "read"}), + )) + + self.assertTrue(SUMMARY.omp_tool_call_ok(output)) + + +if __name__ == "__main__": + unittest.main() diff --git a/server/CMakeLists.txt b/server/CMakeLists.txt index 57b663156..5808fb3af 100644 --- a/server/CMakeLists.txt +++ b/server/CMakeLists.txt @@ -1434,6 +1434,28 @@ if(DFLASH27B_TESTS) unset(_pi_timeout_test) unset(_pi_launcher) + set(_omp_config_test + "${CMAKE_CURRENT_SOURCE_DIR}/../harness/tests/test_run_omp_config.sh") + set(_omp_launcher "${CMAKE_CURRENT_SOURCE_DIR}/../harness/clients/run_omp.sh") + if(UNIX AND EXISTS "${_omp_config_test}" AND EXISTS "${_omp_launcher}") + add_test( + NAME server_unit_omp_harness_config + COMMAND bash "${_omp_config_test}" "${_omp_launcher}") + endif() + unset(_omp_config_test) + unset(_omp_launcher) + + set(_backend_pair_summary_test + "${CMAKE_CURRENT_SOURCE_DIR}/../harness/tests/test_summarize_backend_pair.py") + find_program(_harness_python3 python3) + if(_harness_python3 AND EXISTS "${_backend_pair_summary_test}") + add_test( + NAME server_unit_backend_pair_summary + COMMAND "${_harness_python3}" "${_backend_pair_summary_test}") + endif() + unset(_backend_pair_summary_test) + unset(_harness_python3 CACHE) + set(_client_timeout_test "${CMAKE_CURRENT_SOURCE_DIR}/../harness/tests/test_client_launcher_timeouts.sh") if(UNIX AND EXISTS "${_client_timeout_test}") diff --git a/server/README.md b/server/README.md index 1e96b697d..824bb8ec1 100644 --- a/server/README.md +++ b/server/README.md @@ -148,8 +148,8 @@ The default draft path is discovered under `models/draft/`. Scripts prefer `dfla `dflash_server` serves the same client-facing local API surface used by the harnesses. It supports `/health`, `/v1/models`, OpenAI Chat Completions including streaming and tool metadata, -OpenAI Responses for Codex, Anthropic Messages for Claude Code, and Open WebUI -model metadata. +OpenAI Responses for Codex and OMP, Anthropic Messages for Claude Code, and +Open WebUI model metadata. Build it with the rest of the CUDA runtime: