diff --git a/optimizations/ooo_spec_lucebox5_cpu/README.md b/optimizations/ooo_spec_lucebox5_cpu/README.md new file mode 100644 index 000000000..75857b1f4 --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/README.md @@ -0,0 +1,170 @@ +# CPU-isolated tool speculation on Lucebox5 + +The engine can ask a small model for one concrete tool call before the target +model runs. If the call is allowlisted and its measured execution lane is +qualified, the engine starts the read-only tool privately. The target model +remains authoritative: the result is released only when the emitted function +name and canonical arguments match exactly. + +On Lucebox5, Qwen3-0.6B Q8_0 predicts on Strix and then leaves its compute +window. The CPU tool runs on logical CPUs `14-15,30-31` while +DeepSeek-V4-0731 + DS4/DSpark runs on R9700 + Strix and CPUs +`0-13,16-29`. This path neither injects tokens nor replaces DSpark. + +## Measured result + +The 2026-08-18 production run used six paired tasks: two each with 10, 15, +and 20 leaf calls. Each branch contained five serial, deterministic read-only +calls; independent branches ran concurrently. Arm order was randomized. +Every arm included model turns, tool time, and a final answer produced from +the actual assistant/tool conversation. + +| Metric | Result | +| --- | ---: | +| Stage-batched workflow, p50 | 87.998 s | +| Trace-compiled workflow, p50 | 32.877 s | +| Trace-compiled + speculative workflow, p50 | **20.474 s** | +| End-to-end paired speedup, p50 | **4.3597x** | +| End-to-end bootstrap 95% CI | **3.7696x–4.8252x** | +| End-to-end paired speedup, p05 | **3.6680x** | +| Trace compilation alone, paired p50 | **2.6876x** | +| Early launch on top of compilation, paired p50 | **1.7676x** | +| Early-launch bootstrap 95% CI | **1.2590x–1.8002x** | +| Exposed tool wait, compiled / speculative p50 | 10.149 s / **0.030 ms** | +| Qwen prediction latency, p50 | 201.0 ms | +| Target model-compute change, p50 / p95 | -0.027% / +0.090% | +| Target decode change, p50 / p95 | -0.658% / +0.374% | +| Exact Qwen predictor hits | 6 / 6 | + +The slowdown figures come from a separate controlled A/B probe. Each task had +three alternating repetitions per arm, and every observation was preceded by +the same warm request. The gate compares the per-task median ratios and +requires matching cache state, completion tokens, call digest, and active +DS4 decoding. + +All 22 gates passed, including stable leaf-call and result digests, exact +final answers, exact macro calls, CPU isolation, controlled p50/p95 model +slowdown, and a wrong-call probe in which no private result crossed the +exact-match gate. + +These numbers apply to recognized, side-effect-free recurring workflows. The +4.36x combines fewer model/tool synchronization barriers from trace +compilation with the 1.77x gain from early tool launch. It is not a claim that +arbitrary single tool calls become 4.36x faster. The broader Qwen smoke suite +currently protects a 9/12 exact-argument baseline; a predictor miss falls back +to the authoritative call without changing the target output. + +Evidence: + +- `results/trace-compiled-engine-qwen-production-6pairs-compact.json` + (`sha256:b1194bd1447f772dc9c90e6e801e99646bdce121e1b300398c45b54540b87d20`) +- `results/multiturn-cached-wordref-production-6tasks.json` + (`sha256:2475697d418bffed0e9668da26ce6c88a85a952ce97d99749f440f97f9ac5bf9`) +- `results/trace-workflow-registry.json` + (the benchmark artifact records its exact hash) + +The report and registry paths inside the result are repository-relative, and +their recorded hashes match the committed files. + +The full source report is retained deliberately. Its original per-call +speculation performance gate did not pass; the compiler reads only the two +executions whose calls, results, dataflow, and side-effect flags are valid. +That failed baseline is the reason the workflow was compiled. No model was +trained on this report, and the passing 4.36x result is recorded separately. + +## Runtime contract + +- Tools must be explicitly allowlisted and read-only or idempotent. +- A result is committed only on one exact canonical call match. Multiple, + malformed, failed, timed-out, cancelled, or different calls are discarded. +- The child receives a minimal environment and only standard streams. Tool + speculation fails closed unless Linux glibc 2.34+ descriptor isolation is + available. +- CPU affinity is applied before the executor starts and is re-read before + the request payload is released. The model and tool masks must be disjoint. +- Executors run in their own process group with a launch-based deadline and a + bounded output size. Cancellation removes descendants. +- Native predictor IPC has a bounded startup, a private `0700` work directory, + an inherited-descriptor allowlist, serialized requests, and deadline-aware + prompt construction, tokenization, and generation. +- A production profile must declare a qualified non-accelerator or separate + physical-GPU lane and match the configured executor contract. + +Automatic prediction means clients do not need to send a prediction hint. +Clients do need to consume `dflash_tool_speculation.result` on a `hit` and use +it as the tool result; ignoring the extension remains correct but forfeits the +latency gain. + +The schedule also supports a single GPU because predictor compute finishes +before target compute begins; only the CPU tool overlaps target generation. +Both models still have to fit in memory. The production numbers above were +measured on Lucebox5's dual-GPU placement, not on a single-GPU machine. + +## Reproduce on Lucebox5 + +Build the deterministic sparse adapter used to qualify the CPU lane: + +```bash +JSON_INCLUDE=/path/to/server/deps/json/include \ + ./build_cpu_sparse_executor.sh ./cpu_sparse_tool_executor +``` + +Launch the qualified server: + +```bash +./run_native_cpu_server_lucebox5.sh +``` + +The launcher uses Qwen3-0.6B Q8_0 on predictor GPU 1 and the build selected by +the wrapper's adjacent `candidate-build` symlink. It clears ambient variables. +`PREDICTOR_MODEL`, `PREDICTOR_GPU`, `PREDICTOR_MAX_CTX`, +`PREDICTOR_MAX_TOKENS`, and `PREDICTOR_TIMEOUT_MS` apply only when invoking +the wrapper directly. + +Run the single-call gate: + +```bash +python3 benchmark_cpu_tool_speculation.py native-qwen \ + --url http://127.0.0.1:18145/v1/chat/completions \ + --binary ./cpu_sparse_tool_executor \ + --tool-cpus 14-15,30-31 \ + --iterations 172452 \ + --max-tokens 32 \ + --pairs 20 \ + --warmups 5 \ + --bootstrap-resamples 20000 \ + --min-speedup 1.6 \ + --min-speedup-ci-low 1.5 \ + --min-speedup-p05 1.5 \ + --min-prediction-hit-rate 1.0 \ + --max-model-slowdown-percent 5 \ + --output results/qwen-auto-production-20pairs.json +``` + +For the workflow gate, launch with the trace executor and macro allowlist: + +```bash +TOOL_SPEC_EXECUTOR=./trace_compiled_tool_executor.py \ +TOOL_SPEC_ALLOW=resolve_customer,list_open_orders,get_order_details,calculate_shipping,prepare_customer_summary,execute_customer_workflows \ + ./run_native_cpu_server_lucebox5.sh +``` + +Then run: + +```bash +python3 benchmark_trace_compiled_workflows.py \ + --binary ./bfcl_replay_tool_executor.py \ + --training-report results/multiturn-cached-wordref-production-6tasks.json \ + --workflow-registry results/trace-workflow-registry.json \ + --pairs 6 \ + --warmup-tasks 1 \ + --min-branches 2 \ + --max-branches 4 \ + --interference-repetitions 3 \ + --seed 814 \ + --bootstrap-resamples 20000 \ + --output results/trace-compiled-engine-qwen-production-6pairs-compact.json +``` + +The harness exits nonzero on any correctness, privacy, isolation, DS4, +slowdown, hit-rate, or speed failure. diff --git a/optimizations/ooo_spec_lucebox5_cpu/benchmark_cpu_tool_speculation.py b/optimizations/ooo_spec_lucebox5_cpu/benchmark_cpu_tool_speculation.py new file mode 100755 index 000000000..ac70881b2 --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/benchmark_cpu_tool_speculation.py @@ -0,0 +1,1453 @@ +#!/usr/bin/env python3 +"""Qualify and benchmark a disjoint Strix CPU speculative-tool lane. + +The qualification phase runs the official model server without tool +speculation, pins the real sparse-compute tool to reserved physical cores, and +measures sequential versus overlapped execution. It emits a qualified engine +profile only if model output, DS4 activity, tool results, CPU isolation, and +the slowdown gate all pass. + +The native phase then measures the engine's exact-call commit path against the +same strong sequential CPU baseline. Wrong predictions are cancelled and +their private results must never be exposed. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import os +import random +import re +import signal +import statistics +import subprocess +import time +import urllib.request +from pathlib import Path +from typing import Any, Iterable +from urllib.parse import urlsplit, urlunsplit + + +TOOL_NAME = "benchmark_cpu_sparse" +PROTOCOL = "dflash.tool-speculation.v1" +NATIVE_PREDICTION_SOURCE = "native-qwen3" +SPARSE_ROWS = 4096 +SPARSE_NONZEROS_PER_ROW = 16 +SPARSE_THREADS = 2 +SPARSE_SEED = 731 + + +def parse_cpu_list(value: str) -> list[int]: + cpus: set[int] = set() + for item in value.split(","): + if not item: + raise argparse.ArgumentTypeError("CPU list contains an empty item") + if "-" in item: + parts = item.split("-") + if len(parts) != 2 or not all(part.isdigit() for part in parts): + raise argparse.ArgumentTypeError(f"invalid CPU range: {item}") + first, last = map(int, parts) + if first > last: + raise argparse.ArgumentTypeError(f"invalid CPU range: {item}") + cpus.update(range(first, last + 1)) + elif item.isdigit(): + cpus.add(int(item)) + else: + raise argparse.ArgumentTypeError(f"invalid CPU id: {item}") + if not cpus: + raise argparse.ArgumentTypeError("CPU list must not be empty") + return sorted(cpus) + + +def compact_cpu_list(cpus: Iterable[int]) -> str: + return ",".join(str(cpu) for cpu in cpus) + + +def expected_arguments( + rows: int, + nonzeros_per_row: int, + iterations: int, + threads: int, + seed: int, +) -> dict[str, int]: + if ( + rows != SPARSE_ROWS + or nonzeros_per_row != SPARSE_NONZEROS_PER_ROW + or threads != SPARSE_THREADS + or seed != SPARSE_SEED + ): + raise ValueError("this qualification binary has a fixed sparse shape") + # Keep the generated tool call intentionally short. The deterministic + # benchmark binary owns the qualified sparse shape; only work duration is + # request-dependent, matching real tools with a compact identifier. + return {"iterations": iterations} + + +def tool_definition() -> dict[str, Any]: + properties = {"iterations": {"type": "integer"}} + return { + "name": TOOL_NAME, + "parameters": { + "type": "object", + "properties": properties, + "required": list(properties), + "additionalProperties": False, + }, + } + + +def request_body( + arguments: dict[str, int], + max_tokens: int, + *, + prediction: dict[str, int] | None, + automatic_prediction: bool = False, + tool_choice: str | None = None, +) -> dict[str, Any]: + compact = json.dumps(arguments, separators=(",", ":")) + prompt = f"Return only this JSON object and nothing else: {compact}" + body: dict[str, Any] = { + "model": "dflash", + "stream": False, + "max_tokens": max_tokens, + "temperature": 0, + "messages": [ + { + "role": "user", + "content": prompt, + } + ], + "tools": [tool_definition()], + "automatic_tool_speculation": automatic_prediction, + } + if tool_choice is not None: + body["tool_choice"] = tool_choice + if prediction is not None: + body["tool_speculation"] = { + "call": {"name": TOOL_NAME, "arguments": prediction}, + "confidence": 1.0, + } + return body + + +def normalize_tool_call(result: dict[str, Any]) -> dict[str, Any] | None: + message = result.get("choices", [{}])[0].get("message", {}) + tool_calls = message.get("tool_calls") or [] + if len(tool_calls) == 1: + function = tool_calls[0].get("function") or {} + arguments = function.get("arguments", "{}") + if isinstance(arguments, str): + try: + arguments = json.loads(arguments) + except json.JSONDecodeError: + return None + return {"name": function.get("name"), "arguments": arguments} + content = message.get("content") + if not isinstance(content, str) or not content: + return None + bracket_call = re.fullmatch(r'\["([^"]+)"\]\((\{.*\})\)', content) + if bracket_call: + try: + return { + "name": bracket_call.group(1), + "arguments": json.loads(bracket_call.group(2)), + } + except json.JSONDecodeError: + return None + try: + parsed = json.loads(content) + except json.JSONDecodeError: + return None + if not isinstance(parsed, dict): + return None + function = parsed.get("function", parsed.get("name")) + if isinstance(function, dict): + name = function.get("name") + arguments = function.get("arguments", function.get("parameters")) + else: + name = function + arguments = parsed.get( + "params", + parsed.get( + "parameters", + parsed.get("arguments", parsed.get("function_args")), + ), + ) + if ( + arguments is None + and isinstance(parsed.get("parameter"), str) + and "parameter_value" in parsed + ): + # DeepSeek may serialize a one-argument native call as a compact + # name/value envelope. It is semantically the same function call. + arguments = {parsed["parameter"]: parsed["parameter_value"]} + if arguments is None and isinstance(name, str): + arguments = { + key: value + for key, value in parsed.items() + if key not in {"function", "name", "type"} + } + if isinstance(arguments, str): + try: + arguments = json.loads(arguments) + except json.JSONDecodeError: + return None + if not isinstance(name, str) or not isinstance(arguments, dict): + return None + return {"name": name, "arguments": arguments} + + +def post_json( + url: str, body: dict[str, Any], timeout: float +) -> tuple[dict[str, Any], float]: + request = urllib.request.Request( + url, + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"Content-Type": "application/json"}, + method="POST", + ) + started = time.perf_counter() + with urllib.request.urlopen(request, timeout=timeout) as response: + result = json.load(response) + if not isinstance(result, dict): + raise RuntimeError("model response is not a JSON object") + return result, (time.perf_counter() - started) * 1000.0 + + +def get_json(url: str, timeout: float) -> dict[str, Any]: + with urllib.request.urlopen(url, timeout=timeout) as response: + result = json.load(response) + if not isinstance(result, dict): + raise RuntimeError(f"expected a JSON object from {url}") + return result + + +def props_url(completion_url: str) -> str: + parsed = urlsplit(completion_url) + return urlunsplit((parsed.scheme, parsed.netloc, "/props", "", "")) + + +def observation(result: dict[str, Any], wall_ms: float) -> dict[str, Any]: + call = normalize_tool_call(result) + usage = result.get("usage") or {} + timings = usage.get("timings") or {} + message = result.get("choices", [{}])[0].get("message", {}) + content = message.get("content") or "" + canonical_call = json.dumps(call, sort_keys=True, separators=(",", ":")) + return { + "request_wall_ms": wall_ms, + "model_compute_ms": float(timings.get("prefill_ms", 0.0)) + + float(timings.get("decode_ms", 0.0)), + "prefill_ms": float(timings.get("prefill_ms", 0.0)), + "decode_ms": float(timings.get("decode_ms", 0.0)), + "decode_tokens_per_sec": float( + timings.get("decode_tokens_per_sec", 0.0) + ), + "completion_tokens": int(usage.get("completion_tokens", 0)), + "accept_rate": float(usage.get("accept_rate", 0.0)), + "tool_call": call, + "tool_call_sha256": hashlib.sha256(canonical_call.encode()).hexdigest(), + "assistant_content_sha256": hashlib.sha256(content.encode()).hexdigest(), + "speculation": result.get("dflash_tool_speculation"), + } + + +def post_model( + url: str, + arguments: dict[str, int], + max_tokens: int, + timeout: float, + *, + prediction: dict[str, int] | None = None, + automatic_prediction: bool = False, + tool_choice: str | None = None, +) -> dict[str, Any]: + result, wall_ms = post_json( + url, + request_body( + arguments, + max_tokens, + prediction=prediction, + automatic_prediction=automatic_prediction, + tool_choice=tool_choice, + ), + timeout, + ) + return observation(result, wall_ms) + + +def executor_request( + arguments: dict[str, int], cpus: list[int], request_id: str +) -> dict[str, Any]: + return { + "protocol": PROTOCOL, + "request_id": request_id, + "mode": "authoritative-benchmark", + "resource_percentage": 100, + "accelerator_relation": "non_accelerator", + "cpu_affinity": cpus, + "cpu_affinity_isolated": True, + "call": {"name": TOOL_NAME, "arguments": arguments}, + } + + +def start_executor( + binary: Path, + arguments: dict[str, int], + cpus: list[int], + request_id: str, +) -> dict[str, Any]: + environment = os.environ.copy() + environment["DFLASH_TOOL_SPECULATION_CPU_AFFINITY"] = compact_cpu_list(cpus) + + def pin_child() -> None: + os.sched_setaffinity(0, set(cpus)) + + started = time.perf_counter() + process = subprocess.Popen( + [str(binary), "--dflash-tool-spec-v1"], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + env=environment, + preexec_fn=pin_child, + start_new_session=True, + ) + assert process.stdin is not None + process.stdin.write( + json.dumps( + executor_request(arguments, cpus, request_id), + separators=(",", ":"), + ) + + "\n" + ) + process.stdin.close() + process.stdin = None + return {"process": process, "pgid": process.pid, "started": started} + + +def finish_executor(handle: dict[str, Any], timeout: float) -> dict[str, Any]: + process: subprocess.Popen[str] = handle["process"] + elapsed = time.perf_counter() - float(handle["started"]) + remaining = timeout - elapsed + if remaining <= 0 and process.poll() is None: + stop_executor(handle) + raise RuntimeError("CPU executor timed out") + try: + stdout, stderr = process.communicate(timeout=max(0.001, remaining)) + except subprocess.TimeoutExpired: + stop_executor(handle) + raise RuntimeError("CPU executor timed out") + wall_ms = (time.perf_counter() - float(handle["started"])) * 1000.0 + if process.returncode != 0: + raise RuntimeError( + f"CPU executor exited {process.returncode}: {stderr.strip()}" + ) + try: + envelope = json.loads(stdout) + except json.JSONDecodeError as error: + raise RuntimeError(f"CPU executor returned invalid JSON: {stdout!r}") from error + if not isinstance(envelope, dict) or not envelope.get("ok"): + raise RuntimeError(f"CPU executor rejected request: {envelope!r}") + result = envelope.get("result") + if not isinstance(result, dict): + raise RuntimeError("CPU executor result is not an object") + return {"wall_ms": wall_ms, "result": result} + + +def stop_executor(handle: dict[str, Any]) -> None: + process: subprocess.Popen[str] = handle["process"] + pgid = int(handle.get("pgid", process.pid)) + + def group_exists() -> bool: + try: + os.killpg(pgid, 0) + return True + except ProcessLookupError: + return False + except PermissionError: + return True + + def wait_for_group_exit(timeout: float) -> bool: + deadline = time.monotonic() + timeout + while group_exists(): + remaining = deadline - time.monotonic() + if remaining <= 0: + return False + if process.poll() is None: + try: + process.wait(timeout=min(0.01, remaining)) + except subprocess.TimeoutExpired: + pass + else: + time.sleep(min(0.01, remaining)) + return True + + try: + os.killpg(pgid, signal.SIGTERM) + except (ProcessLookupError, PermissionError): + pass + + if not wait_for_group_exit(1.0): + try: + os.killpg(pgid, signal.SIGKILL) + except (ProcessLookupError, PermissionError): + pass + if not wait_for_group_exit(5.0): + raise RuntimeError("CPU executor process group did not stop") + + if process.poll() is None: + try: + process.wait(timeout=5.0) + except subprocess.TimeoutExpired as error: + raise RuntimeError("CPU executor process group did not stop") from error + + +def run_executor( + binary: Path, + arguments: dict[str, int], + cpus: list[int], + timeout: float, + request_id: str, +) -> dict[str, Any]: + return finish_executor( + start_executor(binary, arguments, cpus, request_id), timeout + ) + + +def expected_call(arguments: dict[str, int]) -> dict[str, Any]: + return {"name": TOOL_NAME, "arguments": arguments} + + +def validate_model_call(row: dict[str, Any], arguments: dict[str, int]) -> None: + expected = expected_call(arguments) + if row["tool_call"] != expected: + raise RuntimeError( + f"model emitted {row['tool_call']!r}, expected {expected!r}" + ) + + +def percentile(values: Iterable[float], quantile: float) -> float: + ordered = sorted(float(value) for value in values) + if not ordered: + raise ValueError("cannot take percentile of an empty sequence") + position = (len(ordered) - 1) * quantile + lower = math.floor(position) + upper = math.ceil(position) + if lower == upper: + return ordered[lower] + fraction = position - lower + return ordered[lower] * (1.0 - fraction) + ordered[upper] * fraction + + +def bootstrap_speedup_ci( + pairs: list[dict[str, Any]], resamples: int, seed: int +) -> list[float]: + generator = random.Random(seed) + ratios = [] + for _ in range(resamples): + sample = [pairs[generator.randrange(len(pairs))] for _ in pairs] + control = statistics.median( + float(pair["control"]["task_ms"]) for pair in sample + ) + speculative = statistics.median( + float(pair["speculative"]["task_ms"]) for pair in sample + ) + ratios.append(control / speculative) + return [percentile(ratios, 0.025), percentile(ratios, 0.975)] + + +def read_process_affinity(pid: int) -> list[int]: + return sorted(os.sched_getaffinity(pid)) + + +def calibrate( + args: argparse.Namespace, +) -> tuple[dict[str, int], dict[str, Any]]: + probe_arguments = expected_arguments( + args.rows, + args.nonzeros_per_row, + max(1, args.initial_iterations), + args.threads, + args.tool_seed, + ) + model_samples = [] + for _ in range(args.calibration_model_samples): + row = post_model( + args.url, probe_arguments, args.max_tokens, args.timeout + ) + validate_model_call(row, probe_arguments) + model_samples.append(row) + target_ms = statistics.median( + float(row["request_wall_ms"]) for row in model_samples + ) + + iterations = args.initial_iterations + calibration_steps = [] + for step in range(args.calibration_steps): + arguments = expected_arguments( + args.rows, + args.nonzeros_per_row, + iterations, + args.threads, + args.tool_seed, + ) + samples = [ + run_executor( + args.binary, + arguments, + args.tool_cpus, + args.timeout, + f"calibrate-{step}-{sample}", + ) + for sample in range(args.calibration_tool_samples) + ] + observed_ms = statistics.median( + float(sample["wall_ms"]) for sample in samples + ) + calibration_steps.append( + { + "iterations": iterations, + "tool_wall_p50_ms": observed_ms, + "samples": samples, + } + ) + if observed_ms <= 0: + raise RuntimeError("CPU executor calibration returned zero time") + ratio = target_ms / observed_ms + if 0.97 <= ratio <= 1.03: + break + iterations = max(1, round(iterations * ratio)) + + final_arguments = expected_arguments( + args.rows, + args.nonzeros_per_row, + iterations, + args.threads, + args.tool_seed, + ) + return final_arguments, { + "target_model_request_p50_ms": target_ms, + "model_samples": model_samples, + "steps": calibration_steps, + "selected_iterations": iterations, + } + + +def run_direct_control( + args: argparse.Namespace, + arguments: dict[str, int], + label: str, +) -> dict[str, Any]: + started = time.perf_counter() + model = post_model(args.url, arguments, args.max_tokens, args.timeout) + tool = run_executor( + args.binary, arguments, args.tool_cpus, args.timeout, f"{label}-tool" + ) + validate_model_call(model, arguments) + return { + "mode": "control", + "task_ms": (time.perf_counter() - started) * 1000.0, + "model": model, + "tool": tool, + } + + +def run_direct_overlap( + args: argparse.Namespace, + arguments: dict[str, int], + label: str, +) -> dict[str, Any]: + started = time.perf_counter() + handle = start_executor(args.binary, arguments, args.tool_cpus, label) + try: + model = post_model(args.url, arguments, args.max_tokens, args.timeout) + tool = finish_executor(handle, args.timeout) + except BaseException: + stop_executor(handle) + raise + validate_model_call(model, arguments) + return { + "mode": "speculative", + "task_ms": (time.perf_counter() - started) * 1000.0, + "model": model, + "tool": tool, + } + + +def run_direct_miss( + args: argparse.Namespace, + arguments: dict[str, int], + label: str, +) -> dict[str, Any]: + wrong = dict(arguments) + wrong["iterations"] = max(1, arguments["iterations"] - 1) + if wrong["iterations"] == arguments["iterations"]: + wrong["iterations"] += 1 + started = time.perf_counter() + private = start_executor(args.binary, wrong, args.tool_cpus, f"{label}-wrong") + try: + model = post_model(args.url, arguments, args.max_tokens, args.timeout) + finally: + stop_executor(private) + authoritative = run_executor( + args.binary, + arguments, + args.tool_cpus, + args.timeout, + f"{label}-authoritative", + ) + validate_model_call(model, arguments) + return { + "mode": "miss", + "task_ms": (time.perf_counter() - started) * 1000.0, + "model": model, + "authoritative_tool": authoritative, + } + + +def qualify(args: argparse.Namespace) -> None: + model_affinity = read_process_affinity(args.model_pid) + overlap = sorted(set(model_affinity).intersection(args.tool_cpus)) + if overlap: + raise SystemExit(f"model/tool CPU affinity overlaps: {overlap}") + if args.threads > len(args.tool_cpus): + raise SystemExit("tool threads exceed reserved logical CPUs") + + arguments, calibration = calibrate(args) + for warmup in range(args.warmups): + run_direct_control(args, arguments, f"warmup-control-{warmup}") + run_direct_overlap(args, arguments, f"warmup-overlap-{warmup}") + + generator = random.Random(args.seed) + pairs = [] + for pair_index in range(args.pairs): + order = ["control", "speculative"] + generator.shuffle(order) + rows: dict[str, dict[str, Any]] = {} + for arm in order: + rows[arm] = ( + run_direct_control( + args, arguments, f"pair-{pair_index}-control" + ) + if arm == "control" + else run_direct_overlap( + args, arguments, f"pair-{pair_index}-speculative" + ) + ) + pairs.append({"pair_index": pair_index, "arm_order": order, **rows}) + print( + json.dumps( + { + "phase": "qualify", + "pair": pair_index + 1, + "control_ms": round(rows["control"]["task_ms"], 3), + "overlap_ms": round(rows["speculative"]["task_ms"], 3), + "speedup": round( + rows["control"]["task_ms"] + / rows["speculative"]["task_ms"], + 3, + ), + }, + sort_keys=True, + ), + flush=True, + ) + + misses = [ + run_direct_miss(args, arguments, f"miss-{index}") + for index in range(args.miss_samples) + ] + controls = [pair["control"] for pair in pairs] + speculative = [pair["speculative"] for pair in pairs] + control_task = statistics.median(row["task_ms"] for row in controls) + speculative_task = statistics.median( + row["task_ms"] for row in speculative + ) + control_model = statistics.median( + row["model"]["model_compute_ms"] for row in controls + ) + speculative_model = statistics.median( + row["model"]["model_compute_ms"] for row in speculative + ) + miss_model = statistics.median( + row["model"]["model_compute_ms"] for row in misses + ) + slowdown_percent = 100.0 * ( + max(speculative_model, miss_model) / control_model - 1.0 + ) + expected_checksum = controls[0]["tool"]["result"]["checksum"] + canonical_model_identity = { + ( + row["model"]["tool_call_sha256"], + row["model"]["assistant_content_sha256"], + row["model"]["completion_tokens"], + ) + for row in controls + speculative + misses + } + all_tools_equal = all( + row["tool"]["result"]["checksum"] == expected_checksum + and row["tool"]["result"]["cpu_affinity"] == args.tool_cpus + for row in controls + speculative + ) and all( + row["authoritative_tool"]["result"]["checksum"] == expected_checksum + for row in misses + ) + ds4_active = all( + row["model"]["accept_rate"] > 0 + for row in controls + speculative + misses + ) + speedup = control_task / speculative_task + checks = { + "disjoint_cpu_affinity": not overlap, + "identical_model_outputs": len(canonical_model_identity) == 1, + "identical_tool_outputs": all_tools_equal, + "ds4_active": ds4_active, + "model_slowdown": slowdown_percent <= args.max_model_slowdown_percent, + "direct_speedup": speedup >= args.min_qualification_speedup, + } + passed = all(checks.values()) + profile = { + "profile_status": "qualified" if passed else "rejected", + "executor": "child_process_cpu_affinity", + "profile_kind": "disjoint_strix_cpu_sparse_compute", + "qualification": { + "host": "lucebox5", + "model_cpu_affinity": model_affinity, + "tool_cpu_affinity": args.tool_cpus, + "checks": checks, + }, + "path_summary": { + "100": { + "accelerator_relation": "non_accelerator", + "decode_interference_qualified": passed, + "hit": { + "control_task_mean_ms": statistics.fmean( + row["task_ms"] for row in controls + ), + "speculative_task_mean_ms": statistics.fmean( + row["task_ms"] for row in speculative + ), + "model_slowdown_percent": slowdown_percent, + }, + "miss": { + "control_task_mean_ms": statistics.fmean( + row["task_ms"] for row in controls + ), + "speculative_task_mean_ms": statistics.fmean( + row["task_ms"] for row in misses + ), + "model_slowdown_percent": slowdown_percent, + }, + } + }, + } + summary = { + "pairs": len(pairs), + "control_task_p50_ms": control_task, + "overlap_task_p50_ms": speculative_task, + "direct_exact_hit_speedup": speedup, + "control_model_compute_p50_ms": control_model, + "overlap_model_compute_p50_ms": speculative_model, + "model_compute_slowdown_percent": slowdown_percent, + "control_tool_wall_p50_ms": statistics.median( + row["tool"]["wall_ms"] for row in controls + ), + "overlap_tool_wall_p50_ms": statistics.median( + row["tool"]["wall_ms"] for row in speculative + ), + "miss_task_p50_ms": statistics.median( + row["task_ms"] for row in misses + ), + "median_accept_rate": statistics.median( + row["model"]["accept_rate"] + for row in controls + speculative + misses + ), + "checks": checks, + "passed": passed, + } + report = { + "phase": "qualification", + "host": "lucebox5", + "config": report_config(args, arguments), + "model_pid": args.model_pid, + "model_cpu_affinity": model_affinity, + "tool_cpu_affinity": args.tool_cpus, + "calibration": calibration, + "summary": summary, + "profile": profile, + "pairs": pairs, + "misses": misses, + } + write_report(args.output, report) + if passed: + write_report(args.profile_output, profile) + print(json.dumps(summary, indent=2, sort_keys=True), flush=True) + if not passed: + raise SystemExit("CPU-lane qualification failed") + + +def native_control( + args: argparse.Namespace, + arguments: dict[str, int], + label: str, +) -> dict[str, Any]: + direct = run_direct_control(args, arguments, label) + tool_result = direct["tool"]["result"] + return { + "mode": "control", + "task_ms": direct["task_ms"], + **direct["model"], + "tool_wall_ms": direct["tool"]["wall_ms"], + "tool_compute_ms": float(tool_result["compute_ms"]), + "tool_checksum": tool_result["checksum"], + "tool_cpu_affinity": tool_result["cpu_affinity"], + } + + +def native_speculative( + args: argparse.Namespace, + arguments: dict[str, int], + *, + prediction: dict[str, int] | None = None, +) -> dict[str, Any]: + row = post_model( + args.url, + arguments, + args.max_tokens, + args.timeout, + prediction=prediction or arguments, + ) + validate_model_call(row, arguments) + metadata = row["speculation"] if isinstance(row["speculation"], dict) else {} + tool_result = metadata.get("result", {}) + return { + "mode": "speculative", + "task_ms": row["request_wall_ms"], + **row, + "tool_wall_ms": float(metadata.get("executor_wall_ms", math.nan)), + "tool_compute_ms": float(tool_result.get("compute_ms", math.nan)), + "tool_checksum": tool_result.get("checksum"), + "tool_cpu_affinity": tool_result.get("cpu_affinity"), + } + + +def native_qwen_control( + args: argparse.Namespace, + arguments: dict[str, int], + label: str, +) -> dict[str, Any]: + started = time.perf_counter() + model = post_model( + args.url, + arguments, + args.max_tokens, + args.timeout, + automatic_prediction=False, + tool_choice="required", + ) + tool = run_executor( + args.binary, + arguments, + args.tool_cpus, + args.timeout, + f"{label}-tool", + ) + validate_model_call(model, arguments) + result = tool["result"] + return { + "mode": "control", + "task_ms": (time.perf_counter() - started) * 1000.0, + **model, + "tool_wall_ms": float(tool["wall_ms"]), + "tool_compute_ms": float(result["compute_ms"]), + "tool_checksum": result["checksum"], + "tool_cpu_affinity": result["cpu_affinity"], + "prediction_hit": False, + "predictor_wall_ms": 0.0, + } + + +def native_qwen_speculative( + args: argparse.Namespace, + arguments: dict[str, int], + label: str, +) -> dict[str, Any]: + started = time.perf_counter() + model = post_model( + args.url, + arguments, + args.max_tokens, + args.timeout, + automatic_prediction=True, + tool_choice="required", + ) + validate_model_call(model, arguments) + metadata = model["speculation"] if isinstance( + model["speculation"], dict + ) else {} + prediction_hit = metadata.get("status") == "hit" + if prediction_hit: + tool_result = metadata.get("result") + if not isinstance(tool_result, dict): + raise RuntimeError("automatic hit did not expose a tool result") + tool_wall_ms = float(metadata.get("executor_wall_ms", math.nan)) + else: + # This is the real miss path: discard the private speculative result, + # then execute the authoritative model call normally. + fallback = run_executor( + args.binary, + arguments, + args.tool_cpus, + args.timeout, + f"{label}-fallback", + ) + tool_result = fallback["result"] + tool_wall_ms = float(fallback["wall_ms"]) + return { + "mode": "qwen_speculative", + "task_ms": (time.perf_counter() - started) * 1000.0, + **model, + "tool_wall_ms": tool_wall_ms, + "tool_compute_ms": float(tool_result["compute_ms"]), + "tool_checksum": tool_result["checksum"], + "tool_cpu_affinity": tool_result["cpu_affinity"], + "prediction_hit": prediction_hit, + "predictor_wall_ms": float(metadata.get("predictor_wall_ms", 0.0)), + "prediction_source": metadata.get("prediction_source"), + "prediction_status": metadata.get("status"), + "prediction_reason": metadata.get("reason"), + } + + +def summarize_native( + pairs: list[dict[str, Any]], resamples: int, seed: int +) -> dict[str, Any]: + controls = [pair["control"] for pair in pairs] + speculative = [pair["speculative"] for pair in pairs] + control_task = statistics.median(row["task_ms"] for row in controls) + speculative_task = statistics.median( + row["task_ms"] for row in speculative + ) + control_model = statistics.median( + row["model_compute_ms"] for row in controls + ) + speculative_model = statistics.median( + row["model_compute_ms"] for row in speculative + ) + control_tool = statistics.median( + row["tool_compute_ms"] for row in controls + ) + speculative_tool = statistics.median( + row["tool_compute_ms"] for row in speculative + ) + paired_speedups = [ + float(pair["control"]["task_ms"]) + / float(pair["speculative"]["task_ms"]) + for pair in pairs + ] + return { + "pairs": len(pairs), + "control_task_p50_ms": control_task, + "control_task_p95_ms": percentile( + (row["task_ms"] for row in controls), 0.95 + ), + "control_task_max_ms": max(row["task_ms"] for row in controls), + "speculative_task_p50_ms": speculative_task, + "speculative_task_p95_ms": percentile( + (row["task_ms"] for row in speculative), 0.95 + ), + "speculative_task_max_ms": max( + row["task_ms"] for row in speculative + ), + "exact_hit_speedup": control_task / speculative_task, + "paired_speedup_p05": percentile(paired_speedups, 0.05), + "paired_speedup_min": min(paired_speedups), + "exact_hit_speedup_bootstrap_95ci": bootstrap_speedup_ci( + pairs, resamples, seed + ), + "task_latency_reduction_percent": 100.0 + * (control_task - speculative_task) + / control_task, + "control_model_compute_p50_ms": control_model, + "speculative_model_compute_p50_ms": speculative_model, + "model_compute_slowdown_percent": 100.0 + * (speculative_model / control_model - 1.0), + "control_tool_compute_p50_ms": control_tool, + "speculative_tool_compute_p50_ms": speculative_tool, + "tool_compute_slowdown_percent": 100.0 + * (speculative_tool / control_tool - 1.0), + "latency_match_ratio": min(control_model, control_tool) + / max(control_model, control_tool), + "ideal_zero_interference_speedup_ceiling": ( + control_model + control_tool + ) + / max(control_model, control_tool), + "median_decode_tokens_per_sec": statistics.median( + row["decode_tokens_per_sec"] for row in controls + speculative + ), + "median_accept_rate": statistics.median( + row["accept_rate"] for row in controls + speculative + ), + "native_hits": sum( + isinstance(row["speculation"], dict) + and row["speculation"].get("status") == "hit" + for row in speculative + ), + "all_calls_identical": all( + pair["control"]["tool_call_sha256"] + == pair["speculative"]["tool_call_sha256"] + for pair in pairs + ), + "all_model_outputs_identical": all( + pair["control"]["assistant_content_sha256"] + == pair["speculative"]["assistant_content_sha256"] + and pair["control"]["completion_tokens"] + == pair["speculative"]["completion_tokens"] + for pair in pairs + ), + "all_tool_outputs_equivalent": all( + pair["control"]["tool_checksum"] + == pair["speculative"]["tool_checksum"] + and pair["control"]["tool_cpu_affinity"] + == pair["speculative"]["tool_cpu_affinity"] + for pair in pairs + ), + } + + +def require_qualified_cpu_tool_props( + props: dict[str, Any], args: argparse.Namespace, *, automatic: bool = False +) -> dict[str, Any]: + tool_props = props.get("tool_speculation") + if not isinstance(tool_props, dict) or not tool_props.get("enabled"): + raise SystemExit("server tool speculation is not enabled") + if automatic and not tool_props.get("automatic_prediction_enabled"): + raise SystemExit("server automatic Qwen prediction is not enabled") + expected_props = { + "execution_mode": "child_process_cpu_affinity", + "profile_status": "qualified", + "compute_isolation": "disjoint_cpu_affinity", + "cpu_affinity_isolated": True, + "preserves_token_speculation": True, + } + for key, expected in expected_props.items(): + if tool_props.get(key) != expected: + raise SystemExit( + f"server tool_speculation.{key}={tool_props.get(key)!r}, " + f"expected {expected!r}" + ) + if tool_props.get("tool_cpu_affinity") != args.tool_cpus: + raise SystemExit("server tool CPU affinity differs from benchmark") + model_affinity = tool_props.get("model_cpu_affinity") + if not isinstance(model_affinity, list) or set(model_affinity) & set( + args.tool_cpus + ): + raise SystemExit("server model/tool CPU affinity is not disjoint") + return tool_props + + +def native(args: argparse.Namespace) -> None: + props = get_json(props_url(args.url), args.timeout) + tool_props = require_qualified_cpu_tool_props(props, args) + + arguments = expected_arguments( + args.rows, + args.nonzeros_per_row, + args.iterations, + args.threads, + args.tool_seed, + ) + for warmup in range(args.warmups): + native_control(args, arguments, f"native-warm-control-{warmup}") + row = native_speculative(args, arguments) + if (row["speculation"] or {}).get("status") != "hit": + raise RuntimeError("native speculative warmup did not commit") + + generator = random.Random(args.seed) + pairs = [] + for pair_index in range(args.pairs): + order = ["control", "speculative"] + generator.shuffle(order) + rows: dict[str, dict[str, Any]] = {} + for arm in order: + rows[arm] = ( + native_control( + args, arguments, f"native-pair-{pair_index}-control" + ) + if arm == "control" + else native_speculative(args, arguments) + ) + pairs.append({"pair_index": pair_index, "arm_order": order, **rows}) + print( + json.dumps( + { + "phase": "native", + "pair": pair_index + 1, + "control_ms": round(rows["control"]["task_ms"], 3), + "speculative_ms": round( + rows["speculative"]["task_ms"], 3 + ), + "speedup": round( + rows["control"]["task_ms"] + / rows["speculative"]["task_ms"], + 3, + ), + "status": (rows["speculative"]["speculation"] or {}).get( + "status" + ), + }, + sort_keys=True, + ), + flush=True, + ) + + wrong = dict(arguments) + wrong["iterations"] = max(1, arguments["iterations"] - 1) + if wrong["iterations"] == arguments["iterations"]: + wrong["iterations"] += 1 + miss = native_speculative(args, arguments, prediction=wrong) + miss_metadata = miss["speculation"] or {} + miss_check = { + "passed": miss_metadata.get("status") == "miss" + and miss_metadata.get("reason") == "invocation_mismatch" + and "result" not in miss_metadata + and all( + miss["assistant_content_sha256"] + == pair["control"]["assistant_content_sha256"] + and miss["completion_tokens"] + == pair["control"]["completion_tokens"] + for pair in pairs + ), + "status": miss_metadata.get("status"), + "reason": miss_metadata.get("reason"), + "private_result_exposed": "result" in miss_metadata, + } + summary = summarize_native(pairs, args.bootstrap_resamples, args.seed) + correctness_passed = ( + summary["native_hits"] == args.pairs + and summary["all_calls_identical"] + and summary["all_model_outputs_identical"] + and summary["all_tool_outputs_equivalent"] + and miss_check["passed"] + and all( + pair[arm]["accept_rate"] > 0 + for pair in pairs + for arm in ("control", "speculative") + ) + ) + ci_low = summary["exact_hit_speedup_bootstrap_95ci"][0] + checks = { + "correctness": correctness_passed, + "strong_sequential_baseline": True, + "exact_hit_speedup": summary["exact_hit_speedup"] >= args.min_speedup, + "speedup_ci_low": ci_low >= args.min_speedup_ci_low, + "model_slowdown": summary["model_compute_slowdown_percent"] + <= args.max_model_slowdown_percent, + } + production_gate = { + "passed": all(checks.values()), + "checks": checks, + "thresholds": { + "min_exact_hit_speedup": args.min_speedup, + "min_speedup_ci_low": args.min_speedup_ci_low, + "max_model_slowdown_percent": args.max_model_slowdown_percent, + }, + } + report = { + "phase": "native_engine", + "host": "lucebox5", + "config": report_config(args, arguments), + "server_snapshot": { + "runtime": props.get("runtime"), + "speculative": props.get("speculative"), + "tool_speculation": tool_props, + }, + "methodology": { + "control": "model request followed by the identical CPU-pinned sparse tool", + "speculative": "engine starts the identical CPU-pinned sparse tool before DS4 generation", + "commit": "result exposed only after exact canonical call match", + "pairing": "randomized arm order within every warm pair", + }, + "correctness_passed": correctness_passed, + "production_gate": production_gate, + "miss_check": miss_check, + "summary": summary, + "pairs": pairs, + } + write_report(args.output, report) + print( + json.dumps( + { + "correctness_passed": correctness_passed, + "production_gate": production_gate, + "miss_check": miss_check, + "summary": summary, + }, + indent=2, + sort_keys=True, + ), + flush=True, + ) + if not production_gate["passed"]: + raise SystemExit("native CPU tool-speculation production gate failed") + + +def native_qwen(args: argparse.Namespace) -> None: + props = get_json(props_url(args.url), args.timeout) + tool_props = require_qualified_cpu_tool_props( + props, args, automatic=True + ) + + arguments = expected_arguments( + args.rows, + args.nonzeros_per_row, + args.iterations, + args.threads, + args.tool_seed, + ) + for warmup in range(args.warmups): + native_qwen_control( + args, arguments, f"qwen-warm-{warmup}-control" + ) + native_qwen_speculative( + args, arguments, f"qwen-warm-{warmup}-speculative" + ) + + generator = random.Random(args.seed) + pairs: list[dict[str, Any]] = [] + for pair_index in range(args.pairs): + order = ["control", "speculative"] + generator.shuffle(order) + rows: dict[str, dict[str, Any]] = {} + for arm in order: + rows[arm] = ( + native_qwen_control( + args, arguments, f"qwen-{pair_index}-control" + ) + if arm == "control" + else native_qwen_speculative( + args, arguments, f"qwen-{pair_index}-speculative" + ) + ) + pairs.append({"pair_index": pair_index, "arm_order": order, **rows}) + print(json.dumps({ + "phase": "native-qwen", + "pair": pair_index + 1, + "control_ms": round(rows["control"]["task_ms"], 3), + "speculative_ms": round(rows["speculative"]["task_ms"], 3), + "speedup": round( + rows["control"]["task_ms"] / + rows["speculative"]["task_ms"], 3), + "prediction_status": rows["speculative"]["prediction_status"], + "predictor_ms": round( + rows["speculative"]["predictor_wall_ms"], 3), + }, sort_keys=True), flush=True) + + summary = summarize_native(pairs, args.bootstrap_resamples, args.seed) + speculative = [pair["speculative"] for pair in pairs] + hits = sum(row["prediction_hit"] for row in speculative) + summary.update({ + "qwen_prediction_hits": hits, + "qwen_prediction_hit_rate": hits / len(speculative), + "qwen_predictor_p50_ms": statistics.median( + row["predictor_wall_ms"] for row in speculative + ), + "qwen_predictor_p95_ms": percentile( + (row["predictor_wall_ms"] for row in speculative), 0.95 + ), + "qwen_prediction_source_valid": all( + row["prediction_source"] == NATIVE_PREDICTION_SOURCE + for row in speculative + ), + }) + checks = { + "all_model_outputs_identical": summary["all_model_outputs_identical"], + "all_calls_identical": summary["all_calls_identical"], + "all_tool_outputs_equivalent": summary["all_tool_outputs_equivalent"], + "ds4_active": summary["median_accept_rate"] > 0, + "prediction_source": summary["qwen_prediction_source_valid"], + "prediction_hit_rate": + summary["qwen_prediction_hit_rate"] >= args.min_prediction_hit_rate, + "speedup": summary["exact_hit_speedup"] >= args.min_speedup, + "speedup_ci_low": + summary["exact_hit_speedup_bootstrap_95ci"][0] >= + args.min_speedup_ci_low, + "speedup_p05": + summary["paired_speedup_p05"] >= args.min_speedup_p05, + "model_slowdown": summary["model_compute_slowdown_percent"] <= + args.max_model_slowdown_percent, + } + report = { + "phase": "native_qwen_engine", + "host": "lucebox5", + "config": report_config(args, arguments), + "methodology": { + "control": "DS4 generation, then the authoritative CPU-pinned tool", + "speculative": "Qwen3-0.6B predicts the call; the engine launches the private CPU-pinned tool before DS4 finishes", + "prediction_cost": "included in speculative request wall time", + "miss_cost": "included by running the authoritative tool after every miss", + "commit": "exact canonical function name and arguments", + "semantic_token_injection": False, + }, + "server_snapshot": {"tool_speculation": tool_props}, + "production_gate": { + "passed": all(checks.values()), + "checks": checks, + "thresholds": { + "min_speedup": args.min_speedup, + "min_speedup_ci_low": args.min_speedup_ci_low, + "min_speedup_p05": args.min_speedup_p05, + "min_prediction_hit_rate": args.min_prediction_hit_rate, + "max_model_slowdown_percent": + args.max_model_slowdown_percent, + }, + }, + "summary": summary, + "pairs": pairs, + } + write_report(args.output, report) + print(json.dumps({ + "production_gate": report["production_gate"], + "summary": summary, + }, indent=2, sort_keys=True), flush=True) + if not report["production_gate"]["passed"]: + raise SystemExit("automatic Qwen tool-speculation gate failed") + + +def report_config( + args: argparse.Namespace, arguments: dict[str, int] +) -> dict[str, Any]: + return { + "url": args.url, + "binary": str(args.binary.resolve()), + "tool_arguments": arguments, + "fixed_sparse_shape": { + "rows": args.rows, + "nonzeros_per_row": args.nonzeros_per_row, + "threads": args.threads, + "seed": args.tool_seed, + }, + "tool_cpus": args.tool_cpus, + "max_tokens": args.max_tokens, + "pairs": args.pairs, + "warmups": args.warmups, + "seed": args.seed, + } + + +def write_report(path: Path | None, value: dict[str, Any]) -> None: + if path is None: + return + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps(value, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + +def add_common_arguments(parser: argparse.ArgumentParser) -> None: + parser.add_argument( + "--url", default="http://127.0.0.1:18145/v1/chat/completions" + ) + parser.add_argument("--binary", type=Path, required=True) + parser.add_argument("--tool-cpus", type=parse_cpu_list, required=True) + parser.add_argument("--rows", type=int, default=4096) + parser.add_argument("--nonzeros-per-row", type=int, default=16) + parser.add_argument("--threads", type=int, default=2) + parser.add_argument("--tool-seed", type=int, default=731) + parser.add_argument("--max-tokens", type=int, default=32) + parser.add_argument("--pairs", type=int, default=20) + parser.add_argument("--warmups", type=int, default=2) + parser.add_argument("--timeout", type=float, default=180.0) + parser.add_argument("--seed", type=int, default=814) + parser.add_argument("--max-model-slowdown-percent", type=float, default=5.0) + parser.add_argument("--output", type=Path) + + +def main() -> None: + parser = argparse.ArgumentParser() + subparsers = parser.add_subparsers(dest="phase", required=True) + + qualify_parser = subparsers.add_parser("qualify") + add_common_arguments(qualify_parser) + qualify_parser.add_argument("--model-pid", type=int, required=True) + qualify_parser.add_argument("--initial-iterations", type=int, default=20) + qualify_parser.add_argument("--calibration-steps", type=int, default=5) + qualify_parser.add_argument("--calibration-model-samples", type=int, default=3) + qualify_parser.add_argument("--calibration-tool-samples", type=int, default=2) + qualify_parser.add_argument("--miss-samples", type=int, default=5) + qualify_parser.add_argument("--min-qualification-speedup", type=float, default=1.70) + qualify_parser.add_argument("--profile-output", type=Path, required=True) + + native_parser = subparsers.add_parser("native") + add_common_arguments(native_parser) + native_parser.add_argument("--iterations", type=int, required=True) + native_parser.add_argument("--bootstrap-resamples", type=int, default=20_000) + native_parser.add_argument("--min-speedup", type=float, default=1.80) + native_parser.add_argument("--min-speedup-ci-low", type=float, default=1.70) + + qwen_parser = subparsers.add_parser("native-qwen") + add_common_arguments(qwen_parser) + qwen_parser.add_argument("--iterations", type=int, required=True) + qwen_parser.add_argument("--bootstrap-resamples", type=int, default=20_000) + qwen_parser.add_argument("--min-speedup", type=float, default=1.60) + qwen_parser.add_argument("--min-speedup-ci-low", type=float, default=1.50) + qwen_parser.add_argument("--min-speedup-p05", type=float, default=1.25) + qwen_parser.add_argument("--min-prediction-hit-rate", type=float, default=0.90) + + args = parser.parse_args() + if not args.binary.is_file(): + parser.error(f"executor binary does not exist: {args.binary}") + positive = [ + args.rows, + args.nonzeros_per_row, + args.threads, + args.max_tokens, + args.pairs, + args.warmups, + args.timeout, + ] + if any(value <= 0 for value in positive): + parser.error("counts and timeouts must be positive") + if args.tool_seed < 0 or args.max_model_slowdown_percent < 0: + parser.error("seed and slowdown threshold must be non-negative") + if args.phase == "qualify": + if ( + args.model_pid <= 0 + or args.initial_iterations <= 0 + or args.calibration_steps <= 0 + or args.calibration_model_samples <= 0 + or args.calibration_tool_samples <= 0 + or args.miss_samples <= 0 + or args.min_qualification_speedup <= 1.0 + ): + parser.error("qualification settings must be positive") + qualify(args) + else: + if ( + args.iterations <= 0 + or args.bootstrap_resamples <= 0 + or args.min_speedup <= 1.0 + or args.min_speedup_ci_low <= 1.0 + or args.min_speedup_ci_low > args.min_speedup + ): + parser.error("native benchmark settings are invalid") + if args.phase == "native-qwen": + if ( + not 0.0 <= args.min_prediction_hit_rate <= 1.0 + or args.min_speedup_p05 <= 1.0 + or args.min_speedup_p05 > args.min_speedup + ): + parser.error("native Qwen benchmark settings are invalid") + native_qwen(args) + else: + native(args) + + +if __name__ == "__main__": + main() diff --git a/optimizations/ooo_spec_lucebox5_cpu/benchmark_trace_compiled_workflows.py b/optimizations/ooo_spec_lucebox5_cpu/benchmark_trace_compiled_workflows.py new file mode 100644 index 000000000..84837f014 --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/benchmark_trace_compiled_workflows.py @@ -0,0 +1,2083 @@ +#!/usr/bin/env python3 +"""Benchmark no-training trace-compiled tool workflows on Lucebox5. + +The benchmark compares three end-to-end agent paths over 10--20 real model +tool calls: + +* ``stage_batched``: DS4 authorizes one batch per dependency stage and every + call in that stage executes concurrently. This gives the control parallel + tools without speculating or compiling the complete workflow. +* ``compiled``: a recurring, side-effect-free trace is exposed as one generated + macro tool; DS4 authorizes it once and independent branches run in parallel. +* ``speculative``: the PR's Qwen predictor proposes the macro call and the + engine starts its compiled graph before DS4 finishes. The private result is + committed only after an exact name-and-arguments match. + +The workflow compiler is learned from prior successful traces. It performs no +model training and refuses literals, side effects, ambiguous dataflow, or +inconsistent control flow. Every arm uses the production DS4+DSpark endpoint, +includes all model turns and tools in wall time, and must produce the same +underlying calls, tool results, and exact final answer. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import os +import random +import re +import statistics +import subprocess +import time +from concurrent.futures import Future, ThreadPoolExecutor +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from benchmark_cpu_tool_speculation import ( + NATIVE_PREDICTION_SOURCE, + get_json, + normalize_tool_call, + parse_cpu_list, + percentile, + post_json, + props_url, +) +from bfcl_replay_tool_executor import ( + PROTOCOL, + call_ref, + call_sha256, + canonical_call, +) + +RESULTS_DIR = Path(__file__).with_name("results") +CANONICAL_TRAINING_REPORT = ( + RESULTS_DIR / "multiturn-cached-wordref-production-6tasks.json" +) +CANONICAL_WORKFLOW_REGISTRY = RESULTS_DIR / "trace-workflow-registry.json" + + +@dataclass(frozen=True) +class ArgumentBinding: + source: str + key: str + + +@dataclass(frozen=True) +class PatternStep: + tool: str + arguments: tuple[tuple[str, ArgumentBinding], ...] + + +@dataclass(frozen=True) +class CompiledPattern: + steps: tuple[PatternStep, ...] + root_fields: tuple[str, ...] + training_traces: int + + @property + def macro_name(self) -> str: + # Semantic names are materially more reliable for tool-call generation + # than opaque hashes. The registry remains keyed by the full pattern + # fingerprint, so this user-facing alias does not define identity. + subject = self.steps[0].tool.rsplit("_", 1)[-1] + return f"execute_{subject}_workflows" + + @property + def fingerprint(self) -> str: + payload = json.dumps( + [ + [ + step.tool, + [[name, binding.source, binding.key] for name, binding in step.arguments], + ] + for step in self.steps + ], + separators=(",", ":"), + ) + return hashlib.sha256(payload.encode()).hexdigest() + + def instantiate( + self, root: dict[str, str], previous_result: dict[str, Any] | None, index: int + ) -> dict[str, Any]: + step = self.steps[index] + arguments: dict[str, Any] = {} + for name, binding in step.arguments: + source: dict[str, Any] + if binding.source == "root": + source = root + elif binding.source == "previous_result" and previous_result is not None: + source = previous_result + else: + raise RuntimeError( + f"cannot resolve {binding.source}.{binding.key} for {step.tool}" + ) + if binding.key not in source: + raise RuntimeError( + f"missing {binding.source}.{binding.key} for {step.tool}" + ) + arguments[name] = source[binding.key] + return {"name": step.tool, "arguments": arguments} + + def simulate(self, root: dict[str, str]) -> list[dict[str, Any]]: + calls = [] + previous: dict[str, Any] | None = None + for index in range(len(self.steps)): + call = self.instantiate(root, previous, index) + calls.append(call) + previous = simulated_tool_result(call) + return calls + + def macro_tool( + self, max_items: int, workflow_ref: str | None = None + ) -> dict[str, Any]: + if workflow_ref is not None: + parameters = { + "type": "object", + "properties": { + "workflow_ref": { + "type": "string", + "enum": [workflow_ref], + "description": "Bound workflow instance for this request.", + } + }, + "required": ["workflow_ref"], + "additionalProperties": False, + } + else: + properties = {field: {"type": "string"} for field in self.root_fields} + parameters = { + "type": "object", + "properties": { + "customers": { + "type": "array", + "items": { + "type": "object", + "properties": properties, + "required": list(self.root_fields), + "additionalProperties": False, + }, + "minItems": 1, + "maxItems": max_items, + } + }, + "required": ["customers"], + "additionalProperties": False, + } + return { + "type": "function", + "function": { + "name": self.macro_name, + "description": ( + "Execute the validated five-step customer workflow independently " + "for every requested customer." + ), + "parameters": parameters, + }, + } + + +def simulated_tool_result(call: dict[str, Any]) -> dict[str, Any]: + return { + "call_ref": call_ref(call), + "call_sha256": call_sha256(call), + "tool_name": call["name"], + "side_effects": False, + } + + +def _infer_binding( + value: Any, + root: dict[str, str], + previous_result: dict[str, Any] | None, +) -> ArgumentBinding: + previous_matches = [] + if previous_result is not None: + previous_matches = [key for key, candidate in previous_result.items() if candidate == value] + root_matches = [key for key, candidate in root.items() if candidate == value] + if len(previous_matches) == 1: + return ArgumentBinding("previous_result", previous_matches[0]) + if len(root_matches) == 1: + return ArgumentBinding("root", root_matches[0]) + raise ValueError("argument is literal or has ambiguous trace dataflow") + + +def mine_pattern(traces: list[dict[str, Any]]) -> CompiledPattern: + if len(traces) < 2: + raise ValueError("at least two successful traces are required") + signatures = [] + root_fields: set[str] = set() + for trace in traces: + root = trace.get("root") + calls = trace.get("calls") + results = trace.get("results") + if not isinstance(root, dict) or not isinstance(calls, list) or not isinstance(results, list): + raise ValueError("trace must contain root, calls, and results") + if not calls or len(calls) != len(results): + raise ValueError("trace calls and results must be non-empty and aligned") + signature = [] + previous: dict[str, Any] | None = None + for call, result in zip(calls, results, strict=True): + if not isinstance(call, dict) or not isinstance(call.get("arguments"), dict): + raise ValueError("trace call is malformed") + if not isinstance(result, dict) or result.get("side_effects") is not False: + raise ValueError("only explicitly side-effect-free traces can be compiled") + if result.get("call_sha256") != call_sha256(call): + raise ValueError("trace result does not match its call") + bindings = [] + for name, value in sorted(call["arguments"].items()): + binding = _infer_binding(value, root, previous) + if binding.source == "root": + root_fields.add(binding.key) + bindings.append((name, binding)) + signature.append(PatternStep(str(call.get("name", "")), tuple(bindings))) + previous = result + signatures.append(tuple(signature)) + if any(signature != signatures[0] for signature in signatures[1:]): + raise ValueError("training traces do not share one control/data-flow pattern") + if not root_fields: + raise ValueError("compiled workflow exposes no request-bound arguments") + return CompiledPattern( + steps=signatures[0], + root_fields=tuple(sorted(root_fields)), + training_traces=len(traces), + ) + + +def load_training_traces(path: Path, required_steps: int) -> list[dict[str, Any]]: + report = json.loads(path.read_text(encoding="utf-8")) + compact_traces = report.get("traces") if isinstance(report, dict) else None + if isinstance(compact_traces, list): + traces = [ + trace + for trace in compact_traces + if isinstance(trace, dict) + and isinstance(trace.get("calls"), list) + and len(trace["calls"]) == required_steps + ] + if len(traces) < 2: + raise ValueError( + "training trace file contains fewer than two complete traces" + ) + return traces + pairs = report.get("pairs") if isinstance(report, dict) else None + if not isinstance(pairs, list): + raise ValueError("training report has no pairs") + traces = [] + for pair in pairs: + task = pair.get("task") if isinstance(pair, dict) else None + control = pair.get("control") if isinstance(pair, dict) else None + steps = control.get("steps") if isinstance(control, dict) else None + if ( + not isinstance(task, dict) + or not isinstance(steps, list) + or len(steps) != required_steps + or not control.get("all_calls_correct") + ): + continue + traces.append( + { + "root": { + "customer_email": task["customer_email"], + "destination": task["destination"], + }, + "calls": [step["call"] for step in steps], + "results": [step["tool_result"] for step in steps], + } + ) + if len(traces) < 2: + raise ValueError("training report contains fewer than two complete correct traces") + return traces + + +def file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def alphabetic_identifier(value: int) -> str: + """Encode an integer with letters so copy accuracy is not biased by 0/o.""" + prefix = "warm" if value < 0 else "task" + number = abs(value) + encoded = "" + while True: + encoded = chr(ord("a") + number % 26) + encoded + number = number // 26 - 1 + if number < 0: + break + return prefix + encoded + + +def make_task( + index: int, branch_count: int, pattern: CompiledPattern +) -> dict[str, Any]: + destinations = ("Rome", "Milan", "Turin", "Bologna", "Florence", "Naples") + items = [] + used_refs = [set() for _ in pattern.steps] + candidate = max(index, 0) * 100 + while len(items) < branch_count and candidate < max(index, 0) * 100 + 20_000: + root = { + "customer_email": ( + f"agent-{alphabetic_identifier(index)}-" + f"{alphabetic_identifier(candidate)}@example.test" + ), + "destination": destinations[(index + candidate) % len(destinations)], + } + refs = [call_ref(call) for call in pattern.simulate(root)] + if all(reference not in used_refs[step] for step, reference in enumerate(refs)): + items.append(root) + for step, reference in enumerate(refs): + used_refs[step].add(reference) + candidate += 1 + if len(items) != branch_count: + raise RuntimeError("could not construct collision-free workflow branches") + return { + "id": f"trace_compiled_{index:03d}", + "workflow_ref": f"workflow_{alphabetic_identifier(index)}", + "items": items, + "branch_count": branch_count, + "call_count": branch_count * len(pattern.steps), + } + + +def request_content(task: dict[str, Any]) -> str: + rendered = "; ".join( + f"{item['customer_email']} to {item['destination']}" for item in task["items"] + ) + return f"Customers: {rendered}." + + +def workflow_reference(task: dict[str, Any], pattern: CompiledPattern) -> str: + del pattern + workflow_ref = task.get("workflow_ref") + if not isinstance(workflow_ref, str) or re.fullmatch( + r"workflow_[a-z]+", workflow_ref + ) is None: + raise ValueError("task has no valid request-scoped workflow_ref") + return workflow_ref + + +def write_workflow_registry( + path: Path, pattern: CompiledPattern, tasks: list[dict[str, Any]] +) -> None: + workflows = { + workflow_reference(task, pattern): { + "pattern_fingerprint": pattern.fingerprint, + "items": task["items"], + } + for task in tasks + } + if len(workflows) != len(tasks): + raise ValueError("workflow_ref collision in request-scoped registry") + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps( + { + "schema_version": 1, + "pattern_fingerprint": pattern.fingerprint, + "workflows": workflows, + }, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def parse_request_customers(content: str) -> list[dict[str, str]]: + prefix = "Customers: " + if not content.startswith(prefix) or not content.endswith("."): + raise ValueError("request does not use the validated customer-list format") + customers = [] + for entry in content[len(prefix) : -1].split("; "): + match = re.fullmatch(r"([^\s;]+) to ([A-Za-z][A-Za-z -]*)", entry) + if match is None: + raise ValueError("request customer entry is malformed") + customers.append( + {"customer_email": match.group(1), "destination": match.group(2)} + ) + if not customers: + raise ValueError("request contains no customers") + return customers + + +def stage_batch_tool( + pattern: CompiledPattern, + step_index: int, + max_items: int, + stage_ref: str | None = None, +) -> dict[str, Any]: + step = pattern.steps[step_index] + if stage_ref is not None: + parameters = { + "type": "object", + "properties": { + "stage_ref": { + "type": "string", + "enum": [stage_ref], + "description": "Bound batch for this workflow stage.", + } + }, + "required": ["stage_ref"], + "additionalProperties": False, + } + else: + properties = {name: {"type": "string"} for name, _ in step.arguments} + parameters = { + "type": "object", + "properties": { + "calls": { + "type": "array", + "items": { + "type": "object", + "properties": properties, + "required": list(properties), + "additionalProperties": False, + }, + "minItems": 1, + "maxItems": max_items, + } + }, + "required": ["calls"], + "additionalProperties": False, + } + return { + "type": "function", + "function": { + "name": f"batch_{step.tool}", + "description": ( + f"Run {step.tool} once for every item. Calls execute concurrently " + "and results preserve input order." + ), + "parameters": parameters, + }, + } + + +def stage_batched_messages( + task: dict[str, Any], pattern: CompiledPattern +) -> list[dict[str, Any]]: + del pattern + system = ( + "The scheduler exposes exactly one currently-ready batch tool at a time. " + "Call only that declared tool once and copy its bound stage_ref exactly. " + "The bound batch contains every customer in input order. Do not invent or " + "name future tools, and emit no prose." + ) + return [ + {"role": "system", "content": system}, + {"role": "user", "content": request_content(task)}, + ] + + +def macro_messages(task: dict[str, Any], pattern: CompiledPattern) -> list[dict[str, Any]]: + return [ + { + "role": "system", + "content": ( + "Use the workflow tool exactly once for every requested customer. " + "Copy its bound workflow_ref exactly. No prose." + ), + }, + {"role": "user", "content": request_content(task)}, + ] + + +def stage_reference(task: dict[str, Any], pattern: CompiledPattern, index: int) -> str: + stage_names = ("one", "two", "three", "four", "five") + if not 0 <= index < len(stage_names): + raise ValueError("stage index is outside the compiled workflow") + return f"{workflow_reference(task, pattern)}_stage_{stage_names[index]}" + + +def model_observation(result: dict[str, Any], wall_ms: float) -> dict[str, Any]: + choices = result.get("choices") + message = choices[0].get("message") if isinstance(choices, list) and choices else None + if not isinstance(message, dict): + raise RuntimeError("model response has no assistant message") + raw_calls = message.get("tool_calls") + if not isinstance(raw_calls, list): + raw_calls = [] + calls = [] + for raw in raw_calls: + function = raw.get("function") if isinstance(raw, dict) else None + if not isinstance(function, dict): + raise RuntimeError("model emitted a malformed tool call") + arguments = function.get("arguments") + if isinstance(arguments, str): + arguments = json.loads(arguments) + if not isinstance(function.get("name"), str) or not isinstance(arguments, dict): + raise RuntimeError("model emitted invalid tool name or arguments") + call_id = raw.get("id") + if not isinstance(call_id, str) or not call_id: + raise RuntimeError("model tool call has no id") + calls.append( + { + "id": call_id, + "call": {"name": function["name"], "arguments": arguments}, + } + ) + content = message.get("content") + if not isinstance(content, str): + content = "" + usage = result.get("usage") if isinstance(result.get("usage"), dict) else {} + timings = usage.get("timings") if isinstance(usage.get("timings"), dict) else {} + assistant_message = dict(message) + assistant_message["role"] = "assistant" + assistant_message["content"] = content + content_format_call = False + if not calls and content: + parsed_call = normalize_tool_call(result) + if parsed_call is not None: + content_format_call = True + call_id = "call_content_" + hashlib.sha256( + canonical_call(parsed_call).encode() + ).hexdigest()[:16] + calls.append({"id": call_id, "call": parsed_call}) + assistant_message = { + "role": "assistant", + "content": "", + "tool_calls": [ + { + "id": call_id, + "type": "function", + "function": { + "name": parsed_call["name"], + "arguments": json.dumps( + parsed_call["arguments"], + sort_keys=True, + separators=(",", ":"), + ), + }, + } + ], + } + return { + "request_wall_ms": wall_ms, + "model_compute_ms": float(timings.get("prefill_ms", 0.0)) + + float(timings.get("decode_ms", 0.0)), + "prefill_ms": float(timings.get("prefill_ms", 0.0)), + "decode_ms": float(timings.get("decode_ms", 0.0)), + "decode_tokens_per_sec": float(timings.get("decode_tokens_per_sec", 0.0)), + "cache_hit": bool(timings.get("cache_hit", False)), + "cached_prefix_tokens": int(timings.get("cached_prefix_tokens", 0)), + "completion_tokens": int(usage.get("completion_tokens", 0)), + "accept_rate": float(usage.get("accept_rate", 0.0)), + "content": content, + "content_sha256": hashlib.sha256(content.encode()).hexdigest(), + "assistant_message": assistant_message, + "calls": calls, + "content_format_call": content_format_call, + } + + +def post_turn( + args: argparse.Namespace, + messages: list[dict[str, Any]], + tools: list[dict[str, Any]], + tool_choice: Any, + max_tokens: int, + *, + automatic_tool_speculation: bool = False, + tool_speculation: dict[str, Any] | None = None, +) -> dict[str, Any]: + request = { + "model": "dflash", + "messages": messages, + "tools": tools, + "tool_choice": tool_choice, + "temperature": 0, + "seed": args.seed, + "max_tokens": max_tokens, + "stream": False, + "automatic_tool_speculation": automatic_tool_speculation, + } + if tool_speculation is not None: + request["tool_speculation"] = tool_speculation + result, wall_ms = post_json( + args.url, + request, + args.timeout, + ) + observation = model_observation(result, wall_ms) + observation["speculation"] = result.get("dflash_tool_speculation") + return observation + + +def execute_tool_safe( + binary: Path, + call: dict[str, Any], + cpus: list[int], + timeout: float, + request_id: str, +) -> dict[str, Any]: + request = { + "protocol": PROTOCOL, + "request_id": request_id, + "resource_percentage": 100, + "accelerator_relation": "non_accelerator", + "cpu_affinity": cpus, + "cpu_affinity_isolated": True, + "call": call, + } + command = [str(binary), "--dflash-tool-spec-v1"] + if os.name == "posix" and Path("/usr/bin/taskset").is_file(): + command = ["/usr/bin/taskset", "-c", ",".join(map(str, cpus)), *command] + started = time.perf_counter() + process = subprocess.run( + command, + input=json.dumps(request, separators=(",", ":")) + "\n", + text=True, + capture_output=True, + timeout=timeout, + check=False, + ) + wall_ms = (time.perf_counter() - started) * 1_000.0 + if process.returncode != 0: + raise RuntimeError( + f"tool executor exited {process.returncode}: {process.stderr.strip()}" + ) + envelope = json.loads(process.stdout) + result = envelope.get("result") if isinstance(envelope, dict) else None + if not envelope.get("ok") or not isinstance(result, dict): + raise RuntimeError(f"tool executor returned invalid data: {envelope!r}") + if ( + result.get("call_sha256") != call_sha256(call) + or result.get("call_ref") != call_ref(call) + or result.get("tool_name") != call.get("name") + or result.get("side_effects") is not False + ): + raise RuntimeError("tool result does not exactly match the read-only call") + return {"wall_ms": wall_ms, "result": result} + + +def run_branch( + args: argparse.Namespace, + pattern: CompiledPattern, + root: dict[str, str], + label: str, +) -> dict[str, Any]: + started = time.perf_counter() + steps = [] + previous: dict[str, Any] | None = None + for index in range(len(pattern.steps)): + call = pattern.instantiate(root, previous, index) + tool = execute_tool_safe( + args.binary, + call, + args.tool_cpus, + args.timeout, + f"{label}-step-{index + 1}", + ) + previous = tool["result"] + steps.append({"call": call, "tool_result": previous, "tool_wall_ms": tool["wall_ms"]}) + return { + "root": root, + "steps": steps, + "final_ref": steps[-1]["tool_result"]["call_ref"], + "wall_ms": (time.perf_counter() - started) * 1_000.0, + } + + +def start_graph( + args: argparse.Namespace, + pattern: CompiledPattern, + items: list[dict[str, str]], + label: str, +) -> dict[str, Any]: + pool = ThreadPoolExecutor(max_workers=len(items), thread_name_prefix="tool-graph") + started = time.perf_counter() + futures: list[Future[dict[str, Any]]] = [ + pool.submit(run_branch, args, pattern, root, f"{label}-branch-{index}") + for index, root in enumerate(items) + ] + return {"pool": pool, "futures": futures, "started": started} + + +def finish_graph(handle: dict[str, Any], timeout: float) -> dict[str, Any]: + pool: ThreadPoolExecutor = handle["pool"] + futures: list[Future[dict[str, Any]]] = handle["futures"] + try: + branches = [future.result(timeout=timeout) for future in futures] + finally: + pool.shutdown(wait=True, cancel_futures=True) + return { + "branches": branches, + "wall_ms": (time.perf_counter() - float(handle["started"])) * 1_000.0, + } + + +def expected_final(branches: list[dict[str, Any]]) -> str: + return "workflow_complete:" + ",".join(branch["final_ref"] for branch in branches) + + +def final_answer_correct(content: str, expected: str) -> bool: + """Accept the receipt literally or as the equivalent one-field JSON object.""" + content = content.strip() + if content == expected: + return True + prefix = "workflow_complete:" + if not expected.startswith(prefix): + return False + receipt = expected[len(prefix) :] + if content == receipt: + return True + try: + parsed = json.loads(content) + except json.JSONDecodeError: + return False + return parsed == {"workflow_complete": receipt} + + +def post_final( + args: argparse.Namespace, + messages: list[dict[str, Any]], +) -> dict[str, Any]: + """Ask the model to derive the receipt from the actual tool conversation.""" + return post_turn( + args, + [ + *messages, + { + "role": "user", + "content": ( + "Copy the final_ref strings from the completed tool result in " + "their original item order. Return exactly one receipt in the " + "form workflow_complete:ref_one,ref_two. Replace the example " + "references with the actual values. Use no spaces or prose." + ), + }, + ], + [], + "none", + args.final_max_tokens, + ) + + +def stage_result_message( + pattern: CompiledPattern, + step_index: int, + branches: list[dict[str, Any]], + tool_call_id: str, +) -> dict[str, Any]: + step = pattern.steps[step_index] + content = { + "stage": step_index + 1, + "complete": step_index + 1 == len(pattern.steps), + "items": [ + { + **branch["root"], + "final_ref": branch["steps"][-1]["tool_result"]["call_ref"], + } + for branch in branches + ], + "side_effects": False, + } + return { + "role": "tool", + "tool_call_id": tool_call_id, + "name": f"batch_{step.tool}", + "content": json.dumps(content, sort_keys=True, separators=(",", ":")), + } + + +def flatten_graph_calls(graph: dict[str, Any]) -> list[str]: + return [ + canonical_call(step["call"]) + for branch in graph["branches"] + for step in branch["steps"] + ] + + +def flatten_graph_results(graph: dict[str, Any]) -> list[str]: + return [ + step["tool_result"]["call_sha256"] + for branch in graph["branches"] + for step in branch["steps"] + ] + + +def run_stage_batched( + args: argparse.Namespace, + task: dict[str, Any], + pattern: CompiledPattern, + label: str, +) -> dict[str, Any]: + """Run a strong non-speculative baseline with parallel calls per stage.""" + started = time.perf_counter() + stage_messages = stage_batched_messages(task, pattern) + final_messages = list(stage_messages) + current_tools: list[dict[str, Any]] = [] + branches = [ + {"root": root, "steps": [], "final_ref": ""} for root in task["items"] + ] + previous_results: list[dict[str, Any] | None] = [None] * len(branches) + turns = [] + exposed_tool_wait_ms = 0.0 + + for step_index, step in enumerate(pattern.steps): + expected_calls = [ + pattern.instantiate(branch["root"], previous_results[index], step_index) + for index, branch in enumerate(branches) + ] + batch_name = f"batch_{step.tool}" + stage_ref = stage_reference(task, pattern, step_index) + current_tools = [ + stage_batch_tool(pattern, step_index, args.max_branches, stage_ref) + ] + expected_batch = { + "name": batch_name, + "arguments": {"stage_ref": stage_ref}, + } + model = post_turn( + args, + stage_messages, + current_tools, + {"type": "function", "function": {"name": batch_name}}, + args.call_max_tokens, + ) + if len(model["calls"]) != 1 or model["calls"][0]["call"] != expected_batch: + raise RuntimeError( + f"{task['id']} stage {step_index + 1}: batch call " + f"{[item['call'] for item in model['calls']]!r} != " + f"{expected_batch!r}; content={model['content']!r}" + ) + stage_started = time.perf_counter() + with ThreadPoolExecutor( + max_workers=len(expected_calls), thread_name_prefix="batched-tool-stage" + ) as pool: + futures = [ + pool.submit( + execute_tool_safe, + args.binary, + call, + args.tool_cpus, + args.timeout, + f"{label}-stage-{step_index}-branch-{branch_index}", + ) + for branch_index, call in enumerate(expected_calls) + ] + stage_tools = [future.result(timeout=args.timeout) for future in futures] + exposed_tool_wait_ms += (time.perf_counter() - stage_started) * 1_000.0 + + for branch_index, (call, tool) in enumerate( + zip(expected_calls, stage_tools, strict=True) + ): + result = tool["result"] + previous_results[branch_index] = result + branches[branch_index]["steps"].append( + { + "call": call, + "tool_result": result, + "tool_wall_ms": tool["wall_ms"], + } + ) + if step_index + 1 == len(pattern.steps): + final_messages.extend( + [ + model["assistant_message"], + stage_result_message( + pattern, step_index, branches, model["calls"][0]["id"] + ), + ] + ) + turns.append(model) + + for branch in branches: + branch["final_ref"] = branch["steps"][-1]["tool_result"]["call_ref"] + expected = expected_final(branches) + final = post_final(args, final_messages) + all_turns = [*turns, final] + graph = {"branches": branches} + return { + "task_ms": (time.perf_counter() - started) * 1_000.0, + "model_turns": len(all_turns), + "call_turns": turns, + "final": final, + "expected_final": expected, + "final_correct": final_answer_correct(final["content"], expected), + "graph": graph, + "underlying_calls": flatten_graph_calls(graph), + "tool_results": flatten_graph_results(graph), + "model_compute_ms": sum(turn["model_compute_ms"] for turn in all_turns), + "decode_ms": sum(turn["decode_ms"] for turn in all_turns), + "completion_tokens": sum(turn["completion_tokens"] for turn in all_turns), + "exposed_tool_wait_ms": exposed_tool_wait_ms, + "all_ds4_active": all(turn["accept_rate"] > 0.0 for turn in turns), + } + + +def macro_result_message( + pattern: CompiledPattern, + graph: dict[str, Any], + tool_call_id: str, +) -> dict[str, Any]: + content = { + "workflow": pattern.macro_name, + "call_count": sum(len(branch["steps"]) for branch in graph["branches"]), + "items": [ + { + **branch["root"], + "final_ref": branch["final_ref"], + } + for branch in graph["branches"] + ], + "side_effects": False, + } + return { + "role": "tool", + "tool_call_id": tool_call_id, + "name": pattern.macro_name, + "content": json.dumps(content, sort_keys=True, separators=(",", ":")), + } + + +def graph_from_speculative_result( + metadata: dict[str, Any], pattern: CompiledPattern, call: dict[str, Any] +) -> dict[str, Any]: + result = metadata.get("result") + if not isinstance(result, dict): + raise RuntimeError("engine speculation hit has no compiled workflow result") + branches = result.get("branches") + if ( + result.get("call_sha256") != call_sha256(call) + or result.get("call_ref") != call_ref(call) + or result.get("tool_name") != pattern.macro_name + or result.get("workflow_fingerprint") != pattern.fingerprint + or result.get("side_effects") is not False + or not isinstance(branches, list) + ): + raise RuntimeError("engine returned an invalid compiled workflow result") + return {"branches": branches, "wall_ms": float(result.get("elapsed_ms", 0.0))} + + +def run_macro( + args: argparse.Namespace, + task: dict[str, Any], + pattern: CompiledPattern, + speculative: bool, + label: str, +) -> dict[str, Any]: + started = time.perf_counter() + messages = macro_messages(task, pattern) + workflow_ref = workflow_reference(task, pattern) + tools = [pattern.macro_tool(args.max_branches, workflow_ref)] + parsed_items = parse_request_customers(messages[-1]["content"]) + if parsed_items != task["items"]: + raise RuntimeError("event extractor did not reproduce structured request data") + expected_call = { + "name": pattern.macro_name, + "arguments": {"workflow_ref": workflow_ref}, + } + model = post_turn( + args, + messages, + tools, + {"type": "function", "function": {"name": pattern.macro_name}}, + args.macro_max_tokens, + automatic_tool_speculation=speculative, + ) + if len(model["calls"]) != 1: + raise RuntimeError( + f"{task['id']}: macro turn emitted {len(model['calls'])} calls; " + f"content={model['content']!r}" + ) + emitted = model["calls"][0] + macro_correct = emitted["call"] == expected_call + if not macro_correct: + raise RuntimeError( + f"{task['id']}: macro call {emitted['call']!r} != {expected_call!r}" + ) + + metadata = model.get("speculation") if speculative else None + if speculative and not isinstance(metadata, dict): + raise RuntimeError("automatic Qwen speculation returned no engine metadata") + prediction = metadata.get("prediction") if isinstance(metadata, dict) else None + prediction_hit = ( + speculative + and metadata.get("status") == "hit" + and prediction == expected_call + ) + predictor_ms = ( + float(metadata.get("predictor_wall_ms", 0.0)) + if isinstance(metadata, dict) + else 0.0 + ) + if prediction_hit: + graph = graph_from_speculative_result(metadata, pattern, emitted["call"]) + exposed_wait_ms = float(metadata.get("commit_wait_ms", 0.0)) + else: + graph_handle = start_graph(args, pattern, task["items"], f"{label}-authoritative") + wait_started = time.perf_counter() + graph = finish_graph(graph_handle, args.timeout) + exposed_wait_ms = (time.perf_counter() - wait_started) * 1_000.0 + expected_calls = [ + canonical_call(call) + for root in task["items"] + for call in pattern.simulate(root) + ] + actual_calls = flatten_graph_calls(graph) + if actual_calls != expected_calls: + raise RuntimeError(f"{task['id']}: compiled graph diverged from learned pattern") + messages.extend( + [ + model["assistant_message"], + macro_result_message(pattern, graph, emitted["id"]), + ] + ) + expected = expected_final(graph["branches"]) + final = post_final(args, messages) + all_turns = [model, final] + return { + "task_ms": (time.perf_counter() - started) * 1_000.0, + "model_turns": len(all_turns), + "call_turns": [model], + "macro_call": emitted["call"], + "macro_correct": macro_correct, + "prediction_hit": prediction_hit, + "prediction_source": ( + metadata.get("prediction_source") if isinstance(metadata, dict) else None + ), + "prediction_status": metadata.get("status") if isinstance(metadata, dict) else None, + "prediction_reason": metadata.get("reason") if isinstance(metadata, dict) else None, + "predictor_ms": predictor_ms, + "graph": graph, + "graph_wall_ms": graph["wall_ms"], + "exposed_tool_wait_ms": exposed_wait_ms, + "underlying_calls": actual_calls, + "tool_results": flatten_graph_results(graph), + "final": final, + "expected_final": expected, + "final_correct": final_answer_correct(final["content"], expected), + "model_compute_ms": sum(turn["model_compute_ms"] for turn in all_turns), + "decode_ms": sum(turn["decode_ms"] for turn in all_turns), + "completion_tokens": sum(turn["completion_tokens"] for turn in all_turns), + "all_ds4_active": model["accept_rate"] > 0.0, + } + + +def interference_observation_qualified(observation: dict[str, Any]) -> bool: + compiled = observation["compiled"] + speculative = observation["speculative"] + return ( + compiled["cache_hit"] == speculative["cache_hit"] + and compiled["cached_prefix_tokens"] + == speculative["cached_prefix_tokens"] + and compiled["completion_tokens"] + == speculative["completion_tokens"] + and compiled["call_sha256"] == speculative["call_sha256"] + and compiled["accept_rate"] > 0.0 + and speculative["accept_rate"] > 0.0 + ) + + +def measure_private_miss( + args: argparse.Namespace, + target_task: dict[str, Any], + predicted_task: dict[str, Any], + pattern: CompiledPattern, +) -> dict[str, Any]: + """Verify that a wrong workflow never crosses the exact-match gate.""" + target_ref = workflow_reference(target_task, pattern) + predicted_ref = workflow_reference(predicted_task, pattern) + if target_ref == predicted_ref: + raise ValueError("privacy miss requires distinct workflow references") + expected_call = { + "name": pattern.macro_name, + "arguments": {"workflow_ref": target_ref}, + } + predicted_call = { + "name": pattern.macro_name, + "arguments": {"workflow_ref": predicted_ref}, + } + observation = post_turn( + args, + macro_messages(target_task, pattern), + [pattern.macro_tool(args.max_branches, target_ref)], + {"type": "function", "function": {"name": pattern.macro_name}}, + args.macro_max_tokens, + tool_speculation={"call": predicted_call, "confidence": 1.0}, + ) + if ( + len(observation["calls"]) != 1 + or observation["calls"][0]["call"] != expected_call + ): + raise RuntimeError("privacy miss did not emit the expected authoritative call") + metadata = observation.get("speculation") + passed = ( + isinstance(metadata, dict) + and metadata.get("status") == "miss" + and metadata.get("reason") == "invocation_mismatch" + and metadata.get("prediction") == predicted_call + and "result" not in metadata + ) + return { + "passed": passed, + "status": metadata.get("status") if isinstance(metadata, dict) else None, + "reason": metadata.get("reason") if isinstance(metadata, dict) else None, + "private_result_exposed": ( + "result" in metadata if isinstance(metadata, dict) else None + ), + "prediction_sha256": hashlib.sha256( + canonical_call(predicted_call).encode() + ).hexdigest(), + "authoritative_call_sha256": hashlib.sha256( + canonical_call(expected_call).encode() + ).hexdigest(), + } + + +def interference_probe_qualified(probe: dict[str, Any]) -> bool: + """Return whether every repeated probe used matched measured conditions.""" + observations = probe.get("observations") + if isinstance(observations, list): + return bool(observations) and all( + interference_observation_qualified(observation) + for observation in observations + ) + return interference_observation_qualified(probe) + + +def measure_interference_probe( + args: argparse.Namespace, + task: dict[str, Any], + pattern: CompiledPattern, + pair_index: int, + previous: dict[str, Any] | None = None, +) -> dict[str, Any]: + """Compare target compute with repeated, immediately warmed prompts.""" + messages = macro_messages(task, pattern) + workflow_ref = workflow_reference(task, pattern) + tools = [pattern.macro_tool(args.max_branches, workflow_ref)] + expected_call = { + "name": pattern.macro_name, + "arguments": {"workflow_ref": workflow_ref}, + } + tool_choice = { + "type": "function", + "function": {"name": pattern.macro_name}, + } + observations: list[dict[str, Any]] = [] + if isinstance(previous, dict): + prior_observations = previous.get("observations") + if isinstance(prior_observations, list): + observations.extend(prior_observations) + elif isinstance(previous.get("compiled"), dict) and isinstance( + previous.get("speculative"), dict + ): + observations.append( + { + "order": previous.get("order", ["compiled", "speculative"]), + "compiled": previous["compiled"], + "speculative": previous["speculative"], + } + ) + + while len(observations) < args.interference_repetitions: + repetition = len(observations) + order = ( + [False, True] + if (pair_index + repetition) % 2 == 0 + else [True, False] + ) + measured: dict[str, dict[str, Any]] = {} + for automatic in order: + # Prefix-cache order dominated the old six-sample p95. Warm the + # exact prompt immediately before each observation so the probe + # measures predictor/tool interference, not arm ordering. + post_turn( + args, + messages, + tools, + tool_choice, + args.macro_max_tokens, + automatic_tool_speculation=False, + ) + observation = post_turn( + args, + messages, + tools, + tool_choice, + args.macro_max_tokens, + automatic_tool_speculation=automatic, + ) + if ( + len(observation["calls"]) != 1 + or observation["calls"][0]["call"] != expected_call + ): + raise RuntimeError( + f"{task['id']}: interference probe emitted an invalid macro call" + ) + metadata = observation.get("speculation") + if automatic and ( + not isinstance(metadata, dict) + or metadata.get("status") != "hit" + or metadata.get("prediction_source") != NATIVE_PREDICTION_SOURCE + or metadata.get("prediction") != expected_call + ): + raise RuntimeError( + f"{task['id']}: interference probe did not produce an exact Qwen hit" + ) + measured["speculative" if automatic else "compiled"] = { + "model_compute_ms": observation["model_compute_ms"], + "prefill_ms": observation["prefill_ms"], + "decode_ms": observation["decode_ms"], + "completion_tokens": observation["completion_tokens"], + "accept_rate": observation["accept_rate"], + "cache_hit": observation["cache_hit"], + "cached_prefix_tokens": observation["cached_prefix_tokens"], + "call_sha256": hashlib.sha256( + canonical_call(expected_call).encode() + ).hexdigest(), + } + observations.append( + { + "order": [ + "speculative" if value else "compiled" for value in order + ], + **measured, + } + ) + + probe = { + "repetitions": len(observations), + "observations": observations, + } + # A cache miss is valid when both observations miss identically. Requiring + # a hit made an otherwise matched, zero-cache comparison fail closed. + probe["qualified"] = interference_probe_qualified(probe) + return probe + + +def macro_signature(arm: dict[str, Any]) -> dict[str, Any]: + turn = arm["call_turns"][0] + return { + "macro_call": canonical_call(arm["macro_call"]), + "turn_content": turn["content_sha256"], + "turn_tokens": turn["completion_tokens"], + "final_content": arm["final"]["content_sha256"], + "final_tokens": arm["final"]["completion_tokens"], + } + + +def bootstrap_speedup_ci( + pairs: list[dict[str, Any]], numerator: str, denominator: str, resamples: int, seed: int +) -> list[float]: + generator = random.Random(seed) + values = [] + for _ in range(resamples): + sample = [pairs[generator.randrange(len(pairs))] for _ in pairs] + values.append( + statistics.median( + pair[numerator]["task_ms"] / pair[denominator]["task_ms"] + for pair in sample + ) + ) + return [percentile(values, 0.025), percentile(values, 0.975)] + + +def paired_probe_slowdown( + pairs: list[dict[str, Any]], metric: str, quantile: float +) -> float: + ratios = [] + for pair in pairs: + probe = pair["interference_probe"] + observations = probe.get("observations") + if not isinstance(observations, list): + observations = [probe] + paired_ratios = [ + observation["speculative"][metric] + / observation["compiled"][metric] + for observation in observations + if observation["compiled"][metric] > 0.0 + ] + if paired_ratios: + ratios.append(statistics.median(paired_ratios)) + return 100.0 * (percentile(ratios, quantile) - 1.0) + + +def summarize( + pairs: list[dict[str, Any]], resamples: int, seed: int +) -> dict[str, Any]: + baseline_speedups = [ + pair["stage_batched"]["task_ms"] / pair["speculative"]["task_ms"] + for pair in pairs + ] + compiled_speedups = [ + pair["compiled"]["task_ms"] / pair["speculative"]["task_ms"] for pair in pairs + ] + continuation_turns = [ + turn + for pair in pairs + for arm_name in ("stage_batched", "compiled", "speculative") + for turn in [*pair[arm_name]["call_turns"][1:], pair[arm_name]["final"]] + ] + speedup_by_call_count = {} + for call_count in sorted({pair["task"]["call_count"] for pair in pairs}): + bucket = [pair for pair in pairs if pair["task"]["call_count"] == call_count] + speedup_by_call_count[str(call_count)] = { + "tasks": len(bucket), + "stage_batched_task_p50_ms": statistics.median( + pair["stage_batched"]["task_ms"] for pair in bucket + ), + "compiled_task_p50_ms": statistics.median( + pair["compiled"]["task_ms"] for pair in bucket + ), + "speculative_task_p50_ms": statistics.median( + pair["speculative"]["task_ms"] for pair in bucket + ), + "combined_speedup_p50": statistics.median( + pair["stage_batched"]["task_ms"] + / pair["speculative"]["task_ms"] + for pair in bucket + ), + "speculation_speedup_p50": statistics.median( + pair["compiled"]["task_ms"] / pair["speculative"]["task_ms"] + for pair in bucket + ), + } + return { + "tasks": len(pairs), + "calls_per_task": [pair["task"]["call_count"] for pair in pairs], + "stage_batched_task_p50_ms": statistics.median( + pair["stage_batched"]["task_ms"] for pair in pairs + ), + "compiled_task_p50_ms": statistics.median( + pair["compiled"]["task_ms"] for pair in pairs + ), + "speculative_task_p50_ms": statistics.median( + pair["speculative"]["task_ms"] for pair in pairs + ), + "stage_batched_to_speculative_speedup_p50": statistics.median( + baseline_speedups + ), + "stage_batched_to_speculative_speedup_p05": percentile( + baseline_speedups, 0.05 + ), + "stage_batched_to_speculative_speedup_min": min(baseline_speedups), + "stage_batched_to_speculative_bootstrap_95ci": bootstrap_speedup_ci( + pairs, "stage_batched", "speculative", resamples, seed + ), + "stage_batched_to_compiled_speedup_p50": statistics.median( + pair["stage_batched"]["task_ms"] / pair["compiled"]["task_ms"] + for pair in pairs + ), + "compiled_to_speculative_speedup_p50": statistics.median(compiled_speedups), + "compiled_to_speculative_speedup_p05": percentile(compiled_speedups, 0.05), + "compiled_to_speculative_bootstrap_95ci": bootstrap_speedup_ci( + pairs, "compiled", "speculative", resamples, seed + 1 + ), + "total_wall_speedup": sum( + pair["stage_batched"]["task_ms"] for pair in pairs + ) + / sum(pair["speculative"]["task_ms"] for pair in pairs), + "stage_batched_model_turns_p50": statistics.median( + pair["stage_batched"]["model_turns"] for pair in pairs + ), + "compiled_model_turns_p50": statistics.median( + pair["compiled"]["model_turns"] for pair in pairs + ), + "stage_batched_exposed_tool_wait_p50_ms": statistics.median( + pair["stage_batched"]["exposed_tool_wait_ms"] for pair in pairs + ), + "compiled_exposed_tool_wait_p50_ms": statistics.median( + pair["compiled"]["exposed_tool_wait_ms"] for pair in pairs + ), + "speculative_exposed_tool_wait_p50_ms": statistics.median( + pair["speculative"]["exposed_tool_wait_ms"] for pair in pairs + ), + "pattern_prediction_hit_rate": sum( + pair["speculative"]["prediction_hit"] for pair in pairs + ) + / len(pairs), + "all_predictions_from_qwen": all( + pair["speculative"]["prediction_source"] == NATIVE_PREDICTION_SOURCE + for pair in pairs + ), + "predictor_p50_ms": statistics.median( + pair["speculative"]["predictor_ms"] for pair in pairs + ), + "model_compute_slowdown_p50_percent": paired_probe_slowdown( + pairs, "model_compute_ms", 0.50 + ), + "model_compute_slowdown_p95_percent": paired_probe_slowdown( + pairs, "model_compute_ms", 0.95 + ), + "decode_slowdown_p50_percent": paired_probe_slowdown( + pairs, "decode_ms", 0.50 + ), + "decode_slowdown_p95_percent": paired_probe_slowdown( + pairs, "decode_ms", 0.95 + ), + "all_interference_probes_qualified": all( + pair["interference_probe"]["qualified"] for pair in pairs + ), + "continuation_cache_hit_rate": sum( + turn["cache_hit"] and turn["cached_prefix_tokens"] > 0 + for turn in continuation_turns + ) + / len(continuation_turns), + "speedup_by_call_count": speedup_by_call_count, + "all_calls_stable": all( + pair["stage_batched"]["underlying_calls"] + == pair["compiled"]["underlying_calls"] + == pair["speculative"]["underlying_calls"] + for pair in pairs + ), + "all_tool_results_stable": all( + pair["stage_batched"]["tool_results"] + == pair["compiled"]["tool_results"] + == pair["speculative"]["tool_results"] + for pair in pairs + ), + "macro_output_stability_rate": sum( + macro_signature(pair["compiled"]) == macro_signature(pair["speculative"]) + for pair in pairs + ) + / len(pairs), + "all_final_answers_correct": all( + pair[arm]["final_correct"] + for pair in pairs + for arm in ("stage_batched", "compiled", "speculative") + ), + "all_final_outputs_stable": all( + pair["stage_batched"]["final"]["content"].strip() + == pair["compiled"]["final"]["content"].strip() + == pair["speculative"]["final"]["content"].strip() + for pair in pairs + ), + "all_macro_calls_correct": all( + pair[arm]["macro_correct"] + for pair in pairs + for arm in ("compiled", "speculative") + ), + "all_ds4_active": all( + pair[arm]["all_ds4_active"] + for pair in pairs + for arm in ("stage_batched", "compiled", "speculative") + ), + } + + +def production_checks(summary: dict[str, Any], args: argparse.Namespace) -> dict[str, bool]: + return { + "sample_size": summary["tasks"] >= args.min_production_pairs, + "end_to_end_speedup": summary["stage_batched_to_speculative_speedup_p50"] + >= args.min_e2e_speedup, + "end_to_end_ci": summary[ + "stage_batched_to_speculative_bootstrap_95ci" + ][0] + > 1.0, + "end_to_end_tail": summary["stage_batched_to_speculative_speedup_p05"] + >= args.min_e2e_speedup_p05, + "speculation_incremental_gain": summary["compiled_to_speculative_speedup_p50"] + >= args.min_incremental_speedup, + "speculation_incremental_ci": summary[ + "compiled_to_speculative_bootstrap_95ci" + ][0] + > 1.0, + "prediction_hit_rate": summary["pattern_prediction_hit_rate"] == 1.0, + "prediction_source": summary["all_predictions_from_qwen"], + "private_miss_result_hidden": summary["private_miss_result_hidden"], + "interference_probes": summary["all_interference_probes_qualified"], + "model_slowdown_p50": summary["model_compute_slowdown_p50_percent"] + <= args.max_model_slowdown_percent, + "model_slowdown_p95": summary["model_compute_slowdown_p95_percent"] + <= args.max_model_slowdown_p95_percent, + "decode_slowdown_p50": summary["decode_slowdown_p50_percent"] + <= args.max_decode_slowdown_percent, + "decode_slowdown_p95": summary["decode_slowdown_p95_percent"] + <= args.max_decode_slowdown_p95_percent, + "prefix_cache_configured": summary["prefix_cache_configured"], + "calls_stable": summary["all_calls_stable"], + "tool_results_stable": summary["all_tool_results_stable"], + "macro_outputs_stable": summary["macro_output_stability_rate"] == 1.0, + "final_answers_correct": summary["all_final_answers_correct"], + "final_outputs_stable": summary["all_final_outputs_stable"], + "macro_calls_correct": summary["all_macro_calls_correct"], + "ds4_active": summary["all_ds4_active"], + } + + +def compact_arm(arm: dict[str, Any]) -> dict[str, Any]: + """Keep auditable outputs and timings without embedding the full graph.""" + fields = ( + "task_ms", + "model_turns", + "model_compute_ms", + "decode_ms", + "completion_tokens", + "exposed_tool_wait_ms", + "all_ds4_active", + "final_correct", + "graph_wall_ms", + "macro_call", + "macro_correct", + "prediction_hit", + "prediction_reason", + "prediction_source", + "prediction_status", + "predictor_ms", + ) + compact = {field: arm[field] for field in fields if field in arm} + for field in ("underlying_calls", "tool_results"): + values = arm.get(field) + if isinstance(values, list): + compact[f"{field}_count"] = len(values) + compact[f"{field}_sha256"] = hashlib.sha256( + json.dumps(values, separators=(",", ":")).encode() + ).hexdigest() + final = arm.get("final") + if isinstance(final, dict): + compact["final"] = { + field: final[field] + for field in ( + "content_sha256", + "completion_tokens", + "accept_rate", + ) + if field in final + } + return compact + + +def compact_pair(pair: dict[str, Any]) -> dict[str, Any]: + task = pair["task"] + return { + "pair_index": pair["pair_index"], + "task": { + field: task[field] + for field in ("id", "branch_count", "call_count") + if field in task + }, + "arm_order": pair["arm_order"], + **{ + arm: compact_arm(pair[arm]) + for arm in ("stage_batched", "compiled", "speculative") + }, + "interference_probe": pair["interference_probe"], + } + + +def load_partial_pairs( + path: Path, + measured_tasks: list[dict[str, Any]], + arm_orders: list[list[str]], +) -> list[dict[str, Any]]: + """Load and strictly validate a checkpoint before resuming a long run.""" + checkpoint = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(checkpoint, dict): + raise ValueError("benchmark checkpoint is not an object") + pairs = checkpoint.get("pairs") + if ( + checkpoint.get("schema_version") != 1 + or checkpoint.get("complete") is not False + or not isinstance(pairs, list) + or len(pairs) > len(measured_tasks) + ): + raise ValueError("benchmark checkpoint is not resumable") + for index, pair in enumerate(pairs): + if ( + not isinstance(pair, dict) + or pair.get("pair_index") != index + or pair.get("task") != measured_tasks[index] + or pair.get("arm_order") != arm_orders[index] + or not all( + isinstance(pair.get(arm), dict) + for arm in ("stage_batched", "compiled", "speculative") + ) + ): + raise ValueError(f"benchmark checkpoint pair {index} does not match this run") + for arm in ("stage_batched", "compiled", "speculative"): + result = pair[arm] + final = result.get("final") + expected = result.get("expected_final") + if ( + not isinstance(final, dict) + or not isinstance(final.get("content"), str) + or not isinstance(expected, str) + ): + raise ValueError( + f"benchmark checkpoint pair {index} has an invalid {arm} final" + ) + result["final_correct"] = final_answer_correct( + final["content"], expected + ) + return pairs + + +def validate_args(parser: argparse.ArgumentParser, args: argparse.Namespace) -> None: + if not args.binary.is_file() or not os.access(args.binary, os.X_OK): + parser.error("--binary must be an executable tool adapter") + if not args.training_report.is_file(): + parser.error("--training-report must be an existing trace file or report") + if args.training_report != CANONICAL_TRAINING_REPORT.resolve(): + parser.error( + "--training-report must be the committed canonical report used by " + "the deployed trace executor" + ) + if args.workflow_registry != CANONICAL_WORKFLOW_REGISTRY.resolve(): + parser.error( + "--workflow-registry must be the committed canonical registry used " + "by the deployed trace executor" + ) + if ( + args.pairs < 2 + or args.warmup_tasks < 0 + or not 2 <= args.min_branches <= args.max_branches <= 4 + or args.timeout <= 0 + or min(args.call_max_tokens, args.macro_max_tokens, args.final_max_tokens) <= 0 + or args.bootstrap_resamples <= 0 + or args.interference_repetitions < 3 + or args.interference_repetitions % 2 == 0 + or args.min_production_pairs < 2 + or args.min_e2e_speedup <= 1.0 + or args.min_e2e_speedup_p05 <= 1.0 + or args.min_incremental_speedup <= 1.0 + ): + parser.error("benchmark counts and thresholds are invalid") + + +def refresh_existing_report( + args: argparse.Namespace, + pattern: CompiledPattern, + measured_tasks: list[dict[str, Any]], + prefix_cache: dict[str, Any], + tool_speculation: dict[str, Any], +) -> bool: + """Add newly introduced safety gates without repeating long timing arms.""" + report = json.loads(args.output.read_text(encoding="utf-8")) + if not isinstance(report, dict): + raise ValueError("existing report is not a JSON object") + recorded_pattern = report.get("pattern") + production_gate = report.get("production_gate") + recorded_pairs = report.get("pairs") + if ( + report.get("schema_version") != 1 + or not isinstance(recorded_pattern, dict) + or not isinstance(production_gate, dict) + or not isinstance(recorded_pairs, list) + or recorded_pattern.get("fingerprint") != pattern.fingerprint + or recorded_pattern.get("training_report_sha256") + != file_sha256(args.training_report) + or recorded_pattern.get("workflow_registry_sha256") + != file_sha256(args.workflow_registry) + or not production_gate.get("passed") + or len(recorded_pairs) != args.pairs + ): + raise ValueError("existing report does not match this qualified run") + + summary = report.get("summary") + if not isinstance(summary, dict) or summary.get("tasks") != args.pairs: + raise ValueError("existing report summary does not match --pairs") + required_refresh_fields = { + "stage_batched_to_speculative_speedup_p50", + "stage_batched_to_speculative_bootstrap_95ci", + "stage_batched_to_speculative_speedup_p05", + "compiled_to_speculative_speedup_p50", + "compiled_to_speculative_bootstrap_95ci", + "pattern_prediction_hit_rate", + "all_predictions_from_qwen", + "all_interference_probes_qualified", + "model_compute_slowdown_p50_percent", + "model_compute_slowdown_p95_percent", + "decode_slowdown_p50_percent", + "decode_slowdown_p95_percent", + "prefix_cache_configured", + "all_calls_stable", + "all_tool_results_stable", + "macro_output_stability_rate", + "all_final_answers_correct", + "all_final_outputs_stable", + "all_macro_calls_correct", + "all_ds4_active", + } + missing_refresh_fields = sorted(required_refresh_fields - summary.keys()) + if missing_refresh_fields: + raise ValueError( + "existing report predates interference probes; rerun the full " + "benchmark without --refresh-report (missing summary fields: " + + ", ".join(missing_refresh_fields) + + ")" + ) + methodology = report.get("methodology") + if not isinstance(methodology, dict): + raise ValueError("existing report has no methodology object") + server_snapshot = report.get("server_snapshot") + if not isinstance(server_snapshot, dict): + raise ValueError("existing report has no server_snapshot object") + privacy_miss = measure_private_miss( + args, measured_tasks[0], measured_tasks[1], pattern + ) + summary["private_miss_result_hidden"] = privacy_miss["passed"] + checks = production_checks(summary, args) + report["privacy_miss"] = privacy_miss + methodology["additive_gate_refresh"] = ( + "the wrong-call privacy probe and server snapshot were refreshed after " + "the timing arms; no recorded timing was recomputed" + ) + production_gate["checks"] = checks + production_gate["passed"] = all(checks.values()) + ending_props = get_json(props_url(args.url), args.timeout) + server_snapshot["prefix_cache_before"] = prefix_cache + server_snapshot["prefix_cache_after"] = ending_props.get( + "prefix_cache" + ) + server_snapshot["tool_speculation"] = tool_speculation + args.output.write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + print( + json.dumps( + { + "production_gate": report["production_gate"], + "privacy_miss": privacy_miss, + }, + indent=2, + sort_keys=True, + ), + flush=True, + ) + return report["production_gate"]["passed"] + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--url", default="http://127.0.0.1:18145/v1/chat/completions") + parser.add_argument("--binary", type=Path, required=True) + parser.add_argument("--training-report", type=Path, required=True) + parser.add_argument( + "--workflow-registry", + type=Path, + default=CANONICAL_WORKFLOW_REGISTRY, + ) + parser.add_argument("--tool-cpus", type=parse_cpu_list, default="14-15,30-31") + parser.add_argument("--pairs", type=int, default=6) + parser.add_argument("--warmup-tasks", type=int, default=1) + parser.add_argument("--min-branches", type=int, default=2) + parser.add_argument("--max-branches", type=int, default=4) + parser.add_argument("--call-max-tokens", type=int, default=160) + parser.add_argument("--macro-max-tokens", type=int, default=512) + parser.add_argument("--final-max-tokens", type=int, default=96) + parser.add_argument("--timeout", type=float, default=180.0) + parser.add_argument("--seed", type=int, default=814) + parser.add_argument("--bootstrap-resamples", type=int, default=20_000) + parser.add_argument("--interference-repetitions", type=int, default=3) + parser.add_argument("--min-production-pairs", type=int, default=6) + parser.add_argument("--min-e2e-speedup", type=float, default=2.0) + parser.add_argument("--min-e2e-speedup-p05", type=float, default=1.5) + parser.add_argument("--min-incremental-speedup", type=float, default=1.05) + parser.add_argument("--max-model-slowdown-percent", type=float, default=1.0) + parser.add_argument("--max-model-slowdown-p95-percent", type=float, default=5.0) + parser.add_argument("--max-decode-slowdown-percent", type=float, default=1.0) + parser.add_argument("--max-decode-slowdown-p95-percent", type=float, default=5.0) + parser.add_argument( + "--resume-partial", + action="store_true", + help="resume the strictly matching .partial checkpoint", + ) + parser.add_argument( + "--refresh-report", + action="store_true", + help="run additive safety gates against an existing passing report", + ) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + args.binary = args.binary.resolve() + args.training_report = args.training_report.resolve() + args.workflow_registry = args.workflow_registry.resolve() + validate_args(parser, args) + + traces = load_training_traces(args.training_report, required_steps=5) + pattern = mine_pattern(traces) + props = get_json(props_url(args.url), args.timeout) + prefix_cache = props.get("prefix_cache") + if not isinstance(prefix_cache, dict) or int(prefix_cache.get("capacity", 0)) <= 0: + raise SystemExit("prefix cache is not enabled") + tool_speculation = props.get("tool_speculation") + required_tool_props = { + "enabled": True, + "automatic_prediction_enabled": True, + "prediction_source": NATIVE_PREDICTION_SOURCE, + "predictor_schedule": "before-model", + "execution_mode": "child_process_cpu_affinity", + "cpu_affinity_isolated": True, + "preserves_token_speculation": True, + } + if not isinstance(tool_speculation, dict): + raise SystemExit("engine tool speculation is not enabled") + for key, expected in required_tool_props.items(): + if tool_speculation.get(key) != expected: + raise SystemExit( + f"engine tool_speculation.{key}={tool_speculation.get(key)!r}, " + f"expected {expected!r}" + ) + if tool_speculation.get("tool_cpu_affinity") != args.tool_cpus: + raise SystemExit( + "engine tool CPU affinity does not match --tool-cpus: " + f"{tool_speculation.get('tool_cpu_affinity')!r} != {args.tool_cpus!r}" + ) + if pattern.macro_name not in tool_speculation.get("allowed_tools", []): + raise SystemExit(f"engine does not allow compiled macro {pattern.macro_name!r}") + + branch_span = args.max_branches - args.min_branches + 1 + warmup_tasks = [ + make_task(-1000 - index, args.min_branches, pattern) + for index in range(args.warmup_tasks) + ] + measured_tasks = [ + make_task( + pair_index, + args.min_branches + pair_index % branch_span, + pattern, + ) + for pair_index in range(args.pairs) + ] + write_workflow_registry( + args.workflow_registry, pattern, [*warmup_tasks, *measured_tasks] + ) + if args.refresh_report: + if not args.output.is_file(): + parser.error("--refresh-report requires an existing --output") + if not refresh_existing_report( + args, pattern, measured_tasks, prefix_cache, tool_speculation + ): + raise SystemExit("refreshed workflow production gate failed") + return 0 + generator = random.Random(args.seed) + arm_orders = [] + for _ in measured_tasks: + order = ["stage_batched", "compiled", "speculative"] + generator.shuffle(order) + arm_orders.append(order) + partial_output = args.output.with_suffix(args.output.suffix + ".partial") + pairs = ( + load_partial_pairs(partial_output, measured_tasks, arm_orders) + if args.resume_partial + else [] + ) + if pairs: + print(json.dumps({"resumed_pairs": len(pairs)}), flush=True) + else: + for warmup, task in enumerate(warmup_tasks): + run_stage_batched(args, task, pattern, f"warm-{warmup}-stage-batched") + run_macro(args, task, pattern, False, f"warm-{warmup}-compiled") + run_macro(args, task, pattern, True, f"warm-{warmup}-speculative") + + for pair_index in range(len(pairs), len(measured_tasks)): + task = measured_tasks[pair_index] + order = arm_orders[pair_index] + arms = {} + for arm in order: + if arm == "stage_batched": + arms[arm] = run_stage_batched( + args, task, pattern, f"pair-{pair_index}-stage-batched" + ) + else: + arms[arm] = run_macro( + args, + task, + pattern, + arm == "speculative", + f"pair-{pair_index}-{arm}", + ) + pair = {"pair_index": pair_index, "task": task, "arm_order": order, **arms} + pairs.append(pair) + partial_output.parent.mkdir(parents=True, exist_ok=True) + partial_output.write_text( + json.dumps({"schema_version": 1, "complete": False, "pairs": pairs}, indent=2) + + "\n", + encoding="utf-8", + ) + print( + json.dumps( + { + "pair": pair_index + 1, + "calls": task["call_count"], + "order": order, + "stage_batched_ms": round( + arms["stage_batched"]["task_ms"], 1 + ), + "compiled_ms": round(arms["compiled"]["task_ms"], 1), + "speculative_ms": round(arms["speculative"]["task_ms"], 1), + "end_to_end_speedup": round( + arms["stage_batched"]["task_ms"] + / arms["speculative"]["task_ms"], + 3, + ), + "incremental_speedup": round( + arms["compiled"]["task_ms"] + / arms["speculative"]["task_ms"], + 3, + ), + "prediction_hit": arms["speculative"]["prediction_hit"], + "correct": all(arms[arm]["final_correct"] for arm in arms), + }, + sort_keys=True, + ), + flush=True, + ) + + for pair_index, pair in enumerate(pairs): + previous_probe = pair.get("interference_probe") + if ( + isinstance(previous_probe, dict) + and isinstance(previous_probe.get("observations"), list) + and len(previous_probe["observations"]) + >= args.interference_repetitions + ): + pair["interference_probe"]["qualified"] = ( + interference_probe_qualified(pair["interference_probe"]) + ) + continue + pair["interference_probe"] = measure_interference_probe( + args, + measured_tasks[pair_index], + pattern, + pair_index, + previous_probe if isinstance(previous_probe, dict) else None, + ) + partial_output.write_text( + json.dumps( + {"schema_version": 1, "complete": False, "pairs": pairs}, + indent=2, + ) + + "\n", + encoding="utf-8", + ) + print( + json.dumps( + { + "interference_probe": pair_index + 1, + "qualified": pair["interference_probe"]["qualified"], + } + ), + flush=True, + ) + + privacy_miss = measure_private_miss( + args, measured_tasks[0], measured_tasks[1], pattern + ) + summary = summarize(pairs, args.bootstrap_resamples, args.seed) + summary["private_miss_result_hidden"] = privacy_miss["passed"] + ending_props = get_json(props_url(args.url), args.timeout) + ending_prefix_cache = ending_props.get("prefix_cache") + if not isinstance(ending_prefix_cache, dict): + raise RuntimeError("prefix cache disappeared during the benchmark") + summary["prefix_cache_lifetime_hit_delta"] = int( + ending_prefix_cache.get("lifetime_hits", 0) + ) - int(prefix_cache.get("lifetime_hits", 0)) + summary["prefix_cache_configured"] = int(prefix_cache.get("capacity", 0)) > 0 + checks = production_checks(summary, args) + report = { + "schema_version": 1, + "host": "lucebox5", + "feature": "no-training trace-compiled speculative tool graphs", + "pattern": { + "macro_name": pattern.macro_name, + "fingerprint": pattern.fingerprint, + "training_traces": pattern.training_traces, + "training_report": str( + args.training_report.relative_to(Path(__file__).resolve().parent) + ), + "training_report_sha256": file_sha256(args.training_report), + "workflow_registry": str( + args.workflow_registry.relative_to(Path(__file__).resolve().parent) + ), + "workflow_registry_sha256": file_sha256(args.workflow_registry), + "root_fields": list(pattern.root_fields), + "steps": [ + { + "tool": step.tool, + "arguments": { + name: {"source": binding.source, "key": binding.key} + for name, binding in step.arguments + }, + } + for step in pattern.steps + ], + "model_training": False, + "side_effects_allowed": False, + }, + "workload": { + "tasks": args.pairs, + "branches_per_task": f"{args.min_branches}-{args.max_branches}", + "calls_per_task": f"{args.min_branches * len(pattern.steps)}-" + f"{args.max_branches * len(pattern.steps)}", + "dependency": "five serial calls per branch; branches are independent", + "tool_adapter": "deterministic read-only 2-second API replay", + }, + "methodology": { + "stage_batched": ( + "DS4+DSpark sees only the currently-ready typed batch, authorizes " + "one per dependency stage, runs its calls concurrently, and receives " + "a compact rolling state instead of replaying old tool history" + ), + "compiled": ( + "one DS4+DSpark macro authorization; independent branches execute " + "concurrently on the Strix CPU lane" + ), + "speculative": ( + "Qwen predicts the trace-derived macro through the engine; its CPU " + "graph overlaps DS4+DSpark and commits only on an exact call match" + ), + "arm_order": "randomized per task", + "measured_wall_time": "request through all tools and exact final answer", + "oracle_prediction": False, + "argument_binding": ( + "the harness binds validated structured inputs to a request-scoped " + "workflow_ref before either model runs" + ), + "model_seed": args.seed, + "warmup_tasks": args.warmup_tasks, + }, + "server_snapshot": { + "prefix_cache_before": prefix_cache, + "prefix_cache_after": ending_prefix_cache, + "tool_speculation": tool_speculation, + "model": props.get("model"), + }, + "production_gate": { + "passed": all(checks.values()), + "checks": checks, + "thresholds": { + "min_e2e_speedup": args.min_e2e_speedup, + "min_e2e_speedup_p05": args.min_e2e_speedup_p05, + "min_incremental_speedup": args.min_incremental_speedup, + "min_production_pairs": args.min_production_pairs, + "max_model_slowdown_percent": args.max_model_slowdown_percent, + "max_model_slowdown_p95_percent": args.max_model_slowdown_p95_percent, + "max_decode_slowdown_percent": args.max_decode_slowdown_percent, + "max_decode_slowdown_p95_percent": args.max_decode_slowdown_p95_percent, + }, + }, + "privacy_miss": privacy_miss, + "summary": summary, + "pairs": [compact_pair(pair) for pair in pairs], + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + if report["production_gate"]["passed"]: + partial_output.unlink(missing_ok=True) + print( + json.dumps( + {"production_gate": report["production_gate"], "summary": summary}, + indent=2, + sort_keys=True, + ), + flush=True, + ) + return 0 if report["production_gate"]["passed"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/optimizations/ooo_spec_lucebox5_cpu/bfcl_replay_tool_executor.py b/optimizations/ooo_spec_lucebox5_cpu/bfcl_replay_tool_executor.py new file mode 100755 index 000000000..9c3416e95 --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/bfcl_replay_tool_executor.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +"""Read-only executor for end-to-end BFCL tool-speculation replay. + +BFCL functions are specifications rather than deployable APIs. This adapter +therefore performs no external action: it waits for a fixed, documented API +latency and returns a deterministic digest of the canonical call. The engine's +own allowlist remains the authority for which predictions may reach it. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import sys +import time +from typing import Any + + +PROTOCOL = "dflash.tool-speculation.v1" +LATENCY_MS = 2_000 +REFERENCE_WORDS = ( + "amber", "azure", "cedar", "coral", "gold", "ivory", "maple", "olive", + "pearl", "plum", "sable", "silver", "teal", "violet", "willow", "jade", +) + + +def canonical_call(call: dict[str, Any]) -> str: + return json.dumps(call, sort_keys=True, separators=(",", ":")) + + +def call_sha256(call: dict[str, Any]) -> str: + return hashlib.sha256(canonical_call(call).encode()).hexdigest() + + +def call_ref(call: dict[str, Any]) -> str: + return REFERENCE_WORDS[int(call_sha256(call)[:8], 16) % len(REFERENCE_WORDS)] + + +def execute(request: dict[str, Any]) -> dict[str, Any]: + if request.get("protocol") != PROTOCOL: + raise ValueError("unsupported protocol") + call = request.get("call") + if not isinstance(call, dict): + raise ValueError("call must be an object") + name = call.get("name") + arguments = call.get("arguments") + if not isinstance(name, str) or not name: + raise ValueError("tool name must be a non-empty string") + if not isinstance(arguments, dict): + raise ValueError("tool arguments must be an object") + + expected = request.get("cpu_affinity", []) + if not isinstance(expected, list) or not all( + isinstance(cpu, int) and cpu >= 0 for cpu in expected + ): + raise ValueError("cpu_affinity must contain non-negative integers") + observed = sorted(os.sched_getaffinity(0)) if hasattr( + os, "sched_getaffinity" + ) else [] + if expected and observed != sorted(set(expected)): + raise ValueError("observed CPU affinity does not match request") + + canonical = {"name": name, "arguments": arguments} + digest = call_sha256(canonical) + reference = call_ref(canonical) + started = time.perf_counter() + time.sleep(LATENCY_MS / 1_000.0) + elapsed = (time.perf_counter() - started) * 1_000.0 + return { + "ok": True, + "result": { + "call_sha256": digest, + "call_ref": reference, + "tool_name": name, + "latency_ms": LATENCY_MS, + "elapsed_ms": elapsed, + "cpu_affinity": observed, + "side_effects": False, + }, + } + + +def main() -> int: + if sys.argv[1:] != ["--dflash-tool-spec-v1"]: + print("expected --dflash-tool-spec-v1", file=sys.stderr) + return 2 + try: + line = sys.stdin.readline() + if not line: + raise ValueError("missing request") + request = json.loads(line) + if not isinstance(request, dict): + raise ValueError("request must be an object") + print(json.dumps(execute(request), separators=(",", ":")), flush=True) + return 0 + except (OSError, ValueError, json.JSONDecodeError) as error: + print(str(error), file=sys.stderr) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/optimizations/ooo_spec_lucebox5_cpu/build_cpu_sparse_executor.sh b/optimizations/ooo_spec_lucebox5_cpu/build_cpu_sparse_executor.sh new file mode 100755 index 000000000..4e48334e7 --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/build_cpu_sparse_executor.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +set -euo pipefail + +source_file="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cpu_sparse_tool_executor.cpp" +json_include="${JSON_INCLUDE:-}" +output="${1:-$(dirname "$source_file")/cpu_sparse_tool_executor}" + +if [[ -z "$json_include" ]]; then + printf 'JSON_INCLUDE must point to the directory containing nlohmann/json.hpp\n' >&2 + exit 2 +fi +if [[ ! -f "$json_include/nlohmann/json.hpp" ]]; then + printf 'missing JSON header: %s/nlohmann/json.hpp\n' "$json_include" >&2 + exit 2 +fi + +g++ -std=c++17 -O3 -DNDEBUG -pthread \ + -I"$json_include" "$source_file" -o "$output" diff --git a/optimizations/ooo_spec_lucebox5_cpu/cpu_sparse_tool_executor.cpp b/optimizations/ooo_spec_lucebox5_cpu/cpu_sparse_tool_executor.cpp new file mode 100644 index 000000000..4a52f47ee --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/cpu_sparse_tool_executor.cpp @@ -0,0 +1,241 @@ +// Deterministic read-only sparse-compute adapter for +// dflash.tool-speculation.v1 qualification. +// +// This is a benchmark tool, not an application-specific tool. It provides a +// reproducible CPU-bound workload whose exact result can be compared between +// sequential and speculative execution. The engine pins this child before it +// releases the JSON request, and the adapter verifies the observed mask. + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if defined(__linux__) +#include +#include +#endif + +using json = nlohmann::json; + +namespace { + +constexpr const char * kProtocol = "dflash.tool-speculation.v1"; +constexpr const char * kToolName = "benchmark_cpu_sparse"; +constexpr int kRows = 4096; +constexpr int kNonzerosPerRow = 16; +constexpr int kThreads = 2; +constexpr uint64_t kSeed = 731; + +uint64_t splitmix64(uint64_t & state) { + uint64_t value = (state += 0x9e3779b97f4a7c15ULL); + value = (value ^ (value >> 30U)) * 0xbf58476d1ce4e5b9ULL; + value = (value ^ (value >> 27U)) * 0x94d049bb133111ebULL; + return value ^ (value >> 31U); +} + +std::vector observed_affinity() { +#if defined(__linux__) + cpu_set_t mask; + CPU_ZERO(&mask); + if (::sched_getaffinity(0, sizeof(mask), &mask) != 0) { + throw std::runtime_error("sched_getaffinity failed"); + } + std::vector cpus; + for (int cpu = 0; cpu < CPU_SETSIZE; ++cpu) { + if (CPU_ISSET(cpu, &mask)) cpus.push_back(cpu); + } + return cpus; +#else + return {}; +#endif +} + +uint64_t sparse_worker(int worker, + int rows, + int nonzeros_per_row, + int iterations, + uint64_t seed) { + const size_t entries = + static_cast(rows) * static_cast(nonzeros_per_row); + std::vector columns(entries); + std::vector values(entries); + std::vector input(static_cast(rows)); + std::vector output(static_cast(rows)); + + uint64_t state = seed ^ + (0xd6e8feb86659fd93ULL * static_cast(worker + 1)); + for (size_t index = 0; index < entries; ++index) { + columns[index] = static_cast(splitmix64(state) % rows); + values[index] = static_cast(splitmix64(state) | 1ULL); + } + for (uint32_t & value : input) { + value = static_cast(splitmix64(state)); + } + + uint64_t rolling = 0xcbf29ce484222325ULL ^ + static_cast(worker); + for (int iteration = 0; iteration < iterations; ++iteration) { + for (int row = 0; row < rows; ++row) { + uint64_t accumulator = + static_cast(iteration + 1) * 0x9e3779b1U + + static_cast(row); + const size_t start = + static_cast(row) * nonzeros_per_row; + for (int offset = 0; offset < nonzeros_per_row; ++offset) { + const size_t index = start + static_cast(offset); + accumulator += static_cast(values[index]) * + input[columns[index]]; + } + const uint32_t folded = static_cast( + accumulator ^ (accumulator >> 32U)); + output[static_cast(row)] = + folded + static_cast(row * 2654435761U); + } + input.swap(output); + rolling ^= static_cast(input[ + static_cast(iteration) % input.size()]); + rolling *= 0x100000001b3ULL; + } + for (size_t index = 0; index < input.size(); index += 17) { + rolling ^= static_cast(input[index]) + index; + rolling *= 0x100000001b3ULL; + } + return rolling; +} + +int integer_argument(const json & arguments, + const char * name, + int minimum, + int maximum) { + if (!arguments.contains(name) || !arguments[name].is_number_integer()) { + throw std::runtime_error(std::string(name) + " must be an integer"); + } + const int value = arguments[name].get(); + if (value < minimum || value > maximum) { + throw std::runtime_error( + std::string(name) + " is outside the allowed range"); + } + return value; +} + +json execute(const json & request) { + if (!request.is_object() || request.value("protocol", "") != kProtocol) { + throw std::runtime_error("unsupported protocol"); + } + if (!request.contains("call") || !request["call"].is_object() || + request["call"].value("name", "") != kToolName) { + throw std::runtime_error("only benchmark_cpu_sparse is allowed"); + } + const json & arguments = request["call"].at("arguments"); + if (!arguments.is_object() || arguments.size() != 1 || + !arguments.contains("iterations")) { + throw std::runtime_error("arguments must contain only iterations"); + } + const int rows = kRows; + const int nonzeros = kNonzerosPerRow; + const int iterations = integer_argument( + arguments, "iterations", 1, 1'000'000); + const int threads = kThreads; + const uint64_t seed = kSeed; + + std::vector expected_affinity; + const auto affinity_value = request.find("cpu_affinity"); + if (affinity_value == request.end() || !affinity_value->is_array() || + affinity_value->empty()) { + throw std::runtime_error("cpu_affinity must be a non-empty array"); + } + expected_affinity = affinity_value->get>(); + std::sort(expected_affinity.begin(), expected_affinity.end()); + expected_affinity.erase( + std::unique(expected_affinity.begin(), expected_affinity.end()), + expected_affinity.end()); + const std::vector affinity = observed_affinity(); + if (affinity != expected_affinity) { + throw std::runtime_error("observed CPU affinity does not match request"); + } + if (!affinity.empty() && threads > static_cast(affinity.size())) { + throw std::runtime_error("threads exceed the pinned logical CPU count"); + } + + const auto started = std::chrono::steady_clock::now(); + std::vector partial(static_cast(threads)); + std::vector pin_errors(static_cast(threads), 0); + std::vector workers; + workers.reserve(static_cast(threads)); + for (int worker = 0; worker < threads; ++worker) { + workers.emplace_back([&, worker]() { +#if defined(__linux__) + if (!affinity.empty()) { + cpu_set_t worker_mask; + CPU_ZERO(&worker_mask); + CPU_SET(affinity[static_cast(worker)], &worker_mask); + pin_errors[static_cast(worker)] = + ::pthread_setaffinity_np( + ::pthread_self(), sizeof(worker_mask), &worker_mask); + if (pin_errors[static_cast(worker)] != 0) return; + } +#endif + partial[static_cast(worker)] = sparse_worker( + worker, rows, nonzeros, iterations, seed); + }); + } + for (std::thread & worker : workers) worker.join(); + if (std::any_of(pin_errors.begin(), pin_errors.end(), + [](int error) { return error != 0; })) { + throw std::runtime_error("worker CPU pinning failed"); + } + const double compute_ms = + std::chrono::duration( + std::chrono::steady_clock::now() - started).count(); + + uint64_t checksum = 0x6a09e667f3bcc909ULL; + for (const uint64_t value : partial) { + checksum ^= value + 0x9e3779b97f4a7c15ULL + + (checksum << 6U) + (checksum >> 2U); + } + return { + {"ok", true}, + {"result", { + {"checksum", std::to_string(checksum)}, + {"compute_ms", compute_ms}, + {"rows", rows}, + {"nonzeros_per_row", nonzeros}, + {"iterations", iterations}, + {"threads", threads}, + {"seed", seed}, + {"cpu_affinity", affinity}, + {"worker_cpus", std::vector( + affinity.begin(), affinity.begin() + + std::min(affinity.size(), static_cast(threads)))}, + }}, + }; +} + +} // namespace + +int main(int argc, char ** argv) { + if (argc != 2 || std::string(argv[1]) != "--dflash-tool-spec-v1") { + std::cerr << "expected --dflash-tool-spec-v1\n"; + return 2; + } + try { + std::string line; + if (!std::getline(std::cin, line) || line.empty()) { + throw std::runtime_error("missing request"); + } + std::cout << execute(json::parse(line)).dump() << '\n'; + std::cout.flush(); + return 0; + } catch (const std::exception & exception) { + std::cerr << exception.what() << '\n'; + return 2; + } +} diff --git a/optimizations/ooo_spec_lucebox5_cpu/dflash_server_native_tool_predictor_wrapper.sh b/optimizations/ooo_spec_lucebox5_cpu/dflash_server_native_tool_predictor_wrapper.sh new file mode 100755 index 000000000..07e2a399e --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/dflash_server_native_tool_predictor_wrapper.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Preserve every argument from the qualified 0731 launcher and add the native +# Qwen3 tool-prediction lane on the Strix GPU. The qualified launcher clears +# ambient variables, so an adjacent `candidate-build` symlink is the durable +# deployment override; direct launches may still use CANDIDATE_BUILD. +wrapper_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +default_candidate="/home/lucebox5/tool-spec-cpu-20260813/engine-ooo-spec/server/build-hip-dual" +if [[ -d "${wrapper_dir}/candidate-build" ]]; then + default_candidate="${wrapper_dir}/candidate-build" +fi +CANDIDATE_BUILD="${CANDIDATE_BUILD:-${default_candidate}}" +PREDICTOR_MODEL="${PREDICTOR_MODEL:-/home/lucebox5/tool-spec-cpu-20260813/models/Qwen3-0.6B-Q8_0.gguf}" +PREDICTOR_IPC_BIN="${PREDICTOR_IPC_BIN:-${CANDIDATE_BUILD}/backend_ipc_daemon}" +PREDICTOR_GPU="${PREDICTOR_GPU:-1}" +PREDICTOR_MAX_CTX="${PREDICTOR_MAX_CTX:-4096}" +PREDICTOR_MAX_TOKENS="${PREDICTOR_MAX_TOKENS:-256}" +PREDICTOR_TIMEOUT_MS="${PREDICTOR_TIMEOUT_MS:-2000}" +PREDICTOR_CONFIDENCE="${PREDICTOR_CONFIDENCE:-0.75}" +# The qualified 0731 launcher disables caches for cold throughput benchmarks. +# Tool-using agent loops need turn-boundary reuse; this later CLI flag wins +# without modifying the qualified model/DSpark arguments. +PREFIX_CACHE_SLOTS_OVERRIDE="${PREFIX_CACHE_SLOTS_OVERRIDE:-32}" + +cache_args=() +if [[ -n "${PREFIX_CACHE_SLOTS_OVERRIDE}" ]]; then + [[ "${PREFIX_CACHE_SLOTS_OVERRIDE}" =~ ^(0|[1-9][0-9]*)$ ]] || { + printf 'invalid PREFIX_CACHE_SLOTS_OVERRIDE: %s\n' \ + "${PREFIX_CACHE_SLOTS_OVERRIDE}" >&2 + exit 2 + } + (( PREFIX_CACHE_SLOTS_OVERRIDE <= 64 )) || { + printf 'PREFIX_CACHE_SLOTS_OVERRIDE exceeds the 64-slot engine limit\n' >&2 + exit 2 + } + cache_args+=(--prefix-cache-slots "${PREFIX_CACHE_SLOTS_OVERRIDE}") +fi + +for binary in "${CANDIDATE_BUILD}/dflash_server" "${PREDICTOR_IPC_BIN}"; do + [[ -f "${binary}" && -x "${binary}" ]] || { + printf 'Qwen tool-predictor binary is not executable: %s\n' "${binary}" >&2 + exit 2 + } +done +[[ -f "${PREDICTOR_MODEL}" ]] || { + printf 'Qwen tool-predictor model is not a regular file: %s\n' \ + "${PREDICTOR_MODEL}" >&2 + exit 2 +} + +export LD_LIBRARY_PATH="${CANDIDATE_BUILD}/deps/llama.cpp/ggml/src:${CANDIDATE_BUILD}/deps/llama.cpp/ggml/src/ggml-hip:${LD_LIBRARY_PATH:-}" +export LUCE_MMVQ_MAX_NCOLS=5 + +exec "${CANDIDATE_BUILD}/dflash_server" "$@" \ + --tool-hint-native-model "${PREDICTOR_MODEL}" \ + --tool-hint-native-ipc-bin "${PREDICTOR_IPC_BIN}" \ + --tool-hint-native-gpu "${PREDICTOR_GPU}" \ + --tool-hint-native-max-ctx "${PREDICTOR_MAX_CTX}" \ + --tool-hint-max-tokens "${PREDICTOR_MAX_TOKENS}" \ + --tool-hint-timeout-ms "${PREDICTOR_TIMEOUT_MS}" \ + --tool-hint-execution-confidence "${PREDICTOR_CONFIDENCE}" \ + "${cache_args[@]}" diff --git a/optimizations/ooo_spec_lucebox5_cpu/profiles/lucebox5-cpu-lane-qualified.json b/optimizations/ooo_spec_lucebox5_cpu/profiles/lucebox5-cpu-lane-qualified.json new file mode 100644 index 000000000..64be189a5 --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/profiles/lucebox5-cpu-lane-qualified.json @@ -0,0 +1,70 @@ +{ + "executor": "child_process_cpu_affinity", + "path_summary": { + "100": { + "accelerator_relation": "non_accelerator", + "decode_interference_qualified": true, + "hit": { + "control_task_mean_ms": 5581.2723303339835, + "model_slowdown_percent": 0.11341375399527287, + "speculative_task_mean_ms": 2973.236727998786 + }, + "miss": { + "control_task_mean_ms": 5581.2723303339835, + "model_slowdown_percent": 0.11341375399527287, + "speculative_task_mean_ms": 5550.103543003206 + } + } + }, + "profile_kind": "disjoint_strix_cpu_sparse_compute", + "profile_status": "qualified", + "qualification": { + "checks": { + "direct_speedup": true, + "disjoint_cpu_affinity": true, + "ds4_active": true, + "identical_model_outputs": true, + "identical_tool_outputs": true, + "model_slowdown": true, + "private_miss_result_hidden": true + }, + "private_miss_evidence": "results/trace-compiled-engine-qwen-production-6pairs-compact.json", + "host": "lucebox5", + "model_cpu_affinity": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29 + ], + "tool_cpu_affinity": [ + 14, + 15, + 30, + 31 + ] + } +} diff --git a/optimizations/ooo_spec_lucebox5_cpu/results/multiturn-cached-wordref-production-6tasks.json b/optimizations/ooo_spec_lucebox5_cpu/results/multiturn-cached-wordref-production-6tasks.json new file mode 100644 index 000000000..5bcaa9d9f --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/results/multiturn-cached-wordref-production-6tasks.json @@ -0,0 +1,5595 @@ +{ + "host": "lucebox5", + "methodology": { + "arm_order": "randomized per task", + "control": "DS4+DSpark generates each call, then the authoritative tool runs", + "measured_wall_time": "initial request through every dependent model/tool turn and final answer", + "model_seed": 814, + "oracle_prediction": false, + "per_task_cache_warmups": 1, + "semantic_token_injection": false, + "speculative": "Qwen3-0.6B predicts each call, launches the private CPU tool before DS4+DSpark, and commits only after exact target verification", + "warmup_pairs": 0 + }, + "pairs": [ + { + "arm_order": [ + "speculative", + "control" + ], + "control": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 3, + "completion_tokens": 153, + "decode_ms": 8322.7, + "expected_final": "workflow_complete:coral", + "exposed_tool_wait_ms": 6080.452936002985, + "final": { + "accept_rate": 0.625, + "assistant_message": { + "content": "workflow_complete:coral", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 804, + "call": null, + "canonical_call": null, + "completion_tokens": 7, + "content": "workflow_complete:coral", + "content_sha256": "1b68642fe5b149f128d3b8b3eff6391869b30634dc6b45bad25d878cb644f588", + "decode_ms": 434.6, + "decode_tokens_per_sec": 16.1, + "model_compute_ms": 10982.9, + "prefill_ms": 10548.3, + "prefilled_tokens": 245, + "request_wall_ms": 11004.08883899945, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 32930.0, + "prediction_hits": 0, + "predictor_wall_ms": 0.0, + "steps": [ + { + "accept_rate": 0.59375, + "assistant_message": { + "content": "I need to process the customer for destination Rome. Let me start with the first step.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-0@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_d8a0080f5f9a77cfc6750bd5", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 602, + "call": { + "arguments": { + "customer_email": "agent-benchmark-0@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-0@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 55, + "content": "I need to process the customer for destination Rome. Let me start with the first step.\n\n", + "content_sha256": "e16dd4db257f04920bcbb955a31511f0a7f9a9e1190fa46726d89b9a1907b2a1", + "decode_ms": 3256.7, + "decode_tokens_per_sec": 16.9, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-0@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 2026.3470799982315, + "model_compute_ms": 3256.7, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 3269.99559700198, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_d8a0080f5f9a77cfc6750bd5", + "tool_result": { + "call_ref": "jade", + "call_sha256": "fa84028f950e9e7f0c3c6265db076d178d96fdb6a5b0d34ac8e14f2982d3e3e4", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1487330009695, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 2026.3470799982315 + }, + { + "accept_rate": 0.6346153616905212, + "assistant_message": { + "content": "Now I need to list open orders using the customer_ref from the previous step.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"jade\"}", + "name": "list_open_orders" + }, + "id": "call_fc905ce6914aa15ba13e127f", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 658, + "call": { + "arguments": { + "customer_ref": "jade" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"jade\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 46, + "content": "Now I need to list open orders using the customer_ref from the previous step.\n\n", + "content_sha256": "7de2bb2ccde5cf6b6fc9c84c9ba2744046323df3d0313b5d3cebc00f881a5dac", + "decode_ms": 2718.8, + "decode_tokens_per_sec": 16.9, + "expected_call": { + "arguments": { + "customer_ref": "jade" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 2026.6102390014566, + "model_compute_ms": 6878.0, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 4159.2, + "prefilled_tokens": 98, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6897.866800001793, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_fc905ce6914aa15ba13e127f", + "tool_result": { + "call_ref": "silver", + "call_sha256": "a2696e6bea709259ca1771ea3d90facbabda20e73fed8d3c3b33efe2d39719d9", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1448610018997, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 2026.6102390014566 + }, + { + "accept_rate": 0.7727272510528564, + "assistant_message": { + "content": "Now I need to get order details using the orders_ref from the previous step.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"silver\"}", + "name": "get_order_details" + }, + "id": "call_ef21ac0bd404a2e1e7ac2012", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 658, + "call": { + "arguments": { + "orders_ref": "silver" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"silver\"},\"name\":\"get_order_details\"}", + "completion_tokens": 45, + "content": "Now I need to get order details using the orders_ref from the previous step.\n\n", + "content_sha256": "53d3157b38fb7ba68290dce87e61a6c6a519446016f2db0acb7f5041229612de", + "decode_ms": 1912.6, + "decode_tokens_per_sec": 23.5, + "expected_call": { + "arguments": { + "orders_ref": "silver" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 2027.4956170032965, + "model_compute_ms": 11812.4, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 9899.8, + "prefilled_tokens": 236, + "private_result_exposed_on_miss": false, + "request_wall_ms": 11832.999707999988, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_ef21ac0bd404a2e1e7ac2012", + "tool_result": { + "call_ref": "coral", + "call_sha256": "9acecd135ddfeab105553c2cb4a4fb60b47aeb0a24152723e4e319f35fe55c46", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.4064589993504, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 2027.4956170032965 + } + ], + "task_id": "dependent_workflow_000", + "task_ms": 39086.805807997735 + }, + "pair_index": 0, + "speculative": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 3, + "completion_tokens": 153, + "decode_ms": 8348.0, + "expected_final": "workflow_complete:coral", + "exposed_tool_wait_ms": 0.062267, + "final": { + "accept_rate": 0.625, + "assistant_message": { + "content": "workflow_complete:coral", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 804, + "call": null, + "canonical_call": null, + "completion_tokens": 7, + "content": "workflow_complete:coral", + "content_sha256": "1b68642fe5b149f128d3b8b3eff6391869b30634dc6b45bad25d878cb644f588", + "decode_ms": 437.1, + "decode_tokens_per_sec": 16.0, + "model_compute_ms": 10953.800000000001, + "prefill_ms": 10516.7, + "prefilled_tokens": 245, + "request_wall_ms": 10969.198104998213, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 32847.1, + "prediction_hits": 3, + "predictor_wall_ms": 860.884941, + "steps": [ + { + "accept_rate": 0.59375, + "assistant_message": { + "content": "I need to process the customer for destination Rome. Let me start with the first step.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-0@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_2e555e8ccc5b1932edd87ac2", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 602, + "call": { + "arguments": { + "customer_email": "agent-benchmark-0@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-0@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 55, + "content": "I need to process the customer for destination Rome. Let me start with the first step.\n\n", + "content_sha256": "e16dd4db257f04920bcbb955a31511f0a7f9a9e1190fa46726d89b9a1907b2a1", + "decode_ms": 3261.8, + "decode_tokens_per_sec": 16.9, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-0@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 0.020609, + "model_compute_ms": 3261.8, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-0@example.test" + }, + "name": "resolve_customer" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 285.480545, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 3561.9963290009764, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_2e555e8ccc5b1932edd87ac2", + "commit_signal_sent": false, + "commit_wait_ms": 0.020609, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 3274.570428, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-0@example.test" + }, + "name": "resolve_customer" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 285.480545, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "jade", + "call_sha256": "fa84028f950e9e7f0c3c6265db076d178d96fdb6a5b0d34ac8e14f2982d3e3e4", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1391930018144, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "status": "hit" + }, + "tool_call_id": "call_2e555e8ccc5b1932edd87ac2", + "tool_result": { + "call_ref": "jade", + "call_sha256": "fa84028f950e9e7f0c3c6265db076d178d96fdb6a5b0d34ac8e14f2982d3e3e4", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1391930018144, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 3274.570428 + }, + { + "accept_rate": 0.6346153616905212, + "assistant_message": { + "content": "Now I need to list open orders using the customer_ref from the previous step.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"jade\"}", + "name": "list_open_orders" + }, + "id": "call_1f8ca05cb9c76e5c84513ca1", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 658, + "call": { + "arguments": { + "customer_ref": "jade" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"jade\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 46, + "content": "Now I need to list open orders using the customer_ref from the previous step.\n\n", + "content_sha256": "7de2bb2ccde5cf6b6fc9c84c9ba2744046323df3d0313b5d3cebc00f881a5dac", + "decode_ms": 2733.6, + "decode_tokens_per_sec": 16.8, + "expected_call": { + "arguments": { + "customer_ref": "jade" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 0.021571, + "model_compute_ms": 6851.0, + "prediction": { + "arguments": { + "customer_ref": "jade" + }, + "name": "list_open_orders" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 278.107146, + "prefill_ms": 4117.4, + "prefilled_tokens": 98, + "private_result_exposed_on_miss": false, + "request_wall_ms": 7143.900198996562, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_1f8ca05cb9c76e5c84513ca1", + "commit_signal_sent": false, + "commit_wait_ms": 0.021571, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6863.607577, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_ref": "jade" + }, + "name": "list_open_orders" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 278.107146, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "silver", + "call_sha256": "a2696e6bea709259ca1771ea3d90facbabda20e73fed8d3c3b33efe2d39719d9", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.0727330007066, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "status": "hit" + }, + "tool_call_id": "call_1f8ca05cb9c76e5c84513ca1", + "tool_result": { + "call_ref": "silver", + "call_sha256": "a2696e6bea709259ca1771ea3d90facbabda20e73fed8d3c3b33efe2d39719d9", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.0727330007066, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 6863.607577 + }, + { + "accept_rate": 0.7727272510528564, + "assistant_message": { + "content": "Now I need to get order details using the orders_ref from the previous step.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"silver\"}", + "name": "get_order_details" + }, + "id": "call_5616db367d592ec04fcd50de", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 658, + "call": { + "arguments": { + "orders_ref": "silver" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"silver\"},\"name\":\"get_order_details\"}", + "completion_tokens": 45, + "content": "Now I need to get order details using the orders_ref from the previous step.\n\n", + "content_sha256": "53d3157b38fb7ba68290dce87e61a6c6a519446016f2db0acb7f5041229612de", + "decode_ms": 1915.5, + "decode_tokens_per_sec": 23.5, + "expected_call": { + "arguments": { + "orders_ref": "silver" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 0.020087, + "model_compute_ms": 11780.5, + "prediction": { + "arguments": { + "orders_ref": "silver" + }, + "name": "get_order_details" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 297.29725, + "prefill_ms": 9865.0, + "prefilled_tokens": 236, + "private_result_exposed_on_miss": false, + "request_wall_ms": 12093.058860999008, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_5616db367d592ec04fcd50de", + "commit_signal_sent": false, + "commit_wait_ms": 0.020087, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 11793.382272, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "orders_ref": "silver" + }, + "name": "get_order_details" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 297.29725, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "coral", + "call_sha256": "9acecd135ddfeab105553c2cb4a4fb60b47aeb0a24152723e4e319f35fe55c46", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.0927090004552, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "status": "hit" + }, + "tool_call_id": "call_5616db367d592ec04fcd50de", + "tool_result": { + "call_ref": "coral", + "call_sha256": "9acecd135ddfeab105553c2cb4a4fb60b47aeb0a24152723e4e319f35fe55c46", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.0927090004552, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 11793.382272 + } + ], + "task_id": "dependent_workflow_000", + "task_ms": 33768.915132000984 + }, + "task": { + "call_count": 3, + "customer_email": "agent-benchmark-0@example.test", + "destination": "Rome", + "id": "dependent_workflow_000" + } + }, + { + "arm_order": [ + "speculative", + "control" + ], + "control": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 4, + "completion_tokens": 239, + "decode_ms": 10301.2, + "expected_final": "workflow_complete:jade", + "exposed_tool_wait_ms": 8106.0383890035155, + "final": { + "accept_rate": 0.75, + "assistant_message": { + "content": "workflow_complete:jade", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 687, + "call": null, + "canonical_call": null, + "completion_tokens": 8, + "content": "workflow_complete:jade", + "content_sha256": "0abc4136c23738770edf0aed5459816de88d70c9b5011c2e5359bdb52557bf42", + "decode_ms": 450.4, + "decode_tokens_per_sec": 17.8, + "model_compute_ms": 24705.100000000002, + "prefill_ms": 24254.7, + "prefilled_tokens": 564, + "request_wall_ms": 24727.02061599921, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 53200.1, + "prediction_hits": 0, + "predictor_wall_ms": 0.0, + "steps": [ + { + "accept_rate": 0.7678571343421936, + "assistant_message": { + "content": "I need to start with step 1: resolve_customer using the customer email from the request.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-1@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_dc82c3b16a83eb8eb0abf576", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 627, + "call": { + "arguments": { + "customer_email": "agent-benchmark-1@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-1@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 58, + "content": "I need to start with step 1: resolve_customer using the customer email from the request.\n\n", + "content_sha256": "9eb32ae482f67f3e7c0fd5f011a2c9c7fb3e76804c484bdf6b40bf8add1aa381", + "decode_ms": 2651.4, + "decode_tokens_per_sec": 21.9, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-1@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 2026.4215749994037, + "model_compute_ms": 2651.4, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 2664.9901300006604, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_dc82c3b16a83eb8eb0abf576", + "tool_result": { + "call_ref": "teal", + "call_sha256": "c42842bc1ac50a52be75d0524ab1a3ac7acd920b2f5cd44f26c522edb56075b7", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.261090001004, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 2026.4215749994037 + }, + { + "accept_rate": 0.7884615659713745, + "assistant_message": { + "content": "Now I need to proceed with step 2: list_open_orders using the customer_ref from the previous result.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"teal\"}", + "name": "list_open_orders" + }, + "id": "call_9de33f3a61b9f1b82be801f2", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 686, + "call": { + "arguments": { + "customer_ref": "teal" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"teal\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 54, + "content": "Now I need to proceed with step 2: list_open_orders using the customer_ref from the previous result.\n\n", + "content_sha256": "5f6404b4b7e127c40a28941b02c473bb20d8759aef9a9e50c5baacec7a12f53a", + "decode_ms": 2140.8, + "decode_tokens_per_sec": 25.2, + "expected_call": { + "arguments": { + "customer_ref": "teal" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 2026.7217100008565, + "model_compute_ms": 6317.6, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 4176.8, + "prefilled_tokens": 94, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6337.450651000836, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_9de33f3a61b9f1b82be801f2", + "tool_result": { + "call_ref": "maple", + "call_sha256": "5f7c0536c4a387c5334ef97c8834cd3472841f24b358d7de06412ab784c031f6", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1781250030035, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 2026.7217100008565 + }, + { + "accept_rate": 0.7115384340286255, + "assistant_message": { + "content": "Now I proceed to step 3: get_order_details using the orders_ref from the previous result.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"maple\"}", + "name": "get_order_details" + }, + "id": "call_c389a4a304a4855583d096e3", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 835, + "call": { + "arguments": { + "orders_ref": "maple" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"maple\"},\"name\":\"get_order_details\"}", + "completion_tokens": 50, + "content": "Now I proceed to step 3: get_order_details using the orders_ref from the previous result.\n\n", + "content_sha256": "bc190728cc301be15a35259dc01ce8c5880575dc6bfcc34e09e1382ec03aa00b", + "decode_ms": 2395.8, + "decode_tokens_per_sec": 20.9, + "expected_call": { + "arguments": { + "orders_ref": "maple" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 2027.0823430000746, + "model_compute_ms": 6656.7, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 4260.9, + "prefilled_tokens": 99, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6677.666080002382, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_c389a4a304a4855583d096e3", + "tool_result": { + "call_ref": "teal", + "call_sha256": "b390ab8cab21b6aa9ac895e1c383fbeac5835b056f6158ee5bac5384eee6c90c", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.0883639986569, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 2027.0823430000746 + }, + { + "accept_rate": 0.828125, + "assistant_message": { + "content": "Now I proceed to step 4: calculate_shipping using the order_ref from the previous result and the destination from the user request.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"destination\":\"Milan\",\"order_ref\":\"teal\"}", + "name": "calculate_shipping" + }, + "id": "call_0ee1078a3640325878166687", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 835, + "call": { + "arguments": { + "destination": "Milan", + "order_ref": "teal" + }, + "name": "calculate_shipping" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"destination\":\"Milan\",\"order_ref\":\"teal\"},\"name\":\"calculate_shipping\"}", + "completion_tokens": 69, + "content": "Now I proceed to step 4: calculate_shipping using the order_ref from the previous result and the destination from the user request.\n\n", + "content_sha256": "eeddf10343ca1f79fbf9009b3ce6ecb712a9d084d51a516f0d84e085b8e69f56", + "decode_ms": 2662.8, + "decode_tokens_per_sec": 25.9, + "expected_call": { + "arguments": { + "destination": "Milan", + "order_ref": "teal" + }, + "name": "calculate_shipping" + }, + "exposed_tool_wait_ms": 2025.8127610031806, + "model_compute_ms": 12869.3, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 10206.5, + "prefilled_tokens": 240, + "private_result_exposed_on_miss": false, + "request_wall_ms": 12891.017510002712, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_0ee1078a3640325878166687", + "tool_result": { + "call_ref": "jade", + "call_sha256": "75ff43afae4473f01580bb761f28d2e8152af72cb46c29ead46cd8817feeddf7", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1382260015816, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "tool_wall_ms": 2025.8127610031806 + } + ], + "task_id": "dependent_workflow_001", + "task_ms": 61406.28354300134 + }, + "pair_index": 1, + "speculative": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 4, + "completion_tokens": 239, + "decode_ms": 10295.7, + "expected_final": "workflow_complete:jade", + "exposed_tool_wait_ms": 0.089957, + "final": { + "accept_rate": 0.75, + "assistant_message": { + "content": "workflow_complete:jade", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 578, + "call": null, + "canonical_call": null, + "completion_tokens": 8, + "content": "workflow_complete:jade", + "content_sha256": "0abc4136c23738770edf0aed5459816de88d70c9b5011c2e5359bdb52557bf42", + "decode_ms": 446.5, + "decode_tokens_per_sec": 17.9, + "model_compute_ms": 29053.9, + "prefill_ms": 28607.4, + "prefilled_tokens": 673, + "request_wall_ms": 29070.463013002154, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 57444.0, + "prediction_hits": 4, + "predictor_wall_ms": 1266.8827410000001, + "steps": [ + { + "accept_rate": 0.7678571343421936, + "assistant_message": { + "content": "I need to start with step 1: resolve_customer using the customer email from the request.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-1@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_fecf851a9599a5ae4934a551", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 627, + "call": { + "arguments": { + "customer_email": "agent-benchmark-1@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-1@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 58, + "content": "I need to start with step 1: resolve_customer using the customer email from the request.\n\n", + "content_sha256": "9eb32ae482f67f3e7c0fd5f011a2c9c7fb3e76804c484bdf6b40bf8add1aa381", + "decode_ms": 2653.3, + "decode_tokens_per_sec": 21.9, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-1@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 0.019887, + "model_compute_ms": 2653.3, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-1@example.test" + }, + "name": "resolve_customer" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 290.616952, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 2958.456275999197, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_fecf851a9599a5ae4934a551", + "commit_signal_sent": false, + "commit_wait_ms": 0.019887, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 2665.994315, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-1@example.test" + }, + "name": "resolve_customer" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 290.616952, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "teal", + "call_sha256": "c42842bc1ac50a52be75d0524ab1a3ac7acd920b2f5cd44f26c522edb56075b7", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.2695100010897, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "status": "hit" + }, + "tool_call_id": "call_fecf851a9599a5ae4934a551", + "tool_result": { + "call_ref": "teal", + "call_sha256": "c42842bc1ac50a52be75d0524ab1a3ac7acd920b2f5cd44f26c522edb56075b7", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.2695100010897, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 2665.994315 + }, + { + "accept_rate": 0.7884615659713745, + "assistant_message": { + "content": "Now I need to proceed with step 2: list_open_orders using the customer_ref from the previous result.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"teal\"}", + "name": "list_open_orders" + }, + "id": "call_bacb659055ea0a6988c7a8eb", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 686, + "call": { + "arguments": { + "customer_ref": "teal" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"teal\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 54, + "content": "Now I need to proceed with step 2: list_open_orders using the customer_ref from the previous result.\n\n", + "content_sha256": "5f6404b4b7e127c40a28941b02c473bb20d8759aef9a9e50c5baacec7a12f53a", + "decode_ms": 2146.2, + "decode_tokens_per_sec": 25.2, + "expected_call": { + "arguments": { + "customer_ref": "teal" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 0.020778, + "model_compute_ms": 6288.3, + "prediction": { + "arguments": { + "customer_ref": "teal" + }, + "name": "list_open_orders" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 283.382872, + "prefill_ms": 4142.1, + "prefilled_tokens": 94, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6587.413663000916, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_bacb659055ea0a6988c7a8eb", + "commit_signal_sent": false, + "commit_wait_ms": 0.020778, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6301.160629, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_ref": "teal" + }, + "name": "list_open_orders" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 283.382872, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "maple", + "call_sha256": "5f7c0536c4a387c5334ef97c8834cd3472841f24b358d7de06412ab784c031f6", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1142179971794, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "status": "hit" + }, + "tool_call_id": "call_bacb659055ea0a6988c7a8eb", + "tool_result": { + "call_ref": "maple", + "call_sha256": "5f7c0536c4a387c5334ef97c8834cd3472841f24b358d7de06412ab784c031f6", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1142179971794, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 6301.160629 + }, + { + "accept_rate": 0.7115384340286255, + "assistant_message": { + "content": "Now I proceed to step 3: get_order_details using the orders_ref from the previous result.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"maple\"}", + "name": "get_order_details" + }, + "id": "call_6d5d5875cd96a358f7046361", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 835, + "call": { + "arguments": { + "orders_ref": "maple" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"maple\"},\"name\":\"get_order_details\"}", + "completion_tokens": 50, + "content": "Now I proceed to step 3: get_order_details using the orders_ref from the previous result.\n\n", + "content_sha256": "bc190728cc301be15a35259dc01ce8c5880575dc6bfcc34e09e1382ec03aa00b", + "decode_ms": 2393.8, + "decode_tokens_per_sec": 20.9, + "expected_call": { + "arguments": { + "orders_ref": "maple" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 0.019486, + "model_compute_ms": 6621.3, + "prediction": { + "arguments": { + "orders_ref": "maple" + }, + "name": "get_order_details" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 314.519747, + "prefill_ms": 4227.5, + "prefilled_tokens": 99, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6952.102143000957, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_6d5d5875cd96a358f7046361", + "commit_signal_sent": false, + "commit_wait_ms": 0.019486, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6634.442859, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "orders_ref": "maple" + }, + "name": "get_order_details" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 314.519747, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "teal", + "call_sha256": "b390ab8cab21b6aa9ac895e1c383fbeac5835b056f6158ee5bac5384eee6c90c", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.0938069970289, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "status": "hit" + }, + "tool_call_id": "call_6d5d5875cd96a358f7046361", + "tool_result": { + "call_ref": "teal", + "call_sha256": "b390ab8cab21b6aa9ac895e1c383fbeac5835b056f6158ee5bac5384eee6c90c", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.0938069970289, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 6634.442859 + }, + { + "accept_rate": 0.828125, + "assistant_message": { + "content": "Now I proceed to step 4: calculate_shipping using the order_ref from the previous result and the destination from the user request.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"destination\":\"Milan\",\"order_ref\":\"teal\"}", + "name": "calculate_shipping" + }, + "id": "call_e465f52877b49a434aa95da7", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 835, + "call": { + "arguments": { + "destination": "Milan", + "order_ref": "teal" + }, + "name": "calculate_shipping" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"destination\":\"Milan\",\"order_ref\":\"teal\"},\"name\":\"calculate_shipping\"}", + "completion_tokens": 69, + "content": "Now I proceed to step 4: calculate_shipping using the order_ref from the previous result and the destination from the user request.\n\n", + "content_sha256": "eeddf10343ca1f79fbf9009b3ce6ecb712a9d084d51a516f0d84e085b8e69f56", + "decode_ms": 2655.9, + "decode_tokens_per_sec": 26.0, + "expected_call": { + "arguments": { + "destination": "Milan", + "order_ref": "teal" + }, + "name": "calculate_shipping" + }, + "exposed_tool_wait_ms": 0.029806, + "model_compute_ms": 12827.199999999999, + "prediction": { + "arguments": { + "destination": "Milan", + "order_ref": "teal" + }, + "name": "calculate_shipping" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 378.36317, + "prefill_ms": 10171.3, + "prefilled_tokens": 240, + "private_result_exposed_on_miss": false, + "request_wall_ms": 13222.511880001548, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_e465f52877b49a434aa95da7", + "commit_signal_sent": false, + "commit_wait_ms": 0.029806, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 12840.559652, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "destination": "Milan", + "order_ref": "teal" + }, + "name": "calculate_shipping" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 378.36317, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "jade", + "call_sha256": "75ff43afae4473f01580bb761f28d2e8152af72cb46c29ead46cd8817feeddf7", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1044350028678, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "status": "hit" + }, + "tool_call_id": "call_e465f52877b49a434aa95da7", + "tool_result": { + "call_ref": "jade", + "call_sha256": "75ff43afae4473f01580bb761f28d2e8152af72cb46c29ead46cd8817feeddf7", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1044350028678, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "tool_wall_ms": 12840.559652 + } + ], + "task_id": "dependent_workflow_001", + "task_ms": 58791.965293999965 + }, + "task": { + "call_count": 4, + "customer_email": "agent-benchmark-1@example.test", + "destination": "Milan", + "id": "dependent_workflow_001" + } + }, + { + "arm_order": [ + "speculative", + "control" + ], + "control": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 5, + "completion_tokens": 250, + "decode_ms": 11595.7, + "expected_final": "workflow_complete:ivory", + "exposed_tool_wait_ms": 10131.762909997633, + "final": { + "accept_rate": 0.75, + "assistant_message": { + "content": "workflow_complete:ivory", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 714, + "call": null, + "canonical_call": null, + "completion_tokens": 8, + "content": "workflow_complete:ivory", + "content_sha256": "0496080db67bb468d83025f09f59e8cd14c425908fde8632a3b9ba98c23dc9cf", + "decode_ms": 451.5, + "decode_tokens_per_sec": 17.7, + "model_compute_ms": 29204.3, + "prefill_ms": 28752.8, + "prefilled_tokens": 673, + "request_wall_ms": 29227.80936699928, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 63055.1, + "prediction_hits": 0, + "predictor_wall_ms": 0.0, + "steps": [ + { + "accept_rate": 0.625, + "assistant_message": { + "content": "I need to process this customer request through the 5-step workflow. Let me start with step 1: resolving the customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-2@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_69cd1757544b2a9bd324f10a", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 648, + "call": { + "arguments": { + "customer_email": "agent-benchmark-2@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-2@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 64, + "content": "I need to process this customer request through the 5-step workflow. Let me start with step 1: resolving the customer.\n\n", + "content_sha256": "caac2c887d5589ba57ca98c8c77ea68cdd12f4569133cc824cdc1cc82c831c09", + "decode_ms": 3060.1, + "decode_tokens_per_sec": 20.9, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-2@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 2026.8547260020569, + "model_compute_ms": 3060.1, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 3073.1119080010103, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_69cd1757544b2a9bd324f10a", + "tool_result": { + "call_ref": "plum", + "call_sha256": "93824d1968f2c3ab058a3ef69d2625c0fea88fad3b27b8c30f95418fb83ae6cf", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.3111229998467, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 2026.8547260020569 + }, + { + "accept_rate": 0.7272727489471436, + "assistant_message": { + "content": "Now I need to list open orders for this resolved customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"plum\"}", + "name": "list_open_orders" + }, + "id": "call_50c8ce688d2080917a63f3d4", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 713, + "call": { + "arguments": { + "customer_ref": "plum" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"plum\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 43, + "content": "Now I need to list open orders for this resolved customer.\n\n", + "content_sha256": "3ce7d0f18b723f0f13b497c10e5799d52d899e25e977e82e22a5ba8e25b52e6e", + "decode_ms": 2100.4, + "decode_tokens_per_sec": 20.5, + "expected_call": { + "arguments": { + "customer_ref": "plum" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 2025.9938390008756, + "model_compute_ms": 6204.1, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 4103.7, + "prefilled_tokens": 95, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6224.153801998909, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_50c8ce688d2080917a63f3d4", + "tool_result": { + "call_ref": "jade", + "call_sha256": "772a4b9fde9e69fb1da323ba4c22e7a7d01061c0600c133e74de7bd73bba931f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.209031997656, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 2025.9938390008756 + }, + { + "accept_rate": 0.6363636255264282, + "assistant_message": { + "content": "Now I need to inspect the selected order.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"jade\"}", + "name": "get_order_details" + }, + "id": "call_deb21d13f2d6efbc6f1b841f", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 852, + "call": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"jade\"},\"name\":\"get_order_details\"}", + "completion_tokens": 39, + "content": "Now I need to inspect the selected order.\n\n", + "content_sha256": "0501df1f5ec6fcf19c50a853fe0631d19fcc4dac651569fc7278ecc84ffcdb0c", + "decode_ms": 2273.4, + "decode_tokens_per_sec": 17.2, + "expected_call": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 2026.1670279978716, + "model_compute_ms": 6370.5, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 4097.1, + "prefilled_tokens": 95, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6392.090148001444, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_deb21d13f2d6efbc6f1b841f", + "tool_result": { + "call_ref": "amber", + "call_sha256": "428dbfc05f17e7a9295c2b4741dbae2bb91d3f107150b77bb34c6ca25bd8af23", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.138072999107, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 2026.1670279978716 + }, + { + "accept_rate": 0.7692307829856873, + "assistant_message": { + "content": "Now I need to calculate shipping for the order to Turin.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"destination\":\"Turin\",\"order_ref\":\"amber\"}", + "name": "calculate_shipping" + }, + "id": "call_2854e0d7420f2516ae707775", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 987, + "call": { + "arguments": { + "destination": "Turin", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"destination\":\"Turin\",\"order_ref\":\"amber\"},\"name\":\"calculate_shipping\"}", + "completion_tokens": 53, + "content": "Now I need to calculate shipping for the order to Turin.\n\n", + "content_sha256": "069cd393e5fe66dd10d07d7f7927ada35244e0a01f87cb5e75c95a67b736d0ad", + "decode_ms": 2405.6, + "decode_tokens_per_sec": 22.0, + "expected_call": { + "arguments": { + "destination": "Turin", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "exposed_tool_wait_ms": 2026.3736019987846, + "model_compute_ms": 6388.7, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 3983.1, + "prefilled_tokens": 93, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6411.000621999847, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_2854e0d7420f2516ae707775", + "tool_result": { + "call_ref": "amber", + "call_sha256": "21ec3c60e83bfbc0ab4e6e8356a5c87e7210ffd16b53ca8299017261d97f70a1", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.345896001818, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "tool_wall_ms": 2026.3736019987846 + }, + { + "accept_rate": 0.9444444179534912, + "assistant_message": { + "content": "Now I need to prepare the final customer summary.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"shipping_ref\":\"amber\"}", + "name": "prepare_customer_summary" + }, + "id": "call_b8783a11222c4962dd41d911", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 987, + "call": { + "arguments": { + "shipping_ref": "amber" + }, + "name": "prepare_customer_summary" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"shipping_ref\":\"amber\"},\"name\":\"prepare_customer_summary\"}", + "completion_tokens": 43, + "content": "Now I need to prepare the final customer summary.\n\n", + "content_sha256": "6d12faaaf38ef53fc446a8a34f8fe0f3a22a7bdc6d3dab6ef4f1f25c5ba91b44", + "decode_ms": 1304.7, + "decode_tokens_per_sec": 33.0, + "expected_call": { + "arguments": { + "shipping_ref": "amber" + }, + "name": "prepare_customer_summary" + }, + "exposed_tool_wait_ms": 2026.3737149980443, + "model_compute_ms": 11827.400000000001, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 10522.7, + "prefilled_tokens": 242, + "private_result_exposed_on_miss": false, + "request_wall_ms": 11850.366531998588, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_b8783a11222c4962dd41d911", + "tool_result": { + "call_ref": "ivory", + "call_sha256": "adaa761561a84b7e457acb7881e856285fcdccf21a6d82751dd2bea984b1d6d0", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1452489996154, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "prepare_customer_summary" + }, + "tool_wall_ms": 2026.3737149980443 + } + ], + "task_id": "dependent_workflow_002", + "task_ms": 73312.62574300126 + }, + "pair_index": 2, + "speculative": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 5, + "completion_tokens": 250, + "decode_ms": 11593.5, + "expected_final": "workflow_complete:ivory", + "exposed_tool_wait_ms": 0.125415, + "final": { + "accept_rate": 0.75, + "assistant_message": { + "content": "workflow_complete:ivory", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 603, + "call": null, + "canonical_call": null, + "completion_tokens": 8, + "content": "workflow_complete:ivory", + "content_sha256": "0496080db67bb468d83025f09f59e8cd14c425908fde8632a3b9ba98c23dc9cf", + "decode_ms": 457.7, + "decode_tokens_per_sec": 17.5, + "model_compute_ms": 33725.1, + "prefill_ms": 33267.4, + "prefilled_tokens": 784, + "request_wall_ms": 33742.18803799886, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 67450.6, + "prediction_hits": 5, + "predictor_wall_ms": 1638.720663, + "steps": [ + { + "accept_rate": 0.625, + "assistant_message": { + "content": "I need to process this customer request through the 5-step workflow. Let me start with step 1: resolving the customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-2@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_bec28a4286f77110bf0d49e5", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 648, + "call": { + "arguments": { + "customer_email": "agent-benchmark-2@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-2@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 64, + "content": "I need to process this customer request through the 5-step workflow. Let me start with step 1: resolving the customer.\n\n", + "content_sha256": "caac2c887d5589ba57ca98c8c77ea68cdd12f4569133cc824cdc1cc82c831c09", + "decode_ms": 3061.5, + "decode_tokens_per_sec": 20.9, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-2@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 0.02094, + "model_compute_ms": 3061.5, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-2@example.test" + }, + "name": "resolve_customer" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 295.452857, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 3372.0264969997515, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_bec28a4286f77110bf0d49e5", + "commit_signal_sent": false, + "commit_wait_ms": 0.02094, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 3074.451431, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-2@example.test" + }, + "name": "resolve_customer" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 295.452857, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "plum", + "call_sha256": "93824d1968f2c3ab058a3ef69d2625c0fea88fad3b27b8c30f95418fb83ae6cf", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1503530002083, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "status": "hit" + }, + "tool_call_id": "call_bec28a4286f77110bf0d49e5", + "tool_result": { + "call_ref": "plum", + "call_sha256": "93824d1968f2c3ab058a3ef69d2625c0fea88fad3b27b8c30f95418fb83ae6cf", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1503530002083, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 3074.451431 + }, + { + "accept_rate": 0.7272727489471436, + "assistant_message": { + "content": "Now I need to list open orders for this resolved customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"plum\"}", + "name": "list_open_orders" + }, + "id": "call_01a8a2497c9c63207b596c32", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 713, + "call": { + "arguments": { + "customer_ref": "plum" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"plum\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 43, + "content": "Now I need to list open orders for this resolved customer.\n\n", + "content_sha256": "3ce7d0f18b723f0f13b497c10e5799d52d899e25e977e82e22a5ba8e25b52e6e", + "decode_ms": 2098.9, + "decode_tokens_per_sec": 20.5, + "expected_call": { + "arguments": { + "customer_ref": "plum" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 0.030296, + "model_compute_ms": 6174.5, + "prediction": { + "arguments": { + "customer_ref": "plum" + }, + "name": "list_open_orders" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 287.479375, + "prefill_ms": 4075.6, + "prefilled_tokens": 95, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6477.406130001327, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_01a8a2497c9c63207b596c32", + "commit_signal_sent": false, + "commit_wait_ms": 0.030296, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6187.553086, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_ref": "plum" + }, + "name": "list_open_orders" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 287.479375, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "jade", + "call_sha256": "772a4b9fde9e69fb1da323ba4c22e7a7d01061c0600c133e74de7bd73bba931f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.0936220021686, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "status": "hit" + }, + "tool_call_id": "call_01a8a2497c9c63207b596c32", + "tool_result": { + "call_ref": "jade", + "call_sha256": "772a4b9fde9e69fb1da323ba4c22e7a7d01061c0600c133e74de7bd73bba931f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.0936220021686, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 6187.553086 + }, + { + "accept_rate": 0.6363636255264282, + "assistant_message": { + "content": "Now I need to inspect the selected order.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"jade\"}", + "name": "get_order_details" + }, + "id": "call_210fd40fb9531732179be43e", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 852, + "call": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"jade\"},\"name\":\"get_order_details\"}", + "completion_tokens": 39, + "content": "Now I need to inspect the selected order.\n\n", + "content_sha256": "0501df1f5ec6fcf19c50a853fe0631d19fcc4dac651569fc7278ecc84ffcdb0c", + "decode_ms": 2273.1, + "decode_tokens_per_sec": 17.2, + "expected_call": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 0.020418, + "model_compute_ms": 6347.6, + "prediction": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 314.968808, + "prefill_ms": 4074.5, + "prefilled_tokens": 95, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6678.62192100074, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_210fd40fb9531732179be43e", + "commit_signal_sent": false, + "commit_wait_ms": 0.020418, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6360.937862, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 314.968808, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "amber", + "call_sha256": "428dbfc05f17e7a9295c2b4741dbae2bb91d3f107150b77bb34c6ca25bd8af23", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.123739002447, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "status": "hit" + }, + "tool_call_id": "call_210fd40fb9531732179be43e", + "tool_result": { + "call_ref": "amber", + "call_sha256": "428dbfc05f17e7a9295c2b4741dbae2bb91d3f107150b77bb34c6ca25bd8af23", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.123739002447, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 6360.937862 + }, + { + "accept_rate": 0.7692307829856873, + "assistant_message": { + "content": "Now I need to calculate shipping for the order to Turin.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"destination\":\"Turin\",\"order_ref\":\"amber\"}", + "name": "calculate_shipping" + }, + "id": "call_b3b83d71f87f78b979030f5b", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 987, + "call": { + "arguments": { + "destination": "Turin", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"destination\":\"Turin\",\"order_ref\":\"amber\"},\"name\":\"calculate_shipping\"}", + "completion_tokens": 53, + "content": "Now I need to calculate shipping for the order to Turin.\n\n", + "content_sha256": "069cd393e5fe66dd10d07d7f7927ada35244e0a01f87cb5e75c95a67b736d0ad", + "decode_ms": 2398.6, + "decode_tokens_per_sec": 22.1, + "expected_call": { + "arguments": { + "destination": "Turin", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "exposed_tool_wait_ms": 0.019717, + "model_compute_ms": 6368.7, + "prediction": { + "arguments": { + "destination": "Turin", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 371.829653, + "prefill_ms": 3970.1, + "prefilled_tokens": 93, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6757.145636998757, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_b3b83d71f87f78b979030f5b", + "commit_signal_sent": false, + "commit_wait_ms": 0.019717, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6382.563455, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "destination": "Turin", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 371.829653, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "amber", + "call_sha256": "21ec3c60e83bfbc0ab4e6e8356a5c87e7210ffd16b53ca8299017261d97f70a1", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1194319993374, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "status": "hit" + }, + "tool_call_id": "call_b3b83d71f87f78b979030f5b", + "tool_result": { + "call_ref": "amber", + "call_sha256": "21ec3c60e83bfbc0ab4e6e8356a5c87e7210ffd16b53ca8299017261d97f70a1", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1194319993374, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "tool_wall_ms": 6382.563455 + }, + { + "accept_rate": 0.9444444179534912, + "assistant_message": { + "content": "Now I need to prepare the final customer summary.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"shipping_ref\":\"amber\"}", + "name": "prepare_customer_summary" + }, + "id": "call_832c53b5fcceba769236c378", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 987, + "call": { + "arguments": { + "shipping_ref": "amber" + }, + "name": "prepare_customer_summary" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"shipping_ref\":\"amber\"},\"name\":\"prepare_customer_summary\"}", + "completion_tokens": 43, + "content": "Now I need to prepare the final customer summary.\n\n", + "content_sha256": "6d12faaaf38ef53fc446a8a34f8fe0f3a22a7bdc6d3dab6ef4f1f25c5ba91b44", + "decode_ms": 1303.7, + "decode_tokens_per_sec": 33.0, + "expected_call": { + "arguments": { + "shipping_ref": "amber" + }, + "name": "prepare_customer_summary" + }, + "exposed_tool_wait_ms": 0.034044, + "model_compute_ms": 11773.2, + "prediction": { + "arguments": { + "shipping_ref": "amber" + }, + "name": "prepare_customer_summary" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 368.98997, + "prefill_ms": 10469.5, + "prefilled_tokens": 242, + "private_result_exposed_on_miss": false, + "request_wall_ms": 12160.27835999921, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_832c53b5fcceba769236c378", + "commit_signal_sent": false, + "commit_wait_ms": 0.034044, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 11787.198031, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "shipping_ref": "amber" + }, + "name": "prepare_customer_summary" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 368.98997, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "ivory", + "call_sha256": "adaa761561a84b7e457acb7881e856285fcdccf21a6d82751dd2bea984b1d6d0", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1132219986175, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "prepare_customer_summary" + }, + "status": "hit" + }, + "tool_call_id": "call_832c53b5fcceba769236c378", + "tool_result": { + "call_ref": "ivory", + "call_sha256": "adaa761561a84b7e457acb7881e856285fcdccf21a6d82751dd2bea984b1d6d0", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1132219986175, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "prepare_customer_summary" + }, + "tool_wall_ms": 11787.198031 + } + ], + "task_id": "dependent_workflow_002", + "task_ms": 69188.68444200052 + }, + "task": { + "call_count": 5, + "customer_email": "agent-benchmark-2@example.test", + "destination": "Turin", + "id": "dependent_workflow_002" + } + }, + { + "arm_order": [ + "speculative", + "control" + ], + "control": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 3, + "completion_tokens": 140, + "decode_ms": 8523.9, + "expected_final": "workflow_complete:amber", + "exposed_tool_wait_ms": 6080.720226997073, + "final": { + "accept_rate": 0.625, + "assistant_message": { + "content": "workflow_complete:amber", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 791, + "call": null, + "canonical_call": null, + "completion_tokens": 7, + "content": "workflow_complete:amber", + "content_sha256": "432bcc6fd099a0a7a97da30984ac9e2d3760ca9fc31f5b74162f2961c04c83b8", + "decode_ms": 440.8, + "decode_tokens_per_sec": 15.9, + "model_compute_ms": 10922.3, + "prefill_ms": 10481.5, + "prefilled_tokens": 241, + "request_wall_ms": 10943.100953001704, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 32427.3, + "prediction_hits": 0, + "predictor_wall_ms": 0.0, + "steps": [ + { + "accept_rate": 0.59375, + "assistant_message": { + "content": "I need to process the customer for destination Bologna. Let me start by resolving the customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-3@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_e984fc331504596163ab27cf", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 602, + "call": { + "arguments": { + "customer_email": "agent-benchmark-3@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-3@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 55, + "content": "I need to process the customer for destination Bologna. Let me start by resolving the customer.\n\n", + "content_sha256": "62c201aea98c566f6e95b6774e46fea965dc0ccd0e7b299ff5e2a3a96a5226d4", + "decode_ms": 3286.9, + "decode_tokens_per_sec": 16.7, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-3@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 2026.4642199981608, + "model_compute_ms": 3286.9, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 3299.9924879986793, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_e984fc331504596163ab27cf", + "tool_result": { + "call_ref": "azure", + "call_sha256": "102e55312f96c8ec8207cda555ac4117176de2eed12b6b829409dd9ce2f3cecd", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1172100019176, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 2026.4642199981608 + }, + { + "accept_rate": 0.5833333134651184, + "assistant_message": { + "content": "Now I need to list open orders for this customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"azure\"}", + "name": "list_open_orders" + }, + "id": "call_697fd06608e2005b6913881d", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 658, + "call": { + "arguments": { + "customer_ref": "azure" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"azure\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 40, + "content": "Now I need to list open orders for this customer.\n\n", + "content_sha256": "083d3fef5ad5bbc72a6d4d9dbd01570eb5d080efc3116dfb0e78467825ef2ae0", + "decode_ms": 2335.8, + "decode_tokens_per_sec": 17.1, + "expected_call": { + "arguments": { + "customer_ref": "azure" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 2026.8712299985054, + "model_compute_ms": 6218.8, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 3883.0, + "prefilled_tokens": 91, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6240.539802998683, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_697fd06608e2005b6913881d", + "tool_result": { + "call_ref": "jade", + "call_sha256": "f9bb9bbfe57b8faa80fd5f25ba23eb97ae6031ad8e254049f174e1072d96142f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.085728999693, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 2026.8712299985054 + }, + { + "accept_rate": 0.6136363744735718, + "assistant_message": { + "content": "Now I need to get the order details.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"jade\"}", + "name": "get_order_details" + }, + "id": "call_fc74542559f704d7895bd360", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 658, + "call": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"jade\"},\"name\":\"get_order_details\"}", + "completion_tokens": 38, + "content": "Now I need to get the order details.\n\n", + "content_sha256": "4855a9d3760fe1217b6d0b8d6dd52f84ec1cb58b5a1b797b630e5dc9519a5417", + "decode_ms": 2460.4, + "decode_tokens_per_sec": 15.4, + "expected_call": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 2027.3847770004068, + "model_compute_ms": 11999.3, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 9538.9, + "prefilled_tokens": 226, + "private_result_exposed_on_miss": false, + "request_wall_ms": 12019.706646999111, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_fc74542559f704d7895bd360", + "tool_result": { + "call_ref": "amber", + "call_sha256": "428dbfc05f17e7a9295c2b4741dbae2bb91d3f107150b77bb34c6ca25bd8af23", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.4602429980878, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 2027.3847770004068 + } + ], + "task_id": "dependent_workflow_003", + "task_ms": 38585.75580899924 + }, + "pair_index": 3, + "speculative": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 3, + "completion_tokens": 140, + "decode_ms": 9557.2, + "expected_final": "workflow_complete:amber", + "exposed_tool_wait_ms": 0.071994, + "final": { + "accept_rate": 0.625, + "assistant_message": { + "content": "workflow_complete:amber", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 659, + "call": null, + "canonical_call": null, + "completion_tokens": 7, + "content": "workflow_complete:amber", + "content_sha256": "432bcc6fd099a0a7a97da30984ac9e2d3760ca9fc31f5b74162f2961c04c83b8", + "decode_ms": 437.2, + "decode_tokens_per_sec": 16.0, + "model_compute_ms": 16659.4, + "prefill_ms": 16222.2, + "prefilled_tokens": 373, + "request_wall_ms": 16674.765046001994, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 39114.100000000006, + "prediction_hits": 3, + "predictor_wall_ms": 858.5581070000001, + "steps": [ + { + "accept_rate": 0.59375, + "assistant_message": { + "content": "I need to process the customer for destination Bologna. Let me start by resolving the customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-3@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_83b7681c10a779328b97cd10", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 602, + "call": { + "arguments": { + "customer_email": "agent-benchmark-3@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-3@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 55, + "content": "I need to process the customer for destination Bologna. Let me start by resolving the customer.\n\n", + "content_sha256": "62c201aea98c566f6e95b6774e46fea965dc0ccd0e7b299ff5e2a3a96a5226d4", + "decode_ms": 4325.2, + "decode_tokens_per_sec": 12.7, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-3@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 0.022101, + "model_compute_ms": 4325.2, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-3@example.test" + }, + "name": "resolve_customer" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 286.040186, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 4625.921421000385, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_83b7681c10a779328b97cd10", + "commit_signal_sent": false, + "commit_wait_ms": 0.022101, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 4338.023602, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-3@example.test" + }, + "name": "resolve_customer" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 286.040186, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "azure", + "call_sha256": "102e55312f96c8ec8207cda555ac4117176de2eed12b6b829409dd9ce2f3cecd", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.2032990014413, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "status": "hit" + }, + "tool_call_id": "call_83b7681c10a779328b97cd10", + "tool_result": { + "call_ref": "azure", + "call_sha256": "102e55312f96c8ec8207cda555ac4117176de2eed12b6b829409dd9ce2f3cecd", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.2032990014413, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 4338.023602 + }, + { + "accept_rate": 0.5833333134651184, + "assistant_message": { + "content": "Now I need to list open orders for this customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"azure\"}", + "name": "list_open_orders" + }, + "id": "call_750886b39d7cdf0d6340234d", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 658, + "call": { + "arguments": { + "customer_ref": "azure" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"azure\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 40, + "content": "Now I need to list open orders for this customer.\n\n", + "content_sha256": "083d3fef5ad5bbc72a6d4d9dbd01570eb5d080efc3116dfb0e78467825ef2ae0", + "decode_ms": 2334.0, + "decode_tokens_per_sec": 17.1, + "expected_call": { + "arguments": { + "customer_ref": "azure" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 0.019967, + "model_compute_ms": 6185.3, + "prediction": { + "arguments": { + "customer_ref": "azure" + }, + "name": "list_open_orders" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 270.206183, + "prefill_ms": 3851.3, + "prefilled_tokens": 91, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6504.486581998208, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_750886b39d7cdf0d6340234d", + "commit_signal_sent": false, + "commit_wait_ms": 0.019967, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6231.715703, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_ref": "azure" + }, + "name": "list_open_orders" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 270.206183, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "jade", + "call_sha256": "f9bb9bbfe57b8faa80fd5f25ba23eb97ae6031ad8e254049f174e1072d96142f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.112101999548, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "status": "hit" + }, + "tool_call_id": "call_750886b39d7cdf0d6340234d", + "tool_result": { + "call_ref": "jade", + "call_sha256": "f9bb9bbfe57b8faa80fd5f25ba23eb97ae6031ad8e254049f174e1072d96142f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.112101999548, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 6231.715703 + }, + { + "accept_rate": 0.6136363744735718, + "assistant_message": { + "content": "Now I need to get the order details.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"jade\"}", + "name": "get_order_details" + }, + "id": "call_b1ec97e6ea85bf4ef9afb4d5", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 658, + "call": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"jade\"},\"name\":\"get_order_details\"}", + "completion_tokens": 38, + "content": "Now I need to get the order details.\n\n", + "content_sha256": "4855a9d3760fe1217b6d0b8d6dd52f84ec1cb58b5a1b797b630e5dc9519a5417", + "decode_ms": 2460.8, + "decode_tokens_per_sec": 15.4, + "expected_call": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 0.029926, + "model_compute_ms": 11944.2, + "prediction": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 302.311738, + "prefill_ms": 9483.4, + "prefilled_tokens": 226, + "private_result_exposed_on_miss": false, + "request_wall_ms": 12262.759239001753, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_b1ec97e6ea85bf4ef9afb4d5", + "commit_signal_sent": false, + "commit_wait_ms": 0.029926, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 11957.321687, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "orders_ref": "jade" + }, + "name": "get_order_details" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 302.311738, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "amber", + "call_sha256": "428dbfc05f17e7a9295c2b4741dbae2bb91d3f107150b77bb34c6ca25bd8af23", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.0955059986154, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "status": "hit" + }, + "tool_call_id": "call_b1ec97e6ea85bf4ef9afb4d5", + "tool_result": { + "call_ref": "amber", + "call_sha256": "428dbfc05f17e7a9295c2b4741dbae2bb91d3f107150b77bb34c6ca25bd8af23", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.0955059986154, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 11957.321687 + } + ], + "task_id": "dependent_workflow_003", + "task_ms": 40068.941525998525 + }, + "task": { + "call_count": 3, + "customer_email": "agent-benchmark-3@example.test", + "destination": "Bologna", + "id": "dependent_workflow_003" + } + }, + { + "arm_order": [ + "control", + "speculative" + ], + "control": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 4, + "completion_tokens": 239, + "decode_ms": 10499.3, + "expected_final": "workflow_complete:cedar", + "exposed_tool_wait_ms": 8107.510414996796, + "final": { + "accept_rate": 0.75, + "assistant_message": { + "content": "workflow_complete:cedar", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 687, + "call": null, + "canonical_call": null, + "completion_tokens": 8, + "content": "workflow_complete:cedar", + "content_sha256": "819da1a0d70737e95681c7bb3ab9df9cfab50c9a2142f5c63b7f8e07482557c8", + "decode_ms": 453.5, + "decode_tokens_per_sec": 17.6, + "model_compute_ms": 25074.2, + "prefill_ms": 24620.7, + "prefilled_tokens": 573, + "request_wall_ms": 25096.153071997833, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 54668.5, + "prediction_hits": 0, + "predictor_wall_ms": 0.0, + "steps": [ + { + "accept_rate": 0.7678571343421936, + "assistant_message": { + "content": "I need to start with step 1: resolve_customer using the customer email from the request.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-4@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_05f0932a462b9bfb40ee2bf8", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 627, + "call": { + "arguments": { + "customer_email": "agent-benchmark-4@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-4@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 58, + "content": "I need to start with step 1: resolve_customer using the customer email from the request.\n\n", + "content_sha256": "9eb32ae482f67f3e7c0fd5f011a2c9c7fb3e76804c484bdf6b40bf8add1aa381", + "decode_ms": 2664.3, + "decode_tokens_per_sec": 21.8, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-4@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 2026.8038169997453, + "model_compute_ms": 2664.3, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 2677.637868997408, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_05f0932a462b9bfb40ee2bf8", + "tool_result": { + "call_ref": "coral", + "call_sha256": "3b6684a3dbe4bce123f571a33575e15d7010b9449679680f7be2e7228dc69bb4", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.2625609995448, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 2026.8038169997453 + }, + { + "accept_rate": 0.7884615659713745, + "assistant_message": { + "content": "Now I need to proceed to step 2: list_open_orders using the customer_ref from the previous result.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"coral\"}", + "name": "list_open_orders" + }, + "id": "call_84a190fa8d91b85eb6b6724e", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 686, + "call": { + "arguments": { + "customer_ref": "coral" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"coral\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 54, + "content": "Now I need to proceed to step 2: list_open_orders using the customer_ref from the previous result.\n\n", + "content_sha256": "a50b5403267366caf3f836c1af4d67edc7470a96137213b4edc93bfb9c0f54fd", + "decode_ms": 2410.2, + "decode_tokens_per_sec": 22.4, + "expected_call": { + "arguments": { + "customer_ref": "coral" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 2026.8922049981484, + "model_compute_ms": 6708.9, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 4298.7, + "prefilled_tokens": 97, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6728.953044999798, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_84a190fa8d91b85eb6b6724e", + "tool_result": { + "call_ref": "coral", + "call_sha256": "12703ba32ec4682fbae3c4aba8cf73057f6427072573ce2f69f4252058fb7ad3", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1366260003124, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 2026.8922049981484 + }, + { + "accept_rate": 0.7916666865348816, + "assistant_message": { + "content": "Now I proceed to step 3: get_order_details using the orders_ref from the previous result.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"coral\"}", + "name": "get_order_details" + }, + "id": "call_160bcb296c5d588c79ef1016", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 838, + "call": { + "arguments": { + "orders_ref": "coral" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"coral\"},\"name\":\"get_order_details\"}", + "completion_tokens": 50, + "content": "Now I proceed to step 3: get_order_details using the orders_ref from the previous result.\n\n", + "content_sha256": "bc190728cc301be15a35259dc01ce8c5880575dc6bfcc34e09e1382ec03aa00b", + "decode_ms": 2322.5, + "decode_tokens_per_sec": 21.5, + "expected_call": { + "arguments": { + "orders_ref": "coral" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 2025.844790998235, + "model_compute_ms": 6352.7, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 4030.2, + "prefilled_tokens": 93, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6374.144949000765, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_160bcb296c5d588c79ef1016", + "tool_result": { + "call_ref": "amber", + "call_sha256": "765b3880efd7c71d4ef691862f97efc09d65b30520b309ca6e780fa539a9c14f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1390049983456, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 2025.844790998235 + }, + { + "accept_rate": 0.8125, + "assistant_message": { + "content": "Now I proceed to step 4: calculate_shipping using the order_ref from the previous result and the destination from the user request.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"destination\":\"Florence\",\"order_ref\":\"amber\"}", + "name": "calculate_shipping" + }, + "id": "call_b6edca20200edc145115f6a6", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 838, + "call": { + "arguments": { + "destination": "Florence", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"destination\":\"Florence\",\"order_ref\":\"amber\"},\"name\":\"calculate_shipping\"}", + "completion_tokens": 69, + "content": "Now I proceed to step 4: calculate_shipping using the order_ref from the previous result and the destination from the user request.\n\n", + "content_sha256": "eeddf10343ca1f79fbf9009b3ce6ecb712a9d084d51a516f0d84e085b8e69f56", + "decode_ms": 2648.8, + "decode_tokens_per_sec": 26.0, + "expected_call": { + "arguments": { + "destination": "Florence", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "exposed_tool_wait_ms": 2027.9696020006668, + "model_compute_ms": 13868.400000000001, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 11219.6, + "prefilled_tokens": 240, + "private_result_exposed_on_miss": false, + "request_wall_ms": 13890.46880800015, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_b6edca20200edc145115f6a6", + "tool_result": { + "call_ref": "cedar", + "call_sha256": "ec7b7dd2a51de438258f7c506a273b314691aa7b33181bc0c1750ed790bf6820", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.3140449989587, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "tool_wall_ms": 2027.9696020006668 + } + ], + "task_id": "dependent_workflow_004", + "task_ms": 62876.86303100054 + }, + "pair_index": 4, + "speculative": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 4, + "completion_tokens": 239, + "decode_ms": 10580.1, + "expected_final": "workflow_complete:cedar", + "exposed_tool_wait_ms": 0.097472, + "final": { + "accept_rate": 0.75, + "assistant_message": { + "content": "workflow_complete:cedar", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 983, + "call": null, + "canonical_call": null, + "completion_tokens": 8, + "content": "workflow_complete:cedar", + "content_sha256": "819da1a0d70737e95681c7bb3ab9df9cfab50c9a2142f5c63b7f8e07482557c8", + "decode_ms": 450.3, + "decode_tokens_per_sec": 17.8, + "model_compute_ms": 13370.3, + "prefill_ms": 12920.0, + "prefilled_tokens": 277, + "request_wall_ms": 13387.4202650004, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 41931.7, + "prediction_hits": 4, + "predictor_wall_ms": 1259.667634, + "steps": [ + { + "accept_rate": 0.7678571343421936, + "assistant_message": { + "content": "I need to start with step 1: resolve_customer using the customer email from the request.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-4@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_2837859c6312e6288e81c090", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 627, + "call": { + "arguments": { + "customer_email": "agent-benchmark-4@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-4@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 58, + "content": "I need to start with step 1: resolve_customer using the customer email from the request.\n\n", + "content_sha256": "9eb32ae482f67f3e7c0fd5f011a2c9c7fb3e76804c484bdf6b40bf8add1aa381", + "decode_ms": 2741.1, + "decode_tokens_per_sec": 21.2, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-4@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 0.020398, + "model_compute_ms": 2741.1, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-4@example.test" + }, + "name": "resolve_customer" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 291.061038, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 3047.313964998466, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_2837859c6312e6288e81c090", + "commit_signal_sent": false, + "commit_wait_ms": 0.020398, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 2753.997678, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-4@example.test" + }, + "name": "resolve_customer" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 291.061038, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "coral", + "call_sha256": "3b6684a3dbe4bce123f571a33575e15d7010b9449679680f7be2e7228dc69bb4", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1131339995482, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "status": "hit" + }, + "tool_call_id": "call_2837859c6312e6288e81c090", + "tool_result": { + "call_ref": "coral", + "call_sha256": "3b6684a3dbe4bce123f571a33575e15d7010b9449679680f7be2e7228dc69bb4", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1131339995482, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 2753.997678 + }, + { + "accept_rate": 0.7884615659713745, + "assistant_message": { + "content": "Now I need to proceed to step 2: list_open_orders using the customer_ref from the previous result.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"coral\"}", + "name": "list_open_orders" + }, + "id": "call_5cb50c7af353303294850cab", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 686, + "call": { + "arguments": { + "customer_ref": "coral" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"coral\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 54, + "content": "Now I need to proceed to step 2: list_open_orders using the customer_ref from the previous result.\n\n", + "content_sha256": "a50b5403267366caf3f836c1af4d67edc7470a96137213b4edc93bfb9c0f54fd", + "decode_ms": 2435.0, + "decode_tokens_per_sec": 22.2, + "expected_call": { + "arguments": { + "customer_ref": "coral" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 0.025598, + "model_compute_ms": 6698.4, + "prediction": { + "arguments": { + "customer_ref": "coral" + }, + "name": "list_open_orders" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 284.260567, + "prefill_ms": 4263.4, + "prefilled_tokens": 97, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6999.086507999891, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_5cb50c7af353303294850cab", + "commit_signal_sent": false, + "commit_wait_ms": 0.025598, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6711.259778, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_ref": "coral" + }, + "name": "list_open_orders" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 284.260567, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "coral", + "call_sha256": "12703ba32ec4682fbae3c4aba8cf73057f6427072573ce2f69f4252058fb7ad3", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1238490003743, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "status": "hit" + }, + "tool_call_id": "call_5cb50c7af353303294850cab", + "tool_result": { + "call_ref": "coral", + "call_sha256": "12703ba32ec4682fbae3c4aba8cf73057f6427072573ce2f69f4252058fb7ad3", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1238490003743, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 6711.259778 + }, + { + "accept_rate": 0.7916666865348816, + "assistant_message": { + "content": "Now I proceed to step 3: get_order_details using the orders_ref from the previous result.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"coral\"}", + "name": "get_order_details" + }, + "id": "call_48dcbe64642e138fab1f322c", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 838, + "call": { + "arguments": { + "orders_ref": "coral" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"coral\"},\"name\":\"get_order_details\"}", + "completion_tokens": 50, + "content": "Now I proceed to step 3: get_order_details using the orders_ref from the previous result.\n\n", + "content_sha256": "bc190728cc301be15a35259dc01ce8c5880575dc6bfcc34e09e1382ec03aa00b", + "decode_ms": 2313.8, + "decode_tokens_per_sec": 21.6, + "expected_call": { + "arguments": { + "orders_ref": "coral" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 0.020298, + "model_compute_ms": 6294.9, + "prediction": { + "arguments": { + "orders_ref": "coral" + }, + "name": "get_order_details" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 313.131125, + "prefill_ms": 3981.1, + "prefilled_tokens": 93, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6623.575071000232, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_48dcbe64642e138fab1f322c", + "commit_signal_sent": false, + "commit_wait_ms": 0.020298, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6308.194577, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "orders_ref": "coral" + }, + "name": "get_order_details" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 313.131125, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "amber", + "call_sha256": "765b3880efd7c71d4ef691862f97efc09d65b30520b309ca6e780fa539a9c14f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1107650023187, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "status": "hit" + }, + "tool_call_id": "call_48dcbe64642e138fab1f322c", + "tool_result": { + "call_ref": "amber", + "call_sha256": "765b3880efd7c71d4ef691862f97efc09d65b30520b309ca6e780fa539a9c14f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1107650023187, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 6308.194577 + }, + { + "accept_rate": 0.8125, + "assistant_message": { + "content": "Now I proceed to step 4: calculate_shipping using the order_ref from the previous result and the destination from the user request.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"destination\":\"Florence\",\"order_ref\":\"amber\"}", + "name": "calculate_shipping" + }, + "id": "call_0bf7dcfaba1ade732d19282c", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 838, + "call": { + "arguments": { + "destination": "Florence", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"destination\":\"Florence\",\"order_ref\":\"amber\"},\"name\":\"calculate_shipping\"}", + "completion_tokens": 69, + "content": "Now I proceed to step 4: calculate_shipping using the order_ref from the previous result and the destination from the user request.\n\n", + "content_sha256": "eeddf10343ca1f79fbf9009b3ce6ecb712a9d084d51a516f0d84e085b8e69f56", + "decode_ms": 2639.9, + "decode_tokens_per_sec": 26.1, + "expected_call": { + "arguments": { + "destination": "Florence", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "exposed_tool_wait_ms": 0.031178, + "model_compute_ms": 12827.0, + "prediction": { + "arguments": { + "destination": "Florence", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 371.214904, + "prefill_ms": 10187.1, + "prefilled_tokens": 240, + "private_result_exposed_on_miss": false, + "request_wall_ms": 13214.942769001937, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_0bf7dcfaba1ade732d19282c", + "commit_signal_sent": false, + "commit_wait_ms": 0.031178, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 12840.298167, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "destination": "Florence", + "order_ref": "amber" + }, + "name": "calculate_shipping" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 371.214904, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "cedar", + "call_sha256": "ec7b7dd2a51de438258f7c506a273b314691aa7b33181bc0c1750ed790bf6820", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.074912000855, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "status": "hit" + }, + "tool_call_id": "call_0bf7dcfaba1ade732d19282c", + "tool_result": { + "call_ref": "cedar", + "call_sha256": "ec7b7dd2a51de438258f7c506a273b314691aa7b33181bc0c1750ed790bf6820", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.074912000855, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "tool_wall_ms": 12840.298167 + } + ], + "task_id": "dependent_workflow_004", + "task_ms": 43273.09860999958 + }, + "task": { + "call_count": 4, + "customer_email": "agent-benchmark-4@example.test", + "destination": "Florence", + "id": "dependent_workflow_004" + } + }, + { + "arm_order": [ + "speculative", + "control" + ], + "control": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 5, + "completion_tokens": 251, + "decode_ms": 12638.6, + "expected_final": "workflow_complete:amber", + "exposed_tool_wait_ms": 10131.949799000722, + "final": { + "accept_rate": 0.625, + "assistant_message": { + "content": "workflow_complete:amber", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 1136, + "call": null, + "canonical_call": null, + "completion_tokens": 7, + "content": "workflow_complete:amber", + "content_sha256": "432bcc6fd099a0a7a97da30984ac9e2d3760ca9fc31f5b74162f2961c04c83b8", + "decode_ms": 447.6, + "decode_tokens_per_sec": 15.6, + "model_compute_ms": 11109.5, + "prefill_ms": 10661.9, + "prefilled_tokens": 253, + "request_wall_ms": 11133.575896001275, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 46075.2, + "prediction_hits": 0, + "predictor_wall_ms": 0.0, + "steps": [ + { + "accept_rate": 0.5921052694320679, + "assistant_message": { + "content": "I need to process this customer request through the 5-step workflow. Let me start with step 1: resolving the customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-5@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_7f9d4f25ac2551c9726d5b28", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 648, + "call": { + "arguments": { + "customer_email": "agent-benchmark-5@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-5@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 64, + "content": "I need to process this customer request through the 5-step workflow. Let me start with step 1: resolving the customer.\n\n", + "content_sha256": "caac2c887d5589ba57ca98c8c77ea68cdd12f4569133cc824cdc1cc82c831c09", + "decode_ms": 3421.7, + "decode_tokens_per_sec": 18.7, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-5@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 2025.8035280021431, + "model_compute_ms": 3421.7, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 3435.3289089995087, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_7f9d4f25ac2551c9726d5b28", + "tool_result": { + "call_ref": "maple", + "call_sha256": "71063336b1901dd2a0050d4a7a0ebe52ad0b0755f5af079efe22db9c1df00b29", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.163259999681, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 2025.8035280021431 + }, + { + "accept_rate": 0.7272727489471436, + "assistant_message": { + "content": "Now I need to list open orders for this resolved customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"maple\"}", + "name": "list_open_orders" + }, + "id": "call_98a842d8ac148b26dcf0635a", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 713, + "call": { + "arguments": { + "customer_ref": "maple" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"maple\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 43, + "content": "Now I need to list open orders for this resolved customer.\n\n", + "content_sha256": "3ce7d0f18b723f0f13b497c10e5799d52d899e25e977e82e22a5ba8e25b52e6e", + "decode_ms": 2187.2, + "decode_tokens_per_sec": 19.7, + "expected_call": { + "arguments": { + "customer_ref": "maple" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 2026.4790990004258, + "model_compute_ms": 6212.4, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 4025.2, + "prefilled_tokens": 93, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6232.237337000697, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_98a842d8ac148b26dcf0635a", + "tool_result": { + "call_ref": "willow", + "call_sha256": "8da0147e626bec41f9bafd2ec61d07d01a33e176990483fd62a2b93eeea82e2f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1353239968012, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 2026.4790990004258 + }, + { + "accept_rate": 0.6363636255264282, + "assistant_message": { + "content": "Now I need to inspect the selected order.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"willow\"}", + "name": "get_order_details" + }, + "id": "call_27afb3c2c12355b632d21f62", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 850, + "call": { + "arguments": { + "orders_ref": "willow" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"willow\"},\"name\":\"get_order_details\"}", + "completion_tokens": 39, + "content": "Now I need to inspect the selected order.\n\n", + "content_sha256": "0501df1f5ec6fcf19c50a853fe0631d19fcc4dac651569fc7278ecc84ffcdb0c", + "decode_ms": 2278.5, + "decode_tokens_per_sec": 17.1, + "expected_call": { + "arguments": { + "orders_ref": "willow" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 2026.0629559998051, + "model_compute_ms": 6395.9, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 4117.4, + "prefilled_tokens": 95, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6417.946880999807, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_27afb3c2c12355b632d21f62", + "tool_result": { + "call_ref": "olive", + "call_sha256": "66ea7a270ffff62c364786d69dc72144b0b0eb928c262f682dff72635964c6a9", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.2660539976205, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 2026.0629559998051 + }, + { + "accept_rate": 0.7142857313156128, + "assistant_message": { + "content": "Now I need to calculate shipping for the order to Naples.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"destination\":\"Naples\",\"order_ref\":\"olive\"}", + "name": "calculate_shipping" + }, + "id": "call_43fdf79651e007386a692348", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 985, + "call": { + "arguments": { + "destination": "Naples", + "order_ref": "olive" + }, + "name": "calculate_shipping" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"destination\":\"Naples\",\"order_ref\":\"olive\"},\"name\":\"calculate_shipping\"}", + "completion_tokens": 54, + "content": "Now I need to calculate shipping for the order to Naples.\n\n", + "content_sha256": "fe9b497f45334c64b9aa739eddda6779832ad53fc5148834a71fc2d79b49e467", + "decode_ms": 2846.4, + "decode_tokens_per_sec": 19.0, + "expected_call": { + "arguments": { + "destination": "Naples", + "order_ref": "olive" + }, + "name": "calculate_shipping" + }, + "exposed_tool_wait_ms": 2026.464924998436, + "model_compute_ms": 6915.3, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 4068.9, + "prefilled_tokens": 95, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6937.539090999053, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_43fdf79651e007386a692348", + "tool_result": { + "call_ref": "cedar", + "call_sha256": "db784ed2c9e33569d9c6bdd36e6963076032cd8d0bf48cf954420a843cb8d593", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.2615110024635, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "tool_wall_ms": 2026.464924998436 + }, + { + "accept_rate": 0.8500000238418579, + "assistant_message": { + "content": "Now I need to prepare the final customer summary.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"shipping_ref\":\"cedar\"}", + "name": "prepare_customer_summary" + }, + "id": "call_fe73a9484a67df2e2c858e6f", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 985, + "call": { + "arguments": { + "shipping_ref": "cedar" + }, + "name": "prepare_customer_summary" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"shipping_ref\":\"cedar\"},\"name\":\"prepare_customer_summary\"}", + "completion_tokens": 44, + "content": "Now I need to prepare the final customer summary.\n\n", + "content_sha256": "6d12faaaf38ef53fc446a8a34f8fe0f3a22a7bdc6d3dab6ef4f1f25c5ba91b44", + "decode_ms": 1457.2, + "decode_tokens_per_sec": 30.2, + "expected_call": { + "arguments": { + "shipping_ref": "cedar" + }, + "name": "prepare_customer_summary" + }, + "exposed_tool_wait_ms": 2027.1392909999122, + "model_compute_ms": 12020.400000000001, + "prediction": null, + "prediction_hit": false, + "prediction_reason": null, + "prediction_status": null, + "predictor_wall_ms": 0.0, + "prefill_ms": 10563.2, + "prefilled_tokens": 245, + "private_result_exposed_on_miss": false, + "request_wall_ms": 12043.583460002992, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "call_fe73a9484a67df2e2c858e6f", + "tool_result": { + "call_ref": "amber", + "call_sha256": "6ba7d6c0326fbe23206f51a9a4268beb41a3050097565421946c4ac32900f9bc", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.4121849997318, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "prepare_customer_summary" + }, + "tool_wall_ms": 2027.1392909999122 + } + ], + "task_id": "dependent_workflow_005", + "task_ms": 56334.70175000184 + }, + "pair_index": 5, + "speculative": { + "all_calls_correct": true, + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "call_count": 5, + "completion_tokens": 251, + "decode_ms": 12663.6, + "expected_final": "workflow_complete:amber", + "exposed_tool_wait_ms": 0.109194, + "final": { + "accept_rate": 0.625, + "assistant_message": { + "content": "workflow_complete:amber", + "role": "assistant" + }, + "cache_hit": true, + "cached_prefix_tokens": 714, + "call": null, + "canonical_call": null, + "completion_tokens": 7, + "content": "workflow_complete:amber", + "content_sha256": "432bcc6fd099a0a7a97da30984ac9e2d3760ca9fc31f5b74162f2961c04c83b8", + "decode_ms": 445.5, + "decode_tokens_per_sec": 15.7, + "model_compute_ms": 29265.4, + "prefill_ms": 28819.9, + "prefilled_tokens": 675, + "request_wall_ms": 29282.140494000487, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": null, + "tool_call_id": "" + }, + "final_correct": true, + "model_compute_ms": 64146.3, + "prediction_hits": 5, + "predictor_wall_ms": 1818.887272, + "steps": [ + { + "accept_rate": 0.5921052694320679, + "assistant_message": { + "content": "I need to process this customer request through the 5-step workflow. Let me start with step 1: resolving the customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_email\":\"agent-benchmark-5@example.test\"}", + "name": "resolve_customer" + }, + "id": "call_09ab10b04127b5b933f396fb", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 648, + "call": { + "arguments": { + "customer_email": "agent-benchmark-5@example.test" + }, + "name": "resolve_customer" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_email\":\"agent-benchmark-5@example.test\"},\"name\":\"resolve_customer\"}", + "completion_tokens": 64, + "content": "I need to process this customer request through the 5-step workflow. Let me start with step 1: resolving the customer.\n\n", + "content_sha256": "caac2c887d5589ba57ca98c8c77ea68cdd12f4569133cc824cdc1cc82c831c09", + "decode_ms": 3429.0, + "decode_tokens_per_sec": 18.7, + "expected_call": { + "arguments": { + "customer_email": "agent-benchmark-5@example.test" + }, + "name": "resolve_customer" + }, + "exposed_tool_wait_ms": 0.02144, + "model_compute_ms": 3429.0, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-5@example.test" + }, + "name": "resolve_customer" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 295.276084, + "prefill_ms": 0.0, + "prefilled_tokens": 0, + "private_result_exposed_on_miss": false, + "request_wall_ms": 3771.3140349987953, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_09ab10b04127b5b933f396fb", + "commit_signal_sent": false, + "commit_wait_ms": 0.02144, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 3474.123096, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_email": "agent-benchmark-5@example.test" + }, + "name": "resolve_customer" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 295.276084, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "maple", + "call_sha256": "71063336b1901dd2a0050d4a7a0ebe52ad0b0755f5af079efe22db9c1df00b29", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1511240006948, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "status": "hit" + }, + "tool_call_id": "call_09ab10b04127b5b933f396fb", + "tool_result": { + "call_ref": "maple", + "call_sha256": "71063336b1901dd2a0050d4a7a0ebe52ad0b0755f5af079efe22db9c1df00b29", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1511240006948, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "resolve_customer" + }, + "tool_wall_ms": 3474.123096 + }, + { + "accept_rate": 0.7272727489471436, + "assistant_message": { + "content": "Now I need to list open orders for this resolved customer.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"customer_ref\":\"maple\"}", + "name": "list_open_orders" + }, + "id": "call_63e840f8505a49839de2d13a", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 713, + "call": { + "arguments": { + "customer_ref": "maple" + }, + "name": "list_open_orders" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"customer_ref\":\"maple\"},\"name\":\"list_open_orders\"}", + "completion_tokens": 43, + "content": "Now I need to list open orders for this resolved customer.\n\n", + "content_sha256": "3ce7d0f18b723f0f13b497c10e5799d52d899e25e977e82e22a5ba8e25b52e6e", + "decode_ms": 2186.8, + "decode_tokens_per_sec": 19.7, + "expected_call": { + "arguments": { + "customer_ref": "maple" + }, + "name": "list_open_orders" + }, + "exposed_tool_wait_ms": 0.016611, + "model_compute_ms": 6183.6, + "prediction": { + "arguments": { + "customer_ref": "maple" + }, + "name": "list_open_orders" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 286.880963, + "prefill_ms": 3996.8, + "prefilled_tokens": 93, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6486.068479000096, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_63e840f8505a49839de2d13a", + "commit_signal_sent": false, + "commit_wait_ms": 0.016611, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6196.614339, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "customer_ref": "maple" + }, + "name": "list_open_orders" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 286.880963, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "willow", + "call_sha256": "8da0147e626bec41f9bafd2ec61d07d01a33e176990483fd62a2b93eeea82e2f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1012419998005, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "status": "hit" + }, + "tool_call_id": "call_63e840f8505a49839de2d13a", + "tool_result": { + "call_ref": "willow", + "call_sha256": "8da0147e626bec41f9bafd2ec61d07d01a33e176990483fd62a2b93eeea82e2f", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.1012419998005, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "list_open_orders" + }, + "tool_wall_ms": 6196.614339 + }, + { + "accept_rate": 0.6363636255264282, + "assistant_message": { + "content": "Now I need to inspect the selected order.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"orders_ref\":\"willow\"}", + "name": "get_order_details" + }, + "id": "call_b36d77890661c729282b8106", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 850, + "call": { + "arguments": { + "orders_ref": "willow" + }, + "name": "get_order_details" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"orders_ref\":\"willow\"},\"name\":\"get_order_details\"}", + "completion_tokens": 39, + "content": "Now I need to inspect the selected order.\n\n", + "content_sha256": "0501df1f5ec6fcf19c50a853fe0631d19fcc4dac651569fc7278ecc84ffcdb0c", + "decode_ms": 2287.7, + "decode_tokens_per_sec": 17.0, + "expected_call": { + "arguments": { + "orders_ref": "willow" + }, + "name": "get_order_details" + }, + "exposed_tool_wait_ms": 0.018384, + "model_compute_ms": 6361.299999999999, + "prediction": { + "arguments": { + "orders_ref": "willow" + }, + "name": "get_order_details" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 315.010142, + "prefill_ms": 4073.6, + "prefilled_tokens": 95, + "private_result_exposed_on_miss": false, + "request_wall_ms": 6692.558883001766, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_b36d77890661c729282b8106", + "commit_signal_sent": false, + "commit_wait_ms": 0.018384, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6374.658399, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "orders_ref": "willow" + }, + "name": "get_order_details" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 315.010142, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "olive", + "call_sha256": "66ea7a270ffff62c364786d69dc72144b0b0eb928c262f682dff72635964c6a9", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.129840002046, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "status": "hit" + }, + "tool_call_id": "call_b36d77890661c729282b8106", + "tool_result": { + "call_ref": "olive", + "call_sha256": "66ea7a270ffff62c364786d69dc72144b0b0eb928c262f682dff72635964c6a9", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.129840002046, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "get_order_details" + }, + "tool_wall_ms": 6374.658399 + }, + { + "accept_rate": 0.7142857313156128, + "assistant_message": { + "content": "Now I need to calculate shipping for the order to Naples.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"destination\":\"Naples\",\"order_ref\":\"olive\"}", + "name": "calculate_shipping" + }, + "id": "call_21d1fac465004f3136ee9c1d", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 985, + "call": { + "arguments": { + "destination": "Naples", + "order_ref": "olive" + }, + "name": "calculate_shipping" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"destination\":\"Naples\",\"order_ref\":\"olive\"},\"name\":\"calculate_shipping\"}", + "completion_tokens": 54, + "content": "Now I need to calculate shipping for the order to Naples.\n\n", + "content_sha256": "fe9b497f45334c64b9aa739eddda6779832ad53fc5148834a71fc2d79b49e467", + "decode_ms": 2859.3, + "decode_tokens_per_sec": 18.9, + "expected_call": { + "arguments": { + "destination": "Naples", + "order_ref": "olive" + }, + "name": "calculate_shipping" + }, + "exposed_tool_wait_ms": 0.029555, + "model_compute_ms": 6913.5, + "prediction": { + "arguments": { + "destination": "Naples", + "order_ref": "olive" + }, + "name": "calculate_shipping" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 544.435294, + "prefill_ms": 4054.2, + "prefilled_tokens": 95, + "private_result_exposed_on_miss": false, + "request_wall_ms": 7478.634543997032, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_21d1fac465004f3136ee9c1d", + "commit_signal_sent": false, + "commit_wait_ms": 0.029555, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 6931.548898, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "destination": "Naples", + "order_ref": "olive" + }, + "name": "calculate_shipping" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 544.435294, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "cedar", + "call_sha256": "db784ed2c9e33569d9c6bdd36e6963076032cd8d0bf48cf954420a843cb8d593", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.118616000691, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "status": "hit" + }, + "tool_call_id": "call_21d1fac465004f3136ee9c1d", + "tool_result": { + "call_ref": "cedar", + "call_sha256": "db784ed2c9e33569d9c6bdd36e6963076032cd8d0bf48cf954420a843cb8d593", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.118616000691, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "calculate_shipping" + }, + "tool_wall_ms": 6931.548898 + }, + { + "accept_rate": 0.8500000238418579, + "assistant_message": { + "content": "Now I need to prepare the final customer summary.\n\n", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"shipping_ref\":\"cedar\"}", + "name": "prepare_customer_summary" + }, + "id": "call_94fef0533909642dcbd44e46", + "type": "function" + } + ] + }, + "cache_hit": true, + "cached_prefix_tokens": 985, + "call": { + "arguments": { + "shipping_ref": "cedar" + }, + "name": "prepare_customer_summary" + }, + "call_correct": true, + "canonical_call": "{\"arguments\":{\"shipping_ref\":\"cedar\"},\"name\":\"prepare_customer_summary\"}", + "completion_tokens": 44, + "content": "Now I need to prepare the final customer summary.\n\n", + "content_sha256": "6d12faaaf38ef53fc446a8a34f8fe0f3a22a7bdc6d3dab6ef4f1f25c5ba91b44", + "decode_ms": 1455.3, + "decode_tokens_per_sec": 30.2, + "expected_call": { + "arguments": { + "shipping_ref": "cedar" + }, + "name": "prepare_customer_summary" + }, + "exposed_tool_wait_ms": 0.023204, + "model_compute_ms": 11993.5, + "prediction": { + "arguments": { + "shipping_ref": "cedar" + }, + "name": "prepare_customer_summary" + }, + "prediction_hit": true, + "prediction_reason": null, + "prediction_status": "hit", + "predictor_wall_ms": 377.284789, + "prefill_ms": 10538.2, + "prefilled_tokens": 245, + "private_result_exposed_on_miss": false, + "request_wall_ms": 12388.57133199781, + "semantic_hint": { + "accepted_tokens": 0, + "longest_suffix_tokens": 0, + "matched_rounds": 0, + "native_gate_misses": 0, + "proposed_tokens": 0, + "readiness_polls": 0, + "rejected_rounds": 0 + }, + "speculation": { + "accelerator_relation": "non_accelerator", + "call_id": "call_94fef0533909642dcbd44e46", + "commit_signal_sent": false, + "commit_wait_ms": 0.023204, + "confidence": 0.75, + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "cpu_affinity_isolated": true, + "decode_interference_qualified": true, + "executor_wall_ms": 12007.430824, + "expected_speedup": 1.5428733045594794, + "prediction": { + "arguments": { + "shipping_ref": "cedar" + }, + "name": "prepare_customer_summary" + }, + "prediction_source": "qwen3-0.6b", + "predictor_wall_ms": 377.284789, + "protocol": "dflash.tool-speculation.v1", + "resource_percentage": 100, + "result": { + "call_ref": "amber", + "call_sha256": "6ba7d6c0326fbe23206f51a9a4268beb41a3050097565421946c4ac32900f9bc", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.3881299999193, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "prepare_customer_summary" + }, + "status": "hit" + }, + "tool_call_id": "call_94fef0533909642dcbd44e46", + "tool_result": { + "call_ref": "amber", + "call_sha256": "6ba7d6c0326fbe23206f51a9a4268beb41a3050097565421946c4ac32900f9bc", + "cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "elapsed_ms": 2000.3881299999193, + "latency_ms": 2000, + "side_effects": false, + "tool_name": "prepare_customer_summary" + }, + "tool_wall_ms": 12007.430824 + } + ], + "task_id": "dependent_workflow_005", + "task_ms": 66100.22805599874 + }, + "task": { + "call_count": 5, + "customer_email": "agent-benchmark-5@example.test", + "destination": "Naples", + "id": "dependent_workflow_005" + } + } + ], + "production_gate": { + "checks": { + "control_calls_correct": true, + "control_finals_correct": true, + "decode_slowdown_p50": true, + "decode_slowdown_p95": false, + "ds4_active": true, + "full_task_speedup": false, + "full_task_speedup_ci": false, + "full_task_tail": false, + "model_slowdown_p50": false, + "model_slowdown_p95": false, + "prediction_hit_rate": true, + "prefix_cache_active": true, + "private_miss_results_hidden": true, + "speculative_calls_correct": true, + "speculative_finals_correct": true, + "target_calls_stable": true, + "target_outputs_stable": true, + "tool_results_stable": true + }, + "passed": false, + "thresholds": { + "max_decode_slowdown_p95_percent": 5.0, + "max_decode_slowdown_percent": 1.0, + "max_model_slowdown_p95_percent": 5.0, + "max_model_slowdown_percent": 1.0, + "min_continuation_cache_hit_rate": 0.8, + "min_hit_rate": 0.5, + "min_task_speedup": 1.4, + "min_task_speedup_ci_low": 1.0, + "min_task_speedup_p05": 1.0 + } + }, + "schema_version": 1, + "server_snapshot": { + "prefix_cache": { + "capacity": 32, + "in_use": 8, + "lifetime_hits": 12 + }, + "tool_speculation": { + "allowed_tools": [ + "calculate_shipping", + "get_order_details", + "list_open_orders", + "prepare_customer_summary", + "resolve_customer" + ], + "automatic_prediction_enabled": true, + "compute_isolation": "disjoint_cpu_affinity", + "cpu_affinity_isolated": true, + "enabled": true, + "execution_mode": "child_process_cpu_affinity", + "executor_contract": "child_process_cpu_affinity", + "hip_reserved_tool_compute_units": 0, + "hip_tool_device": null, + "max_model_slowdown_ratio": 1.05, + "model_cpu_affinity": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29 + ], + "model_expert_ownership_unique": false, + "model_routing_static": false, + "prediction_confidence": 0.75, + "prediction_source": "qwen3-0.6b", + "predictor_decode_isolated": true, + "predictor_schedule": "before-model", + "preserves_token_speculation": true, + "profile_lanes": [ + { + "accelerator_relation": "non_accelerator", + "decode_interference_qualified": true, + "model_slowdown_ratio": 1.0011341375399527, + "requires_static_model_routing": false, + "requires_unique_expert_ownership": false, + "resource_percentage": 100 + } + ], + "profile_status": "qualified", + "protocol": "dflash.tool-speculation.v1", + "requires_client_support": false, + "tool_cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "unqualified_lane_policy": "defer" + } + }, + "summary": { + "all_ds4_active": true, + "all_private_miss_results_hidden": true, + "all_tool_results_stable": true, + "calls_per_task": [ + 3, + 4, + 5, + 3, + 4, + 5 + ], + "continuation_cache_hit_rate": 1.0, + "control_call_accuracy": 1.0, + "control_exposed_tool_wait_p50_ms": 8106.7744020001555, + "control_final_accuracy": 1.0, + "control_model_compute_p50_ms": 49637.649999999994, + "control_task_p50_ms": 58870.49264650159, + "decode_slowdown_p50_percent": 0.2508973038222173, + "decode_slowdown_p95_percent": 9.284182748470627, + "exposed_tool_wait_reduction_percent": 99.99885625636658, + "exposed_tool_wait_total_speedup": 87432.1806726216, + "ideal_zero_interference_task_speedup_ceiling_p50": 1.1425721859694333, + "model_compute_slowdown_p50_percent": 7.474064005523928, + "model_compute_slowdown_p95_percent": 34.570885935240916, + "paired_saved_p50_ms": 3369.1297750010563, + "paired_task_speedup_bootstrap_95ci": [ + 0.9076229958518257, + 1.3052515771052713 + ], + "paired_task_speedup_min": 0.8522618364081324, + "paired_task_speedup_p05": 0.879942416129979, + "paired_task_speedup_p50": 1.0520357724328036, + "prediction_hit_rate": 1.0, + "prediction_hits": 24, + "predictor_per_call_p50_ms": 296.37505350000004, + "predictor_per_call_p95_ms": 378.20141285, + "predictor_per_task_p50_ms": 1263.2751875, + "semantic_hint_acceptance_rate": null, + "semantic_hint_accepted_tokens": 0, + "semantic_hint_proposed_tokens": 0, + "semantic_hint_ready_calls": 0, + "speculative_call_accuracy": 1.0, + "speculative_exposed_tool_wait_p50_ms": 0.0937145, + "speculative_final_accuracy": 1.0, + "speculative_model_compute_p50_ms": 49687.85, + "speculative_task_p50_ms": 51032.53195199977, + "target_call_stability_rate": 1.0, + "target_output_stability_rate": 1.0, + "tasks": 6, + "tasks_with_all_prediction_hits": 6, + "total_tool_calls_per_arm": 24, + "total_wall_speedup": 1.0655904186922165 + }, + "workload": { + "available_tools": [ + "resolve_customer", + "list_open_orders", + "get_order_details", + "calculate_shipping", + "prepare_customer_summary" + ], + "calls_per_task": "3-5", + "dependency": "every call after the first consumes call_ref returned by the preceding tool", + "name": "dependent multi-turn tool workflow", + "tasks": 6, + "tool_adapter": "deterministic read-only 2-second API replay; no external side effects" + } +} diff --git a/optimizations/ooo_spec_lucebox5_cpu/results/trace-compiled-engine-qwen-production-6pairs-compact.json b/optimizations/ooo_spec_lucebox5_cpu/results/trace-compiled-engine-qwen-production-6pairs-compact.json new file mode 100644 index 000000000..682b57a20 --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/results/trace-compiled-engine-qwen-production-6pairs-compact.json @@ -0,0 +1,1384 @@ +{ + "feature": "no-training trace-compiled speculative tool graphs", + "host": "lucebox5", + "methodology": { + "additive_gate_refresh": "the wrong-call privacy probe and server snapshot were refreshed after the timing arms; no recorded timing was recomputed", + "argument_binding": "the harness binds validated structured inputs to a request-scoped workflow_ref before either model runs", + "arm_order": "randomized per task", + "compiled": "one DS4+DSpark macro authorization; independent branches execute concurrently on the Strix CPU lane", + "measured_wall_time": "request through all tools and exact final answer", + "model_seed": 814, + "oracle_prediction": false, + "speculative": "Qwen predicts the trace-derived macro through the engine; its CPU graph overlaps DS4+DSpark and commits only on an exact call match", + "stage_batched": "DS4+DSpark sees only the currently-ready typed batch, authorizes one per dependency stage, runs its calls concurrently, and receives a compact rolling state instead of replaying old tool history", + "warmup_tasks": 1 + }, + "pairs": [ + { + "arm_order": [ + "compiled", + "stage_batched", + "speculative" + ], + "compiled": { + "all_ds4_active": true, + "completion_tokens": 46, + "decode_ms": 1319.1, + "exposed_tool_wait_ms": 10123.539726002491, + "final": { + "accept_rate": 1.0, + "completion_tokens": 11, + "content_sha256": "1cf1bd9a085192434f76d5bd7ec2044e26bd2d8cd095aed540621b7d837b4196" + }, + "final_correct": true, + "graph_wall_ms": 10124.20064300386, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taska" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 20001.2, + "model_turns": 2, + "prediction_hit": false, + "prediction_reason": null, + "prediction_source": null, + "prediction_status": null, + "predictor_ms": 0.0, + "task_ms": 30215.39950100123, + "tool_results_count": 10, + "tool_results_sha256": "eb2ed6a561ebc1e15c6f1a5226fb751de351f983d52dfe47f94c23a97d8c9cb2", + "underlying_calls_count": 10, + "underlying_calls_sha256": "b17c72ad474ba5d0a431bd35335fcb839c0a329504f6eed9eda5d211249e6841" + }, + "interference_probe": { + "observations": [ + { + "compiled": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "26b31bf2044cd34ec9ea91d14b9758544b6a4a62a533a58bc4f79af1472a6535", + "completion_tokens": 35, + "decode_ms": 1085.7, + "model_compute_ms": 10642.6, + "prefill_ms": 9556.9 + }, + "order": [ + "compiled", + "speculative" + ], + "speculative": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "26b31bf2044cd34ec9ea91d14b9758544b6a4a62a533a58bc4f79af1472a6535", + "completion_tokens": 35, + "decode_ms": 1090.8, + "model_compute_ms": 10675.099999999999, + "prefill_ms": 9584.3 + } + }, + { + "compiled": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "26b31bf2044cd34ec9ea91d14b9758544b6a4a62a533a58bc4f79af1472a6535", + "completion_tokens": 35, + "decode_ms": 1092.3, + "model_compute_ms": 10658.599999999999, + "prefill_ms": 9566.3 + }, + "order": [ + "speculative", + "compiled" + ], + "speculative": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "26b31bf2044cd34ec9ea91d14b9758544b6a4a62a533a58bc4f79af1472a6535", + "completion_tokens": 35, + "decode_ms": 1090.4, + "model_compute_ms": 10671.199999999999, + "prefill_ms": 9580.8 + } + }, + { + "compiled": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "26b31bf2044cd34ec9ea91d14b9758544b6a4a62a533a58bc4f79af1472a6535", + "completion_tokens": 35, + "decode_ms": 1086.8, + "model_compute_ms": 10666.099999999999, + "prefill_ms": 9579.3 + }, + "order": [ + "compiled", + "speculative" + ], + "speculative": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "26b31bf2044cd34ec9ea91d14b9758544b6a4a62a533a58bc4f79af1472a6535", + "completion_tokens": 35, + "decode_ms": 1093.4, + "model_compute_ms": 10674.8, + "prefill_ms": 9581.4 + } + } + ], + "qualified": true, + "repetitions": 3 + }, + "pair_index": 0, + "speculative": { + "all_ds4_active": true, + "completion_tokens": 46, + "decode_ms": 1208.0, + "exposed_tool_wait_ms": 0.031249, + "final": { + "accept_rate": 1.0, + "completion_tokens": 11, + "content_sha256": "1cf1bd9a085192434f76d5bd7ec2044e26bd2d8cd095aed540621b7d837b4196" + }, + "final_correct": true, + "graph_wall_ms": 10002.38108499616, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taska" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 16392.2, + "model_turns": 2, + "prediction_hit": true, + "prediction_reason": null, + "prediction_source": "native-qwen3", + "prediction_status": "hit", + "predictor_ms": 198.454737, + "task_ms": 16606.234144994232, + "tool_results_count": 10, + "tool_results_sha256": "eb2ed6a561ebc1e15c6f1a5226fb751de351f983d52dfe47f94c23a97d8c9cb2", + "underlying_calls_count": 10, + "underlying_calls_sha256": "b17c72ad474ba5d0a431bd35335fcb839c0a329504f6eed9eda5d211249e6841" + }, + "stage_batched": { + "all_ds4_active": true, + "completion_tokens": 207, + "decode_ms": 7398.0, + "exposed_tool_wait_ms": 10140.203142997052, + "final": { + "accept_rate": 1.0, + "completion_tokens": 11, + "content_sha256": "1cf1bd9a085192434f76d5bd7ec2044e26bd2d8cd095aed540621b7d837b4196" + }, + "final_correct": true, + "model_compute_ms": 71711.6, + "model_turns": 6, + "task_ms": 81868.44566600485, + "tool_results_count": 10, + "tool_results_sha256": "eb2ed6a561ebc1e15c6f1a5226fb751de351f983d52dfe47f94c23a97d8c9cb2", + "underlying_calls_count": 10, + "underlying_calls_sha256": "b17c72ad474ba5d0a431bd35335fcb839c0a329504f6eed9eda5d211249e6841" + }, + "task": { + "branch_count": 2, + "call_count": 10, + "id": "trace_compiled_000" + } + }, + { + "arm_order": [ + "compiled", + "speculative", + "stage_batched" + ], + "compiled": { + "all_ds4_active": true, + "completion_tokens": 71, + "decode_ms": 3487.0, + "exposed_tool_wait_ms": 10146.3531469999, + "final": { + "accept_rate": 0.75, + "completion_tokens": 12, + "content_sha256": "1f19e61b2ef79fed2c87e8d945b3cc479796bd8a235bc79ef7579f3bbc22adb3" + }, + "final_correct": true, + "graph_wall_ms": 10147.722080000676, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taskb" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 25378.2, + "model_turns": 2, + "prediction_hit": false, + "prediction_reason": null, + "prediction_source": null, + "prediction_status": null, + "predictor_ms": 0.0, + "task_ms": 35538.563240996154, + "tool_results_count": 15, + "tool_results_sha256": "2037fc8f56106ba9bc50763cc4b82c0f3262f0ffb8682ca5ca727eaf9d46b07b", + "underlying_calls_count": 15, + "underlying_calls_sha256": "2263e8d373d489cb82f03725feec381fb4ec218f6a706ebff2de102ed1e7c40b" + }, + "interference_probe": { + "observations": [ + { + "compiled": { + "accept_rate": 0.8035714030265808, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "4fd23541e355718c0b1ee47f20028d7324bab11dddd5a0a5a460495dd05e3461", + "completion_tokens": 59, + "decode_ms": 2711.4, + "model_compute_ms": 12727.199999999999, + "prefill_ms": 10015.8 + }, + "order": [ + "speculative", + "compiled" + ], + "speculative": { + "accept_rate": 0.7333333492279053, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "4fd23541e355718c0b1ee47f20028d7324bab11dddd5a0a5a460495dd05e3461", + "completion_tokens": 59, + "decode_ms": 2996.9, + "model_compute_ms": 13019.8, + "prefill_ms": 10022.9 + } + }, + { + "compiled": { + "accept_rate": 0.7333333492279053, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "4fd23541e355718c0b1ee47f20028d7324bab11dddd5a0a5a460495dd05e3461", + "completion_tokens": 59, + "decode_ms": 2996.0, + "model_compute_ms": 13017.8, + "prefill_ms": 10021.8 + }, + "order": [ + "compiled", + "speculative" + ], + "speculative": { + "accept_rate": 0.8035714030265808, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "4fd23541e355718c0b1ee47f20028d7324bab11dddd5a0a5a460495dd05e3461", + "completion_tokens": 59, + "decode_ms": 2722.6, + "model_compute_ms": 12733.0, + "prefill_ms": 10010.4 + } + }, + { + "compiled": { + "accept_rate": 0.8035714030265808, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "4fd23541e355718c0b1ee47f20028d7324bab11dddd5a0a5a460495dd05e3461", + "completion_tokens": 59, + "decode_ms": 2733.1, + "model_compute_ms": 12738.4, + "prefill_ms": 10005.3 + }, + "order": [ + "speculative", + "compiled" + ], + "speculative": { + "accept_rate": 0.8035714030265808, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "4fd23541e355718c0b1ee47f20028d7324bab11dddd5a0a5a460495dd05e3461", + "completion_tokens": 59, + "decode_ms": 2698.5, + "model_compute_ms": 12698.8, + "prefill_ms": 10000.3 + } + } + ], + "qualified": true, + "repetitions": 3 + }, + "pair_index": 1, + "speculative": { + "all_ds4_active": true, + "completion_tokens": 71, + "decode_ms": 3204.2, + "exposed_tool_wait_ms": 0.029325, + "final": { + "accept_rate": 0.75, + "completion_tokens": 12, + "content_sha256": "1f19e61b2ef79fed2c87e8d945b3cc479796bd8a235bc79ef7579f3bbc22adb3" + }, + "final_correct": true, + "graph_wall_ms": 10002.10483900446, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taskb" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 19739.2, + "model_turns": 2, + "prediction_hit": true, + "prediction_reason": null, + "prediction_source": "native-qwen3", + "prediction_status": "hit", + "predictor_ms": 200.659826, + "task_ms": 19955.812669002626, + "tool_results_count": 15, + "tool_results_sha256": "2037fc8f56106ba9bc50763cc4b82c0f3262f0ffb8682ca5ca727eaf9d46b07b", + "underlying_calls_count": 15, + "underlying_calls_sha256": "2263e8d373d489cb82f03725feec381fb4ec218f6a706ebff2de102ed1e7c40b" + }, + "stage_batched": { + "all_ds4_active": true, + "completion_tokens": 222, + "decode_ms": 8408.3, + "exposed_tool_wait_ms": 10154.79341000173, + "final": { + "accept_rate": 0.75, + "completion_tokens": 12, + "content_sha256": "1f19e61b2ef79fed2c87e8d945b3cc479796bd8a235bc79ef7579f3bbc22adb3" + }, + "final_correct": true, + "model_compute_ms": 77967.0, + "model_turns": 6, + "task_ms": 88139.69260699378, + "tool_results_count": 15, + "tool_results_sha256": "2037fc8f56106ba9bc50763cc4b82c0f3262f0ffb8682ca5ca727eaf9d46b07b", + "underlying_calls_count": 15, + "underlying_calls_sha256": "2263e8d373d489cb82f03725feec381fb4ec218f6a706ebff2de102ed1e7c40b" + }, + "task": { + "branch_count": 3, + "call_count": 15, + "id": "trace_compiled_001" + } + }, + { + "arm_order": [ + "compiled", + "stage_batched", + "speculative" + ], + "compiled": { + "all_ds4_active": true, + "completion_tokens": 73, + "decode_ms": 2993.8, + "exposed_tool_wait_ms": 10166.34874400188, + "final": { + "accept_rate": 0.75, + "completion_tokens": 16, + "content_sha256": "49780da241938f12bcf7a288b9ebc5fdd19be49df87cc9ada53ba48ee741cb5b" + }, + "final_correct": true, + "graph_wall_ms": 10167.359696002677, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taskc" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 26871.0, + "model_turns": 2, + "prediction_hit": false, + "prediction_reason": null, + "prediction_source": null, + "prediction_status": null, + "predictor_ms": 0.0, + "task_ms": 37056.5853549997, + "tool_results_count": 20, + "tool_results_sha256": "8eb88b665ebcafe693c031c44e5c724a2f3aebe00f5e726008dc316050fa211c", + "underlying_calls_count": 20, + "underlying_calls_sha256": "467ec65a34e98b78d3d4a6990fef3a6292140cb019e9ec1635fd7b5c7aa9b3ef" + }, + "interference_probe": { + "observations": [ + { + "compiled": { + "accept_rate": 0.8461538553237915, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "8450d3e1b215a7ad9c3e3b04edd56c2b099ea8677ea008e6bd13863d5c9d025b", + "completion_tokens": 57, + "decode_ms": 2318.2, + "model_compute_ms": 12788.099999999999, + "prefill_ms": 10469.9 + }, + "order": [ + "compiled", + "speculative" + ], + "speculative": { + "accept_rate": 0.8461538553237915, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "8450d3e1b215a7ad9c3e3b04edd56c2b099ea8677ea008e6bd13863d5c9d025b", + "completion_tokens": 57, + "decode_ms": 2291.2, + "model_compute_ms": 12780.7, + "prefill_ms": 10489.5 + } + }, + { + "compiled": { + "accept_rate": 0.8461538553237915, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "8450d3e1b215a7ad9c3e3b04edd56c2b099ea8677ea008e6bd13863d5c9d025b", + "completion_tokens": 57, + "decode_ms": 2286.6, + "model_compute_ms": 12764.4, + "prefill_ms": 10477.8 + }, + "order": [ + "speculative", + "compiled" + ], + "speculative": { + "accept_rate": 0.8461538553237915, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "8450d3e1b215a7ad9c3e3b04edd56c2b099ea8677ea008e6bd13863d5c9d025b", + "completion_tokens": 57, + "decode_ms": 2284.4, + "model_compute_ms": 12758.6, + "prefill_ms": 10474.2 + } + }, + { + "compiled": { + "accept_rate": 0.8461538553237915, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "8450d3e1b215a7ad9c3e3b04edd56c2b099ea8677ea008e6bd13863d5c9d025b", + "completion_tokens": 57, + "decode_ms": 3337.4, + "model_compute_ms": 13834.699999999999, + "prefill_ms": 10497.3 + }, + "order": [ + "compiled", + "speculative" + ], + "speculative": { + "accept_rate": 0.8461538553237915, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "8450d3e1b215a7ad9c3e3b04edd56c2b099ea8677ea008e6bd13863d5c9d025b", + "completion_tokens": 57, + "decode_ms": 2293.6, + "model_compute_ms": 12788.300000000001, + "prefill_ms": 10494.7 + } + } + ], + "qualified": true, + "repetitions": 3 + }, + "pair_index": 2, + "speculative": { + "all_ds4_active": true, + "completion_tokens": 73, + "decode_ms": 3002.2999999999997, + "exposed_tool_wait_ms": 0.029506, + "final": { + "accept_rate": 0.75, + "completion_tokens": 16, + "content_sha256": "49780da241938f12bcf7a288b9ebc5fdd19be49df87cc9ada53ba48ee741cb5b" + }, + "final_correct": true, + "graph_wall_ms": 10002.156002999982, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taskc" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 20779.4, + "model_turns": 2, + "prediction_hit": true, + "prediction_reason": null, + "prediction_source": "native-qwen3", + "prediction_status": "hit", + "predictor_ms": 206.06213, + "task_ms": 21002.575256003183, + "tool_results_count": 20, + "tool_results_sha256": "8eb88b665ebcafe693c031c44e5c724a2f3aebe00f5e726008dc316050fa211c", + "underlying_calls_count": 20, + "underlying_calls_sha256": "467ec65a34e98b78d3d4a6990fef3a6292140cb019e9ec1635fd7b5c7aa9b3ef" + }, + "stage_batched": { + "all_ds4_active": true, + "completion_tokens": 200, + "decode_ms": 6476.0, + "exposed_tool_wait_ms": 10170.707518991549, + "final": { + "accept_rate": 1.0, + "completion_tokens": 16, + "content_sha256": "49780da241938f12bcf7a288b9ebc5fdd19be49df87cc9ada53ba48ee741cb5b" + }, + "final_correct": true, + "model_compute_ms": 80176.6, + "model_turns": 6, + "task_ms": 90366.67264700372, + "tool_results_count": 20, + "tool_results_sha256": "8eb88b665ebcafe693c031c44e5c724a2f3aebe00f5e726008dc316050fa211c", + "underlying_calls_count": 20, + "underlying_calls_sha256": "467ec65a34e98b78d3d4a6990fef3a6292140cb019e9ec1635fd7b5c7aa9b3ef" + }, + "task": { + "branch_count": 4, + "call_count": 20, + "id": "trace_compiled_002" + } + }, + { + "arm_order": [ + "speculative", + "stage_batched", + "compiled" + ], + "compiled": { + "all_ds4_active": true, + "completion_tokens": 46, + "decode_ms": 2314.2, + "exposed_tool_wait_ms": 10128.93626299774, + "final": { + "accept_rate": 1.0, + "completion_tokens": 11, + "content_sha256": "dd0930fae404b66f022f2c55ec2733331ee7a67d0aba2a63cd1437bf864c5c28" + }, + "final_correct": true, + "graph_wall_ms": 10129.702878002718, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taskd" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 17697.1, + "model_turns": 2, + "prediction_hit": false, + "prediction_reason": null, + "prediction_source": null, + "prediction_status": null, + "predictor_ms": 0.0, + "task_ms": 27845.55858699605, + "tool_results_count": 10, + "tool_results_sha256": "c2041b2f6e3aeddaafa6ffb35bcd8affbc01e75cc62de4258206a53a6559a771", + "underlying_calls_count": 10, + "underlying_calls_sha256": "ff04b3b23c0147caffa852e474ed9f9fec517c227e12674dfacdba771bf34e33" + }, + "interference_probe": { + "observations": [ + { + "compiled": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "f1e79cd991e36c3b41fca5f4ed7143a388d932406914df86b7e5b525295e279c", + "completion_tokens": 35, + "decode_ms": 1095.7, + "model_compute_ms": 10661.300000000001, + "prefill_ms": 9565.6 + }, + "order": [ + "speculative", + "compiled" + ], + "speculative": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "f1e79cd991e36c3b41fca5f4ed7143a388d932406914df86b7e5b525295e279c", + "completion_tokens": 35, + "decode_ms": 1088.0, + "model_compute_ms": 10659.5, + "prefill_ms": 9571.5 + } + }, + { + "compiled": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "f1e79cd991e36c3b41fca5f4ed7143a388d932406914df86b7e5b525295e279c", + "completion_tokens": 35, + "decode_ms": 1102.7, + "model_compute_ms": 10700.900000000001, + "prefill_ms": 9598.2 + }, + "order": [ + "compiled", + "speculative" + ], + "speculative": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "f1e79cd991e36c3b41fca5f4ed7143a388d932406914df86b7e5b525295e279c", + "completion_tokens": 35, + "decode_ms": 1089.7, + "model_compute_ms": 10701.7, + "prefill_ms": 9612.0 + } + }, + { + "compiled": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "f1e79cd991e36c3b41fca5f4ed7143a388d932406914df86b7e5b525295e279c", + "completion_tokens": 35, + "decode_ms": 1090.2, + "model_compute_ms": 10679.5, + "prefill_ms": 9589.3 + }, + "order": [ + "speculative", + "compiled" + ], + "speculative": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "f1e79cd991e36c3b41fca5f4ed7143a388d932406914df86b7e5b525295e279c", + "completion_tokens": 35, + "decode_ms": 1100.2, + "model_compute_ms": 10679.900000000001, + "prefill_ms": 9579.7 + } + } + ], + "qualified": true, + "repetitions": 3 + }, + "pair_index": 3, + "speculative": { + "all_ds4_active": true, + "completion_tokens": 46, + "decode_ms": 1348.0, + "exposed_tool_wait_ms": 0.021149, + "final": { + "accept_rate": 1.0, + "completion_tokens": 11, + "content_sha256": "dd0930fae404b66f022f2c55ec2733331ee7a67d0aba2a63cd1437bf864c5c28" + }, + "final_correct": true, + "graph_wall_ms": 10002.26961899898, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taskd" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 20244.3, + "model_turns": 2, + "prediction_hit": true, + "prediction_reason": null, + "prediction_source": "native-qwen3", + "prediction_status": "hit", + "predictor_ms": 199.397724, + "task_ms": 20496.583997002745, + "tool_results_count": 10, + "tool_results_sha256": "c2041b2f6e3aeddaafa6ffb35bcd8affbc01e75cc62de4258206a53a6559a771", + "underlying_calls_count": 10, + "underlying_calls_sha256": "ff04b3b23c0147caffa852e474ed9f9fec517c227e12674dfacdba771bf34e33" + }, + "stage_batched": { + "all_ds4_active": true, + "completion_tokens": 195, + "decode_ms": 6618.0, + "exposed_tool_wait_ms": 10132.728425989626, + "final": { + "accept_rate": 1.0, + "completion_tokens": 11, + "content_sha256": "dd0930fae404b66f022f2c55ec2733331ee7a67d0aba2a63cd1437bf864c5c28" + }, + "final_correct": true, + "model_compute_ms": 71275.7, + "model_turns": 6, + "task_ms": 81427.70019899763, + "tool_results_count": 10, + "tool_results_sha256": "c2041b2f6e3aeddaafa6ffb35bcd8affbc01e75cc62de4258206a53a6559a771", + "underlying_calls_count": 10, + "underlying_calls_sha256": "ff04b3b23c0147caffa852e474ed9f9fec517c227e12674dfacdba771bf34e33" + }, + "task": { + "branch_count": 2, + "call_count": 10, + "id": "trace_compiled_003" + } + }, + { + "arm_order": [ + "speculative", + "stage_batched", + "compiled" + ], + "compiled": { + "all_ds4_active": true, + "completion_tokens": 50, + "decode_ms": 1653.7, + "exposed_tool_wait_ms": 10150.936931000615, + "final": { + "accept_rate": 0.9166666865348816, + "completion_tokens": 14, + "content_sha256": "752191b12e47a14803f099a3a92e2ffa50510c7002045c5dbcd0a3db939f53a8" + }, + "final_correct": true, + "graph_wall_ms": 10152.029717006371, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taske" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 18391.899999999998, + "model_turns": 2, + "prediction_hit": false, + "prediction_reason": null, + "prediction_source": null, + "prediction_status": null, + "predictor_ms": 0.0, + "task_ms": 28562.6701939982, + "tool_results_count": 15, + "tool_results_sha256": "968113db6b1f63c186b8124472fb64ac38b8d81957a7d9a05898a41eb23af577", + "underlying_calls_count": 15, + "underlying_calls_sha256": "0be2134f957d10c601ed26e39380751ede7d2b05ec5419d36680e9f981c2ba57" + }, + "interference_probe": { + "observations": [ + { + "compiled": { + "accept_rate": 1.0, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "1212a82be816acf93594e58cc76a9350c85d83b9dfc40c8021865dbd9e92e780", + "completion_tokens": 36, + "decode_ms": 2153.3, + "model_compute_ms": 12313.2, + "prefill_ms": 10159.9 + }, + "order": [ + "compiled", + "speculative" + ], + "speculative": { + "accept_rate": 1.0, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "1212a82be816acf93594e58cc76a9350c85d83b9dfc40c8021865dbd9e92e780", + "completion_tokens": 36, + "decode_ms": 1144.8, + "model_compute_ms": 11316.0, + "prefill_ms": 10171.2 + } + }, + { + "compiled": { + "accept_rate": 1.0, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "1212a82be816acf93594e58cc76a9350c85d83b9dfc40c8021865dbd9e92e780", + "completion_tokens": 36, + "decode_ms": 1142.1, + "model_compute_ms": 11324.6, + "prefill_ms": 10182.5 + }, + "order": [ + "speculative", + "compiled" + ], + "speculative": { + "accept_rate": 1.0, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "1212a82be816acf93594e58cc76a9350c85d83b9dfc40c8021865dbd9e92e780", + "completion_tokens": 36, + "decode_ms": 1135.1, + "model_compute_ms": 11336.5, + "prefill_ms": 10201.4 + } + }, + { + "compiled": { + "accept_rate": 1.0, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "1212a82be816acf93594e58cc76a9350c85d83b9dfc40c8021865dbd9e92e780", + "completion_tokens": 36, + "decode_ms": 1137.8, + "model_compute_ms": 11313.9, + "prefill_ms": 10176.1 + }, + "order": [ + "compiled", + "speculative" + ], + "speculative": { + "accept_rate": 1.0, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "1212a82be816acf93594e58cc76a9350c85d83b9dfc40c8021865dbd9e92e780", + "completion_tokens": 36, + "decode_ms": 1142.4, + "model_compute_ms": 11314.699999999999, + "prefill_ms": 10172.3 + } + } + ], + "qualified": true, + "repetitions": 3 + }, + "pair_index": 4, + "speculative": { + "all_ds4_active": true, + "completion_tokens": 50, + "decode_ms": 1975.0, + "exposed_tool_wait_ms": 0.035947, + "final": { + "accept_rate": 0.9166666865348816, + "completion_tokens": 14, + "content_sha256": "752191b12e47a14803f099a3a92e2ffa50510c7002045c5dbcd0a3db939f53a8" + }, + "final_correct": true, + "graph_wall_ms": 10001.882305004983, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taske" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 24415.8, + "model_turns": 2, + "prediction_hit": true, + "prediction_reason": null, + "prediction_source": "native-qwen3", + "prediction_status": "hit", + "predictor_ms": 207.393702, + "task_ms": 24634.228317001543, + "tool_results_count": 15, + "tool_results_sha256": "968113db6b1f63c186b8124472fb64ac38b8d81957a7d9a05898a41eb23af577", + "underlying_calls_count": 15, + "underlying_calls_sha256": "0be2134f957d10c601ed26e39380751ede7d2b05ec5419d36680e9f981c2ba57" + }, + "stage_batched": { + "all_ds4_active": true, + "completion_tokens": 202, + "decode_ms": 6633.0, + "exposed_tool_wait_ms": 10162.887673999649, + "final": { + "accept_rate": 0.9166666865348816, + "completion_tokens": 14, + "content_sha256": "752191b12e47a14803f099a3a92e2ffa50510c7002045c5dbcd0a3db939f53a8" + }, + "final_correct": true, + "model_compute_ms": 77673.4, + "model_turns": 6, + "task_ms": 87855.36901299929, + "tool_results_count": 15, + "tool_results_sha256": "968113db6b1f63c186b8124472fb64ac38b8d81957a7d9a05898a41eb23af577", + "underlying_calls_count": 15, + "underlying_calls_sha256": "0be2134f957d10c601ed26e39380751ede7d2b05ec5419d36680e9f981c2ba57" + }, + "task": { + "branch_count": 3, + "call_count": 15, + "id": "trace_compiled_004" + } + }, + { + "arm_order": [ + "stage_batched", + "compiled", + "speculative" + ], + "compiled": { + "all_ds4_active": true, + "completion_tokens": 50, + "decode_ms": 1986.2, + "exposed_tool_wait_ms": 10164.91345599934, + "final": { + "accept_rate": 0.6875, + "completion_tokens": 15, + "content_sha256": "ecb16885e5e4c6b794e1f17f503d17c6dd38105056884d943ac32afb2caa7c0a" + }, + "final_correct": true, + "graph_wall_ms": 10166.974724998, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taskf" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 26039.8, + "model_turns": 2, + "prediction_hit": false, + "prediction_reason": null, + "prediction_source": null, + "prediction_status": null, + "predictor_ms": 0.0, + "task_ms": 36215.08621699468, + "tool_results_count": 20, + "tool_results_sha256": "de71e45a390fca728e07a8bf4cdc467e91604fc52f68abe6ed0bd75ca77e87d2", + "underlying_calls_count": 20, + "underlying_calls_sha256": "bd91a2316a76490265d64557860d9a2acdc97a8f1a18014dacaa0a21cd241e2f" + }, + "interference_probe": { + "observations": [ + { + "compiled": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "bb4924fb407a01a832f9787a134f2e9494c39afaa14fe4541e5d693a73b9845d", + "completion_tokens": 35, + "decode_ms": 1161.6, + "model_compute_ms": 11761.300000000001, + "prefill_ms": 10599.7 + }, + "order": [ + "speculative", + "compiled" + ], + "speculative": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "bb4924fb407a01a832f9787a134f2e9494c39afaa14fe4541e5d693a73b9845d", + "completion_tokens": 35, + "decode_ms": 1167.4, + "model_compute_ms": 11739.8, + "prefill_ms": 10572.4 + } + }, + { + "compiled": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "bb4924fb407a01a832f9787a134f2e9494c39afaa14fe4541e5d693a73b9845d", + "completion_tokens": 35, + "decode_ms": 1179.7, + "model_compute_ms": 11772.800000000001, + "prefill_ms": 10593.1 + }, + "order": [ + "compiled", + "speculative" + ], + "speculative": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "bb4924fb407a01a832f9787a134f2e9494c39afaa14fe4541e5d693a73b9845d", + "completion_tokens": 35, + "decode_ms": 1172.7, + "model_compute_ms": 11755.5, + "prefill_ms": 10582.8 + } + }, + { + "compiled": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "bb4924fb407a01a832f9787a134f2e9494c39afaa14fe4541e5d693a73b9845d", + "completion_tokens": 35, + "decode_ms": 1164.9, + "model_compute_ms": 11746.9, + "prefill_ms": 10582.0 + }, + "order": [ + "speculative", + "compiled" + ], + "speculative": { + "accept_rate": 0.9642857313156128, + "cache_hit": false, + "cached_prefix_tokens": 0, + "call_sha256": "bb4924fb407a01a832f9787a134f2e9494c39afaa14fe4541e5d693a73b9845d", + "completion_tokens": 35, + "decode_ms": 1165.9, + "model_compute_ms": 11780.699999999999, + "prefill_ms": 10614.8 + } + } + ], + "qualified": true, + "repetitions": 3 + }, + "pair_index": 5, + "speculative": { + "all_ds4_active": true, + "completion_tokens": 50, + "decode_ms": 1985.9, + "exposed_tool_wait_ms": 0.033042, + "final": { + "accept_rate": 0.6875, + "completion_tokens": 15, + "content_sha256": "ecb16885e5e4c6b794e1f17f503d17c6dd38105056884d943ac32afb2caa7c0a" + }, + "final_correct": true, + "graph_wall_ms": 10002.439897994918, + "macro_call": { + "arguments": { + "workflow_ref": "workflow_taskf" + }, + "name": "execute_customer_workflows" + }, + "macro_correct": true, + "model_compute_ms": 20233.4, + "model_turns": 2, + "prediction_hit": true, + "prediction_reason": null, + "prediction_source": "native-qwen3", + "prediction_status": "hit", + "predictor_ms": 201.386338, + "task_ms": 20451.380637998227, + "tool_results_count": 20, + "tool_results_sha256": "de71e45a390fca728e07a8bf4cdc467e91604fc52f68abe6ed0bd75ca77e87d2", + "underlying_calls_count": 20, + "underlying_calls_sha256": "bd91a2316a76490265d64557860d9a2acdc97a8f1a18014dacaa0a21cd241e2f" + }, + "stage_batched": { + "all_ds4_active": true, + "completion_tokens": 261, + "decode_ms": 10547.4, + "exposed_tool_wait_ms": 10179.923661002249, + "final": { + "accept_rate": 0.9166666865348816, + "completion_tokens": 15, + "content_sha256": "ecb16885e5e4c6b794e1f17f503d17c6dd38105056884d943ac32afb2caa7c0a" + }, + "final_correct": true, + "model_compute_ms": 86333.0, + "model_turns": 6, + "task_ms": 96537.8686490003, + "tool_results_count": 20, + "tool_results_sha256": "de71e45a390fca728e07a8bf4cdc467e91604fc52f68abe6ed0bd75ca77e87d2", + "underlying_calls_count": 20, + "underlying_calls_sha256": "bd91a2316a76490265d64557860d9a2acdc97a8f1a18014dacaa0a21cd241e2f" + }, + "task": { + "branch_count": 4, + "call_count": 20, + "id": "trace_compiled_005" + } + } + ], + "pattern": { + "fingerprint": "06d95882b485daf3f30f3fc12ea62ccfd18656de9deafa2f4ed77c49bb8c0645", + "macro_name": "execute_customer_workflows", + "model_training": false, + "root_fields": [ + "customer_email", + "destination" + ], + "side_effects_allowed": false, + "steps": [ + { + "arguments": { + "customer_email": { + "key": "customer_email", + "source": "root" + } + }, + "tool": "resolve_customer" + }, + { + "arguments": { + "customer_ref": { + "key": "call_ref", + "source": "previous_result" + } + }, + "tool": "list_open_orders" + }, + { + "arguments": { + "orders_ref": { + "key": "call_ref", + "source": "previous_result" + } + }, + "tool": "get_order_details" + }, + { + "arguments": { + "destination": { + "key": "destination", + "source": "root" + }, + "order_ref": { + "key": "call_ref", + "source": "previous_result" + } + }, + "tool": "calculate_shipping" + }, + { + "arguments": { + "shipping_ref": { + "key": "call_ref", + "source": "previous_result" + } + }, + "tool": "prepare_customer_summary" + } + ], + "training_report": "results/multiturn-cached-wordref-production-6tasks.json", + "training_report_sha256": "2475697d418bffed0e9668da26ce6c88a85a952ce97d99749f440f97f9ac5bf9", + "training_traces": 2, + "workflow_registry": "results/trace-workflow-registry.json", + "workflow_registry_sha256": "0b73cb4f45284f07a4d6890906d1eaf5a25469c2b80eb542b1d74f1e3a9a0e1b" + }, + "privacy_miss": { + "authoritative_call_sha256": "26b31bf2044cd34ec9ea91d14b9758544b6a4a62a533a58bc4f79af1472a6535", + "passed": true, + "prediction_sha256": "4fd23541e355718c0b1ee47f20028d7324bab11dddd5a0a5a460495dd05e3461", + "private_result_exposed": false, + "reason": "invocation_mismatch", + "status": "miss" + }, + "production_gate": { + "checks": { + "calls_stable": true, + "decode_slowdown_p50": true, + "decode_slowdown_p95": true, + "ds4_active": true, + "end_to_end_ci": true, + "end_to_end_speedup": true, + "end_to_end_tail": true, + "final_answers_correct": true, + "final_outputs_stable": true, + "interference_probes": true, + "macro_calls_correct": true, + "macro_outputs_stable": true, + "model_slowdown_p50": true, + "model_slowdown_p95": true, + "prediction_hit_rate": true, + "prediction_source": true, + "prefix_cache_configured": true, + "private_miss_result_hidden": true, + "sample_size": true, + "speculation_incremental_ci": true, + "speculation_incremental_gain": true, + "tool_results_stable": true + }, + "passed": true, + "thresholds": { + "max_decode_slowdown_p95_percent": 5.0, + "max_decode_slowdown_percent": 1.0, + "max_model_slowdown_p95_percent": 5.0, + "max_model_slowdown_percent": 1.0, + "min_e2e_speedup": 2.0, + "min_e2e_speedup_p05": 1.5, + "min_incremental_speedup": 1.05, + "min_production_pairs": 6 + } + }, + "schema_version": 1, + "server_snapshot": { + "model": { + "arch": "deepseek4", + "draft_path": null, + "tokenizer_id": null + }, + "prefix_cache_after": { + "capacity": 32, + "in_use": 0, + "lifetime_hits": 0 + }, + "prefix_cache_before": { + "capacity": 32, + "in_use": 0, + "lifetime_hits": 0 + }, + "tool_speculation": { + "allowed_tools": [ + "calculate_shipping", + "execute_customer_workflows", + "get_order_details", + "list_open_orders", + "prepare_customer_summary", + "resolve_customer" + ], + "automatic_prediction_enabled": true, + "client_prediction_required": false, + "client_result_handling_required": true, + "compute_isolation": "disjoint_cpu_affinity", + "cpu_affinity_isolated": true, + "enabled": true, + "execution_mode": "child_process_cpu_affinity", + "executor_contract": "child_process_cpu_affinity", + "max_model_slowdown_ratio": 1.05, + "model_cpu_affinity": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29 + ], + "prediction_confidence": 0.75, + "prediction_source": "native-qwen3", + "predictor_decode_isolated": true, + "predictor_schedule": "before-model", + "preserves_token_speculation": true, + "profile_lanes": [ + { + "accelerator_relation": "non_accelerator", + "decode_interference_qualified": true, + "model_slowdown_ratio": 1.0011341375399527, + "resource_percentage": 100 + } + ], + "profile_status": "qualified", + "protocol": "dflash.tool-speculation.v1", + "tool_cpu_affinity": [ + 14, + 15, + 30, + 31 + ], + "unqualified_lane_policy": "defer" + } + }, + "summary": { + "all_calls_stable": true, + "all_ds4_active": true, + "all_final_answers_correct": true, + "all_final_outputs_stable": true, + "all_interference_probes_qualified": true, + "all_macro_calls_correct": true, + "all_predictions_from_qwen": true, + "all_tool_results_stable": true, + "calls_per_task": [ + 10, + 15, + 20, + 10, + 15, + 20 + ], + "compiled_exposed_tool_wait_p50_ms": 10148.645039000257, + "compiled_model_turns_p50": 2.0, + "compiled_task_p50_ms": 32876.98137099869, + "compiled_to_speculative_bootstrap_95ci": [ + 1.2590085918648417, + 1.800192103452801 + ], + "compiled_to_speculative_speedup_p05": 1.2092397320590198, + "compiled_to_speculative_speedup_p50": 1.7675861222850164, + "continuation_cache_hit_rate": 0.14285714285714285, + "decode_slowdown_p50_percent": -0.6578265762836599, + "decode_slowdown_p95_percent": 0.3737683368205902, + "macro_output_stability_rate": 1.0, + "model_compute_slowdown_p50_percent": -0.02706040193529713, + "model_compute_slowdown_p95_percent": 0.09042853676124452, + "pattern_prediction_hit_rate": 1.0, + "predictor_p50_ms": 201.023082, + "prefix_cache_configured": true, + "prefix_cache_lifetime_hit_delta": 0, + "private_miss_result_hidden": true, + "speculative_exposed_tool_wait_p50_ms": 0.0303775, + "speculative_task_p50_ms": 20473.982317500486, + "speedup_by_call_count": { + "10": { + "combined_speedup_p50": 4.4513637744591055, + "compiled_task_p50_ms": 29030.47904399864, + "speculation_speedup_p50": 1.589033888881625, + "speculative_task_p50_ms": 18551.40907099849, + "stage_batched_task_p50_ms": 81648.07293250124, + "tasks": 2 + }, + "15": { + "combined_speedup_p50": 3.9915685175553506, + "compiled_task_p50_ms": 32050.616717497178, + "speculation_speedup_p50": 1.470166806436018, + "speculative_task_p50_ms": 22295.020493002085, + "stage_batched_task_p50_ms": 87997.53080999653, + "tasks": 2 + }, + "20": { + "combined_speedup_p50": 4.511503368783014, + "compiled_task_p50_ms": 36635.83578599719, + "speculation_speedup_p50": 1.7675861222850164, + "speculative_task_p50_ms": 20726.977947000705, + "stage_batched_task_p50_ms": 93452.27064800201, + "tasks": 2 + } + }, + "stage_batched_exposed_tool_wait_p50_ms": 10158.84054200069, + "stage_batched_model_turns_p50": 6.0, + "stage_batched_task_p50_ms": 87997.53080999653, + "stage_batched_to_compiled_speedup_p50": 2.687587552693997, + "stage_batched_to_speculative_bootstrap_95ci": [ + 3.769569566480701, + 4.8251710527069305 + ], + "stage_batched_to_speculative_speedup_min": 3.5663942008836984, + "stage_batched_to_speculative_speedup_p05": 3.6679818836822, + "stage_batched_to_speculative_speedup_p50": 4.359695041609839, + "tasks": 6, + "total_wall_speedup": 4.272913990402306 + }, + "workload": { + "branches_per_task": "2-4", + "calls_per_task": "10-20", + "dependency": "five serial calls per branch; branches are independent", + "tasks": 6, + "tool_adapter": "deterministic read-only 2-second API replay" + } +} diff --git a/optimizations/ooo_spec_lucebox5_cpu/results/trace-workflow-registry.json b/optimizations/ooo_spec_lucebox5_cpu/results/trace-workflow-registry.json new file mode 100644 index 000000000..cd42484d2 --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/results/trace-workflow-registry.json @@ -0,0 +1,121 @@ +{ + "pattern_fingerprint": "06d95882b485daf3f30f3fc12ea62ccfd18656de9deafa2f4ed77c49bb8c0645", + "schema_version": 1, + "workflows": { + "workflow_taska": { + "items": [ + { + "customer_email": "agent-taska-taska@example.test", + "destination": "Rome" + }, + { + "customer_email": "agent-taska-taskb@example.test", + "destination": "Milan" + } + ], + "pattern_fingerprint": "06d95882b485daf3f30f3fc12ea62ccfd18656de9deafa2f4ed77c49bb8c0645" + }, + "workflow_taskb": { + "items": [ + { + "customer_email": "agent-taskb-taskcw@example.test", + "destination": "Naples" + }, + { + "customer_email": "agent-taskb-taskcx@example.test", + "destination": "Rome" + }, + { + "customer_email": "agent-taskb-taskcy@example.test", + "destination": "Milan" + } + ], + "pattern_fingerprint": "06d95882b485daf3f30f3fc12ea62ccfd18656de9deafa2f4ed77c49bb8c0645" + }, + "workflow_taskc": { + "items": [ + { + "customer_email": "agent-taskc-taskgs@example.test", + "destination": "Florence" + }, + { + "customer_email": "agent-taskc-taskgt@example.test", + "destination": "Naples" + }, + { + "customer_email": "agent-taskc-taskgv@example.test", + "destination": "Milan" + }, + { + "customer_email": "agent-taskc-taskhb@example.test", + "destination": "Milan" + } + ], + "pattern_fingerprint": "06d95882b485daf3f30f3fc12ea62ccfd18656de9deafa2f4ed77c49bb8c0645" + }, + "workflow_taskd": { + "items": [ + { + "customer_email": "agent-taskd-taskko@example.test", + "destination": "Bologna" + }, + { + "customer_email": "agent-taskd-taskkp@example.test", + "destination": "Florence" + } + ], + "pattern_fingerprint": "06d95882b485daf3f30f3fc12ea62ccfd18656de9deafa2f4ed77c49bb8c0645" + }, + "workflow_taske": { + "items": [ + { + "customer_email": "agent-taske-taskok@example.test", + "destination": "Turin" + }, + { + "customer_email": "agent-taske-taskom@example.test", + "destination": "Florence" + }, + { + "customer_email": "agent-taske-taskon@example.test", + "destination": "Naples" + } + ], + "pattern_fingerprint": "06d95882b485daf3f30f3fc12ea62ccfd18656de9deafa2f4ed77c49bb8c0645" + }, + "workflow_taskf": { + "items": [ + { + "customer_email": "agent-taskf-tasksg@example.test", + "destination": "Milan" + }, + { + "customer_email": "agent-taskf-tasksh@example.test", + "destination": "Turin" + }, + { + "customer_email": "agent-taskf-tasksi@example.test", + "destination": "Bologna" + }, + { + "customer_email": "agent-taskf-tasksz@example.test", + "destination": "Turin" + } + ], + "pattern_fingerprint": "06d95882b485daf3f30f3fc12ea62ccfd18656de9deafa2f4ed77c49bb8c0645" + }, + "workflow_warmalm": { + "items": [ + { + "customer_email": "agent-warmalm-taska@example.test", + "destination": "Turin" + }, + { + "customer_email": "agent-warmalm-taskb@example.test", + "destination": "Bologna" + } + ], + "pattern_fingerprint": "06d95882b485daf3f30f3fc12ea62ccfd18656de9deafa2f4ed77c49bb8c0645" + } + } +} diff --git a/optimizations/ooo_spec_lucebox5_cpu/run_native_cpu_server_lucebox5.sh b/optimizations/ooo_spec_lucebox5_cpu/run_native_cpu_server_lucebox5.sh new file mode 100755 index 000000000..a99698ad9 --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/run_native_cpu_server_lucebox5.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="/home/lucebox5" +experiment="$root/tool-spec-cpu-20260813" +launcher="${LUCEBOX_LAUNCHER:-$experiment/run-deepseek-0731-cpu-tool.sh}" +executor="${TOOL_SPEC_EXECUTOR:-$experiment/cpu_sparse_tool_executor}" +profile="${TOOL_SPEC_PROFILE:-$experiment/profiles/lucebox5-cpu-lane-qualified.json}" +allowed="${TOOL_SPEC_ALLOW:-benchmark_cpu_sparse}" +native_wrapper_dir="${NATIVE_WRAPPER_DIR:-$experiment/native-wrapper}" +candidate_link="$native_wrapper_dir/candidate-build" +predictor_model="$experiment/models/Qwen3-0.6B-Q8_0.gguf" + +[[ -L "$candidate_link" ]] || { + printf 'missing durable candidate-build symlink: %s\n' "$candidate_link" >&2 + exit 2 +} +if ! candidate_build="$(readlink -f -- "$candidate_link")"; then + printf 'cannot resolve candidate-build symlink: %s\n' "$candidate_link" >&2 + exit 2 +fi +[[ -n "$candidate_build" && -d "$candidate_build" ]] || { + printf 'invalid candidate-build symlink: %s\n' "$candidate_link" >&2 + exit 2 +} + +for binary in \ + "$launcher" \ + "$executor" \ + "$native_wrapper_dir/dflash_server" \ + "$candidate_build/dflash_server" \ + "$candidate_build/backend_ipc_daemon"; do + [[ -f "$binary" && -x "$binary" ]] || { + printf 'required binary is not executable: %s\n' "$binary" >&2 + exit 2 + } +done +for data_file in "$profile" "$predictor_model"; do + [[ -f "$data_file" ]] || { + printf 'required data file is not regular: %s\n' "$data_file" >&2 + exit 2 + } +done +if pgrep -x dflash_server >/dev/null; then + printf 'a dflash_server is already running; refusing to overlap it\n' >&2 + exit 75 +fi +if fuser -s /dev/kfd 2>/dev/null; then + printf '/dev/kfd already has an owner; refusing to overlap it\n' >&2 + fuser -v /dev/kfd >&2 || true + exit 75 +fi + +exec env -i \ + HOME="$root" \ + USER="lucebox5" \ + PATH="$root/.local/bin:/opt/rocm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" \ + ENGINE_DIR="$root/lucebox-engine-0731" \ + BUILD_DIR="$native_wrapper_dir" \ + QUALIFIED_CONFIG_DIR="/opt/lucebox-manage/qualified/r9700_deepseek/runtime-config" \ + TARGET_MODEL="$root/lucebox-models/DeepSeek-V4-Flash-0731-ROCMFPX-MIX-STRIX.gguf" \ + DRAFT_MODEL="$root/lucebox-models/DeepSeek-V4-Flash-0731-DSpark-draft-Q4RMFP4-denseF16.gguf" \ + SERVER_PORT="18145" \ + MODEL_CPU_AFFINITY="0-13,16-29" \ + TOOL_SPEC_EXECUTOR="$executor" \ + TOOL_SPEC_PROFILE="$profile" \ + TOOL_SPEC_ALLOW="$allowed" \ + TOOL_SPEC_CPU_AFFINITY="14-15,30-31" \ + TOOL_SPEC_MAX_MODEL_SLOWDOWN="1.05" \ + LUCEBOX_INFERENCE_PROFILE="quality" \ + "$launcher" diff --git a/optimizations/ooo_spec_lucebox5_cpu/test_benchmark_cpu_tool_speculation.py b/optimizations/ooo_spec_lucebox5_cpu/test_benchmark_cpu_tool_speculation.py new file mode 100644 index 000000000..92377a2f7 --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/test_benchmark_cpu_tool_speculation.py @@ -0,0 +1,194 @@ +from __future__ import annotations + +import argparse +import signal +import subprocess +import unittest +from unittest.mock import Mock, patch + +from benchmark_cpu_tool_speculation import ( + TOOL_NAME, + expected_arguments, + normalize_tool_call, + parse_cpu_list, + percentile, + props_url, + require_qualified_cpu_tool_props, + request_body, + stop_executor, +) + + +class CpuToolSpeculationBenchmarkTest(unittest.TestCase): + def test_stop_executor_signals_group_after_leader_exit(self) -> None: + process = Mock(pid=4321) + process.poll.return_value = 0 + + def kill_group(_pgid: int, sent_signal: int) -> None: + if sent_signal == 0: + raise ProcessLookupError + + with patch( + "benchmark_cpu_tool_speculation.os.killpg", + side_effect=kill_group, + ) as killpg: + stop_executor({"process": process, "pgid": 4321}) + + killpg.assert_any_call(4321, signal.SIGTERM) + + def test_stop_executor_converts_final_wait_timeout(self) -> None: + process = Mock(pid=4321) + process.poll.return_value = None + process.wait.side_effect = subprocess.TimeoutExpired("executor", 5.0) + + def kill_group(_pgid: int, sent_signal: int) -> None: + if sent_signal == 0: + raise ProcessLookupError + + with patch( + "benchmark_cpu_tool_speculation.os.killpg", + side_effect=kill_group, + ): + with self.assertRaisesRegex(RuntimeError, "did not stop"): + stop_executor({"process": process, "pgid": 4321}) + + def test_stop_executor_waits_for_group_after_sigkill(self) -> None: + process = Mock(pid=4321) + process.poll.return_value = 0 + alive = True + + def kill_group(_pgid: int, sent_signal: int) -> None: + nonlocal alive + if sent_signal == signal.SIGKILL: + alive = False + elif sent_signal == 0 and not alive: + raise ProcessLookupError + + with ( + patch( + "benchmark_cpu_tool_speculation.os.killpg", + side_effect=kill_group, + ) as killpg, + patch( + "benchmark_cpu_tool_speculation.time.monotonic", + side_effect=[0.0, 2.0, 3.0], + ), + ): + stop_executor({"process": process, "pgid": 4321}) + + calls = [call.args[1] for call in killpg.call_args_list] + self.assertEqual(calls, [signal.SIGTERM, 0, signal.SIGKILL, 0]) + + def test_stop_executor_reports_permission_failure_cleanly(self) -> None: + process = Mock(pid=4321) + process.poll.return_value = 0 + + with ( + patch( + "benchmark_cpu_tool_speculation.os.killpg", + side_effect=PermissionError, + ), + patch( + "benchmark_cpu_tool_speculation.time.monotonic", + side_effect=[0.0, 2.0, 3.0, 9.0], + ), + ): + with self.assertRaisesRegex(RuntimeError, "did not stop"): + stop_executor({"process": process, "pgid": 4321}) + + def test_cpu_list_parser_canonicalizes_ranges(self) -> None: + self.assertEqual(parse_cpu_list("30-31,15,14-15"), [14, 15, 30, 31]) + with self.assertRaises(argparse.ArgumentTypeError): + parse_cpu_list("14,,15") + + def test_request_has_exact_concrete_prediction(self) -> None: + arguments = expected_arguments(4096, 16, 77, 2, 731) + self.assertEqual(arguments, {"iterations": 77}) + body = request_body(arguments, 32, prediction=arguments) + self.assertEqual( + body["tool_speculation"]["call"], + {"name": TOOL_NAME, "arguments": arguments}, + ) + self.assertEqual(body["tool_speculation"]["confidence"], 1.0) + self.assertEqual(body["tools"][0]["name"], TOOL_NAME) + + def test_props_url_uses_server_origin(self) -> None: + self.assertEqual( + props_url("http://127.0.0.1:18145/v1/chat/completions?x=1"), + "http://127.0.0.1:18145/props", + ) + + def test_percentile_interpolates_sorted_values(self) -> None: + self.assertEqual(percentile([4, 1, 3, 2], 0.0), 1.0) + self.assertEqual(percentile([4, 1, 3, 2], 0.5), 2.5) + self.assertEqual(percentile([4, 1, 3, 2], 1.0), 4.0) + + def test_automatic_qwen_arm_has_no_oracle_prediction(self) -> None: + arguments = {"iterations": 77} + body = request_body( + arguments, + 32, + prediction=None, + automatic_prediction=True, + tool_choice="required", + ) + self.assertNotIn("tool_speculation", body) + self.assertTrue(body["automatic_tool_speculation"]) + self.assertEqual(body["tool_choice"], "required") + + def test_normalizes_deepseek_single_parameter_envelope(self) -> None: + result = { + "choices": [ + { + "message": { + "content": ( + '{"function":"batch_resolve_customer",' + '"parameter":"stage_ref",' + '"parameter_value":"workflow_taskf_stage_one"}' + ) + } + } + ] + } + self.assertEqual( + normalize_tool_call(result), + { + "name": "batch_resolve_customer", + "arguments": {"stage_ref": "workflow_taskf_stage_one"}, + }, + ) + + def test_automatic_gate_requires_qualified_disjoint_lane(self) -> None: + args = argparse.Namespace(tool_cpus=[14, 15]) + tool_props = { + "enabled": True, + "automatic_prediction_enabled": True, + "execution_mode": "child_process_cpu_affinity", + "profile_status": "qualified", + "compute_isolation": "disjoint_cpu_affinity", + "cpu_affinity_isolated": True, + "preserves_token_speculation": True, + "tool_cpu_affinity": [14, 15], + "model_cpu_affinity": [0, 1], + } + self.assertIs( + require_qualified_cpu_tool_props( + {"tool_speculation": tool_props}, args, automatic=True + ), + tool_props, + ) + with self.assertRaisesRegex(SystemExit, "profile_status"): + require_qualified_cpu_tool_props( + { + "tool_speculation": { + **tool_props, + "profile_status": "unqualified", + } + }, + args, + automatic=True, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/optimizations/ooo_spec_lucebox5_cpu/test_benchmark_trace_compiled_workflows.py b/optimizations/ooo_spec_lucebox5_cpu/test_benchmark_trace_compiled_workflows.py new file mode 100644 index 000000000..7c7d5955a --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/test_benchmark_trace_compiled_workflows.py @@ -0,0 +1,549 @@ +from __future__ import annotations + +import argparse +import json +import sys +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +from benchmark_trace_compiled_workflows import ( + CANONICAL_TRAINING_REPORT, + CANONICAL_WORKFLOW_REGISTRY, + alphabetic_identifier, + compact_arm, + final_answer_correct, + file_sha256, + interference_probe_qualified, + load_training_traces, + load_partial_pairs, + make_task, + measure_private_miss, + mine_pattern, + model_observation, + parse_request_customers, + post_final, + production_checks, + refresh_existing_report, + simulated_tool_result, + stage_batch_tool, + stage_batched_messages, + stage_result_message, + validate_args, + stage_reference, + workflow_reference, +) + + +def workflow_trace(email: str, destination: str) -> dict: + root = {"customer_email": email, "destination": destination} + calls = [] + results = [] + + def add(name: str, arguments: dict) -> None: + call = {"name": name, "arguments": arguments} + calls.append(call) + results.append(simulated_tool_result(call)) + + add("resolve_customer", {"customer_email": email}) + add("list_open_orders", {"customer_ref": results[-1]["call_ref"]}) + add("get_order_details", {"orders_ref": results[-1]["call_ref"]}) + add( + "calculate_shipping", + {"order_ref": results[-1]["call_ref"], "destination": destination}, + ) + add("prepare_customer_summary", {"shipping_ref": results[-1]["call_ref"]}) + return {"root": root, "calls": calls, "results": results} + + +class TraceCompiledWorkflowBenchmarkTest(unittest.TestCase): + def setUp(self) -> None: + self.pattern = mine_pattern( + [ + workflow_trace("first@example.test", "Rome"), + workflow_trace("second@example.test", "Milan"), + ] + ) + + def test_mines_control_flow_and_late_bound_arguments(self) -> None: + self.assertEqual(self.pattern.training_traces, 2) + self.assertEqual( + [step.tool for step in self.pattern.steps], + [ + "resolve_customer", + "list_open_orders", + "get_order_details", + "calculate_shipping", + "prepare_customer_summary", + ], + ) + shipping_bindings = dict(self.pattern.steps[3].arguments) + self.assertEqual(shipping_bindings["order_ref"].source, "previous_result") + self.assertEqual(shipping_bindings["order_ref"].key, "call_ref") + self.assertEqual(shipping_bindings["destination"].source, "root") + + def test_loads_compact_training_trace_file(self) -> None: + traces = [ + workflow_trace("first@example.test", "Rome"), + workflow_trace("second@example.test", "Milan"), + ] + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "traces.json" + path.write_text(json.dumps({"traces": traces}), encoding="utf-8") + loaded = load_training_traces(path, required_steps=5) + self.assertEqual(loaded, traces) + + def test_compact_arm_keeps_evidence_but_drops_full_graph(self) -> None: + arm = { + "task_ms": 12.0, + "underlying_calls": ["call"], + "tool_results": ["digest"], + "graph": {"large": "payload"}, + "call_turns": [{"large": "payload"}], + "final": { + "content": "done", + "content_sha256": "hash", + "completion_tokens": 1, + "unused": "payload", + }, + } + compact = compact_arm(arm) + self.assertEqual(compact["underlying_calls_count"], 1) + self.assertEqual( + compact["underlying_calls_sha256"], + "4f2a91df1674ac67599f9835f2d43b0ca94e1e769f6a666ce448ae07ac1d94f7", + ) + self.assertEqual(compact["final"]["content_sha256"], "hash") + self.assertNotIn("underlying_calls", compact) + self.assertNotIn("content", compact["final"]) + self.assertNotIn("graph", compact) + self.assertNotIn("call_turns", compact) + self.assertNotIn("unused", compact["final"]) + + def test_pattern_expands_unseen_request_without_literals(self) -> None: + root = {"customer_email": "new@example.test", "destination": "Turin"} + calls = self.pattern.simulate(root) + self.assertEqual(calls[0]["arguments"], {"customer_email": root["customer_email"]}) + self.assertEqual( + calls[1]["arguments"], {"customer_ref": simulated_tool_result(calls[0])["call_ref"]} + ) + self.assertEqual(calls[3]["arguments"]["destination"], "Turin") + + def test_compiler_rejects_side_effecting_trace(self) -> None: + traces = [ + workflow_trace("first@example.test", "Rome"), + workflow_trace("second@example.test", "Milan"), + ] + traces[1]["results"][2]["side_effects"] = True + with self.assertRaisesRegex(ValueError, "side-effect-free"): + mine_pattern(traces) + + def test_compiler_rejects_literal_argument(self) -> None: + traces = [ + workflow_trace("first@example.test", "Rome"), + workflow_trace("second@example.test", "Milan"), + ] + for trace in traces: + trace["calls"][0]["arguments"]["constant"] = "not-in-request" + trace["results"][0] = simulated_tool_result(trace["calls"][0]) + with self.assertRaisesRegex(ValueError, "literal"): + mine_pattern(traces) + + def test_macro_schema_is_closed_and_typed(self) -> None: + tool = self.pattern.macro_tool(4)["function"] + parameters = tool["parameters"] + item = parameters["properties"]["customers"]["items"] + self.assertFalse(parameters["additionalProperties"]) + self.assertFalse(item["additionalProperties"]) + self.assertEqual(set(item["required"]), set(self.pattern.root_fields)) + self.assertEqual(parameters["properties"]["customers"]["maxItems"], 4) + + def test_bound_macro_uses_a_short_single_value_reference(self) -> None: + task = make_task(4, 4, self.pattern) + workflow_ref = workflow_reference(task, self.pattern) + parameters = self.pattern.macro_tool(4, workflow_ref)["function"]["parameters"] + self.assertEqual(set(parameters["properties"]), {"workflow_ref"}) + self.assertEqual( + parameters["properties"]["workflow_ref"]["enum"], [workflow_ref] + ) + self.assertEqual(workflow_ref, "workflow_taske") + + def test_generated_branches_do_not_collapse_on_short_refs(self) -> None: + task = make_task(3, 4, self.pattern) + refs_by_stage = list( + zip( + *[ + [simulated_tool_result(call)["call_ref"] for call in self.pattern.simulate(root)] + for root in task["items"] + ], + strict=True, + ) + ) + self.assertTrue(all(len(set(refs)) == 4 for refs in refs_by_stage)) + + def test_generated_identifiers_avoid_ambiguous_digits(self) -> None: + self.assertEqual(alphabetic_identifier(0), "taska") + self.assertEqual(alphabetic_identifier(26), "taskaa") + task = make_task(50, 2, self.pattern) + self.assertTrue( + all(not any(character.isdigit() for character in item["customer_email"]) + for item in task["items"]) + ) + + def test_event_extractor_recovers_macro_arguments_without_a_model(self) -> None: + content = "Customers: a@example.test to Rome; b@example.test to Milan." + self.assertEqual( + parse_request_customers(content), + [ + {"customer_email": "a@example.test", "destination": "Rome"}, + {"customer_email": "b@example.test", "destination": "Milan"}, + ], + ) + + def test_stage_batch_schema_preserves_every_call(self) -> None: + task = make_task(2, 3, self.pattern) + tool = stage_batch_tool(self.pattern, 3, 4)["function"] + calls = tool["parameters"]["properties"]["calls"] + self.assertEqual(tool["name"], "batch_calculate_shipping") + self.assertEqual(calls["maxItems"], 4) + self.assertEqual( + set(calls["items"]["required"]), {"order_ref", "destination"} + ) + self.assertIn( + "exactly one currently-ready batch tool", + stage_batched_messages(task, self.pattern)[0]["content"], + ) + + def test_bound_stage_uses_the_request_scoped_reference(self) -> None: + task = make_task(2, 4, self.pattern) + stage_ref = stage_reference(task, self.pattern, 2) + parameters = stage_batch_tool( + self.pattern, 2, 4, stage_ref + )["function"]["parameters"] + self.assertEqual(set(parameters["properties"]), {"stage_ref"}) + self.assertEqual(parameters["properties"]["stage_ref"]["enum"], [stage_ref]) + self.assertEqual(stage_ref, "workflow_taskc_stage_three") + + def test_stage_result_exposes_the_requested_final_ref(self) -> None: + message = stage_result_message( + self.pattern, + len(self.pattern.steps) - 1, + [ + { + "root": { + "customer_email": "a@example.test", + "destination": "Rome", + }, + "steps": [{"tool_result": {"call_ref": "plum"}}], + } + ], + "call_stage", + ) + content = json.loads(message["content"]) + self.assertEqual(content["items"][0]["final_ref"], "plum") + self.assertNotIn("call_ref", content["items"][0]) + + def test_validation_requires_two_pairs_for_privacy_probe(self) -> None: + class RaisingParser: + @staticmethod + def error(message: str) -> None: + raise ValueError(message) + + args = argparse.Namespace( + binary=Path(sys.executable), + training_report=CANONICAL_TRAINING_REPORT.resolve(), + workflow_registry=CANONICAL_WORKFLOW_REGISTRY.resolve(), + pairs=1, + warmup_tasks=0, + min_branches=2, + max_branches=4, + timeout=1.0, + call_max_tokens=1, + macro_max_tokens=1, + final_max_tokens=1, + bootstrap_resamples=1, + interference_repetitions=3, + min_production_pairs=2, + min_e2e_speedup=2.0, + min_e2e_speedup_p05=1.5, + min_incremental_speedup=1.05, + ) + with self.assertRaisesRegex(ValueError, "counts and thresholds"): + validate_args(RaisingParser(), args) + + def test_parses_multiple_native_tool_calls(self) -> None: + response = { + "choices": [ + { + "message": { + "role": "assistant", + "content": None, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "resolve_customer", + "arguments": '{"customer_email":"a@example.test"}', + }, + }, + { + "id": "call_2", + "type": "function", + "function": { + "name": "resolve_customer", + "arguments": {"customer_email": "b@example.test"}, + }, + }, + ], + } + } + ], + "usage": {"timings": {}}, + } + observed = model_observation(response, 12.0) + self.assertEqual(len(observed["calls"]), 2) + self.assertEqual(observed["calls"][0]["id"], "call_1") + self.assertEqual( + observed["calls"][1]["call"]["arguments"]["customer_email"], + "b@example.test", + ) + + def test_normalizes_content_format_call_for_conversation_history(self) -> None: + response = { + "choices": [ + { + "message": { + "role": "assistant", + "content": ( + '{"function":"resolve_customer","parameters":' + '{"customer_email":"a@example.test"},"type":"function"}' + ), + } + } + ], + "usage": {"timings": {}}, + } + observed = model_observation(response, 12.0) + self.assertTrue(observed["content_format_call"]) + self.assertEqual( + observed["calls"][0]["call"], + { + "name": "resolve_customer", + "arguments": {"customer_email": "a@example.test"}, + }, + ) + self.assertEqual( + observed["assistant_message"]["tool_calls"][0]["id"], + observed["calls"][0]["id"], + ) + + def test_production_gate_requires_two_x_and_exact_outputs(self) -> None: + summary = { + "tasks": 6, + "stage_batched_to_speculative_speedup_p50": 1.99, + "stage_batched_to_speculative_bootstrap_95ci": [1.5, 2.4], + "stage_batched_to_speculative_speedup_p05": 1.6, + "compiled_to_speculative_speedup_p50": 1.1, + "compiled_to_speculative_bootstrap_95ci": [1.01, 1.2], + "pattern_prediction_hit_rate": 1.0, + "all_predictions_from_qwen": True, + "private_miss_result_hidden": True, + "all_interference_probes_qualified": True, + "model_compute_slowdown_p50_percent": 0.0, + "model_compute_slowdown_p95_percent": 0.0, + "decode_slowdown_p50_percent": 0.0, + "decode_slowdown_p95_percent": 0.0, + "continuation_cache_hit_rate": 1.0, + "prefix_cache_configured": True, + "all_calls_stable": True, + "all_tool_results_stable": True, + "macro_output_stability_rate": 1.0, + "all_final_answers_correct": True, + "all_final_outputs_stable": True, + "all_macro_calls_correct": True, + "all_ds4_active": True, + } + args = argparse.Namespace( + min_e2e_speedup=2.0, + min_e2e_speedup_p05=1.5, + min_incremental_speedup=1.05, + min_production_pairs=6, + max_model_slowdown_percent=1.0, + max_model_slowdown_p95_percent=5.0, + max_decode_slowdown_percent=1.0, + max_decode_slowdown_p95_percent=5.0, + ) + checks = production_checks(summary, args) + self.assertFalse(checks["end_to_end_speedup"]) + self.assertTrue( + all(value for key, value in checks.items() if key != "end_to_end_speedup") + ) + + def test_interference_probe_accepts_matched_cache_misses(self) -> None: + observation = { + "cache_hit": False, + "cached_prefix_tokens": 0, + "completion_tokens": 35, + "call_sha256": "same-call", + "accept_rate": 0.9, + } + probe = { + "compiled": dict(observation), + "speculative": dict(observation), + } + self.assertTrue(interference_probe_qualified(probe)) + probe["speculative"]["cached_prefix_tokens"] = 128 + self.assertFalse(interference_probe_qualified(probe)) + + def test_private_miss_requires_no_result_in_engine_metadata(self) -> None: + target = make_task(0, 2, self.pattern) + predicted = make_task(1, 3, self.pattern) + target_call = { + "name": self.pattern.macro_name, + "arguments": {"workflow_ref": workflow_reference(target, self.pattern)}, + } + predicted_call = { + "name": self.pattern.macro_name, + "arguments": { + "workflow_ref": workflow_reference(predicted, self.pattern) + }, + } + response = { + "calls": [{"call": target_call}], + "speculation": { + "status": "miss", + "reason": "invocation_mismatch", + "prediction": predicted_call, + }, + } + args = argparse.Namespace(max_branches=4, macro_max_tokens=128) + with patch( + "benchmark_trace_compiled_workflows.post_turn", + return_value=response, + ) as mocked: + result = measure_private_miss( + args, target, predicted, self.pattern + ) + self.assertTrue(result["passed"]) + self.assertFalse(result["private_result_exposed"]) + self.assertEqual( + mocked.call_args.kwargs["tool_speculation"]["call"], predicted_call + ) + + def test_final_turn_consumes_the_real_tool_conversation(self) -> None: + args = argparse.Namespace(final_max_tokens=32) + messages = [ + {"role": "user", "content": "run it"}, + {"role": "assistant", "content": "", "tool_calls": []}, + { + "role": "tool", + "tool_call_id": "call_1", + "content": '{"items":[{"final_ref":"plum"}]}', + }, + ] + with patch( + "benchmark_trace_compiled_workflows.post_turn", + return_value={"content": "workflow_complete:plum"}, + ) as mocked: + post_final(args, messages) + + call_args = mocked.call_args.args + self.assertEqual(call_args[2], []) + self.assertEqual(call_args[3], "none") + self.assertEqual(call_args[4], 32) + self.assertEqual(call_args[1][:-1], messages) + self.assertNotIn("plum", call_args[1][-1]["content"]) + + def test_final_receipt_accepts_literal_or_equivalent_json(self) -> None: + expected = "workflow_complete:plum,ivory" + self.assertTrue(final_answer_correct(expected, expected)) + self.assertTrue( + final_answer_correct('{"workflow_complete": "plum,ivory"}', expected) + ) + self.assertTrue(final_answer_correct("plum,ivory", expected)) + self.assertFalse(final_answer_correct('{"workflow_complete": "plum"}', expected)) + self.assertFalse(final_answer_correct("workflow_complete:ivory,plum", expected)) + + def test_resume_checkpoint_requires_matching_task_and_arm_order(self) -> None: + tasks = [make_task(0, 2, self.pattern), make_task(1, 3, self.pattern)] + orders = [ + ["compiled", "stage_batched", "speculative"], + ["speculative", "stage_batched", "compiled"], + ] + checkpoint = { + "schema_version": 1, + "complete": False, + "pairs": [ + { + "pair_index": 0, + "task": tasks[0], + "arm_order": orders[0], + "stage_batched": { + "final": {"content": "plum,ivory"}, + "expected_final": "workflow_complete:plum,ivory", + }, + "compiled": { + "final": {"content": "plum,ivory"}, + "expected_final": "workflow_complete:plum,ivory", + }, + "speculative": { + "final": {"content": "plum,ivory"}, + "expected_final": "workflow_complete:plum,ivory", + }, + } + ], + } + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "run.json.partial" + path.write_text(json.dumps(checkpoint), encoding="utf-8") + resumed = load_partial_pairs(path, tasks, orders) + self.assertEqual(len(resumed), 1) + self.assertTrue( + all( + resumed[0][arm]["final_correct"] + for arm in ("stage_batched", "compiled", "speculative") + ) + ) + checkpoint["pairs"][0]["arm_order"] = orders[1] + path.write_text(json.dumps(checkpoint), encoding="utf-8") + with self.assertRaisesRegex(ValueError, "does not match"): + load_partial_pairs(path, tasks, orders) + + def test_refresh_rejects_reports_that_predate_interference_probes(self) -> None: + tasks = [make_task(0, 2, self.pattern), make_task(1, 3, self.pattern)] + report = { + "schema_version": 1, + "pattern": { + "fingerprint": self.pattern.fingerprint, + "training_report_sha256": file_sha256( + CANONICAL_TRAINING_REPORT + ), + "workflow_registry_sha256": file_sha256( + CANONICAL_WORKFLOW_REGISTRY + ), + }, + "production_gate": {"passed": True}, + "pairs": [{}, {}], + "summary": {"tasks": 2}, + "methodology": {}, + "server_snapshot": {}, + } + with tempfile.TemporaryDirectory() as directory: + output = Path(directory) / "report.json" + output.write_text(json.dumps(report), encoding="utf-8") + args = argparse.Namespace( + output=output, + pairs=2, + training_report=CANONICAL_TRAINING_REPORT, + workflow_registry=CANONICAL_WORKFLOW_REGISTRY, + ) + with self.assertRaisesRegex(ValueError, "predates interference"): + refresh_existing_report( + args, self.pattern, tasks, {}, {} + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/optimizations/ooo_spec_lucebox5_cpu/test_trace_compiled_tool_executor.py b/optimizations/ooo_spec_lucebox5_cpu/test_trace_compiled_tool_executor.py new file mode 100644 index 000000000..82f12b9e1 --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/test_trace_compiled_tool_executor.py @@ -0,0 +1,101 @@ +from __future__ import annotations + +import unittest + +from benchmark_trace_compiled_workflows import mine_pattern, simulated_tool_result +from bfcl_replay_tool_executor import execute as execute_bfcl +from test_benchmark_trace_compiled_workflows import workflow_trace +from trace_compiled_tool_executor import execute_macro, resolve_items + + +class TraceCompiledToolExecutorTest(unittest.TestCase): + def setUp(self) -> None: + self.pattern = mine_pattern( + [ + workflow_trace("first@example.test", "Rome"), + workflow_trace("second@example.test", "Milan"), + ] + ) + self.items = [ + {"customer_email": "alice@example.test", "destination": "Turin"}, + {"customer_email": "bob@example.test", "destination": "Naples"}, + ] + self.workflow_ref = "workflow_alpha" + self.registry = { + "schema_version": 1, + "pattern_fingerprint": self.pattern.fingerprint, + "workflows": { + self.workflow_ref: { + "pattern_fingerprint": self.pattern.fingerprint, + "items": self.items, + } + }, + } + + @staticmethod + def fake_leaf(request: dict) -> dict: + return {"ok": True, "result": simulated_tool_result(request["call"])} + + def test_executes_every_branch_and_preserves_order(self) -> None: + call = { + "name": self.pattern.macro_name, + "arguments": {"workflow_ref": self.workflow_ref}, + } + request = {"call": call} + envelope = execute_macro( + request, self.pattern, self.fake_leaf, self.registry + ) + result = envelope["result"] + self.assertEqual(result["call_count"], 10) + self.assertEqual([branch["root"] for branch in result["branches"]], self.items) + self.assertEqual( + [len(branch["steps"]) for branch in result["branches"]], [5, 5] + ) + self.assertFalse(result["side_effects"]) + + def test_rejects_unknown_or_missing_inputs(self) -> None: + with self.assertRaisesRegex(ValueError, "fields"): + resolve_items( + {"workflow_ref": self.workflow_ref}, + self.pattern, + { + **self.registry, + "workflows": { + self.workflow_ref: { + "pattern_fingerprint": self.pattern.fingerprint, + "items": [{**self.items[0], "undeclared": "value"}], + } + }, + }, + ) + with self.assertRaisesRegex(ValueError, "workflow_ref"): + resolve_items({"items": self.items}, self.pattern, self.registry) + + def test_rejects_non_object_leaf_response(self) -> None: + request = { + "call": { + "name": self.pattern.macro_name, + "arguments": {"workflow_ref": self.workflow_ref}, + } + } + with self.assertRaisesRegex(RuntimeError, "invalid result"): + execute_macro( + request, + self.pattern, + lambda _request: None, # type: ignore[arg-type,return-value] + self.registry, + ) + + def test_leaf_rejects_explicit_null_affinity(self) -> None: + with self.assertRaisesRegex(ValueError, "cpu_affinity"): + execute_bfcl( + { + "protocol": "dflash.tool-speculation.v1", + "cpu_affinity": None, + "call": {"name": "lookup", "arguments": {}}, + } + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/optimizations/ooo_spec_lucebox5_cpu/trace_compiled_tool_executor.py b/optimizations/ooo_spec_lucebox5_cpu/trace_compiled_tool_executor.py new file mode 100755 index 000000000..15736940f --- /dev/null +++ b/optimizations/ooo_spec_lucebox5_cpu/trace_compiled_tool_executor.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +"""Execute a trace-compiled read-only workflow through the tool-spec protocol. + +The engine treats the compiled workflow like any other predicted tool: Qwen +proposes its typed arguments, DS4 remains authoritative, and the private result +is released only after an exact call match. Independent workflow branches run +concurrently inside the CPU-pinned executor process. +""" + +from __future__ import annotations + +import json +import os +import re +import sys +import time +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from typing import Any, Callable + +from benchmark_trace_compiled_workflows import CompiledPattern, load_training_traces, mine_pattern +from bfcl_replay_tool_executor import ( + PROTOCOL, + call_ref, + call_sha256, + execute as execute_leaf, +) + + +MAX_BRANCHES = 4 +TRAINING_REPORT_ENV = "DFLASH_TRACE_TRAINING_REPORT" +WORKFLOW_REGISTRY_ENV = "DFLASH_TRACE_WORKFLOW_REGISTRY" +LeafExecutor = Callable[[dict[str, Any]], dict[str, Any]] + + +def load_pattern() -> CompiledPattern: + default = ( + Path(__file__).with_name("results") + / "multiturn-cached-wordref-production-6tasks.json" + ) + report = Path(os.environ.get(TRAINING_REPORT_ENV, str(default))) + return mine_pattern(load_training_traces(report, required_steps=5)) + + +def load_registry() -> dict[str, Any]: + default = Path(__file__).with_name("results") / "trace-workflow-registry.json" + path = Path(os.environ.get(WORKFLOW_REGISTRY_ENV, str(default))) + registry = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(registry, dict) or registry.get("schema_version") != 1: + raise ValueError("compiled workflow registry is invalid") + return registry + + +def validate_items(items: Any, pattern: CompiledPattern) -> list[dict[str, str]]: + if not isinstance(items, list) or not 1 <= len(items) <= MAX_BRANCHES: + raise ValueError(f"customers must contain between 1 and {MAX_BRANCHES} items") + expected_fields = set(pattern.root_fields) + validated = [] + for item in items: + if not isinstance(item, dict) or set(item) != expected_fields: + raise ValueError("customer fields do not match the compiled workflow") + if not all(isinstance(value, str) and value for value in item.values()): + raise ValueError("compiled workflow inputs must be non-empty strings") + validated.append(dict(item)) + return validated + + +def resolve_items( + arguments: Any, pattern: CompiledPattern, registry: dict[str, Any] +) -> list[dict[str, str]]: + if not isinstance(arguments, dict) or set(arguments) != {"workflow_ref"}: + raise ValueError("compiled workflow requires only a workflow_ref") + workflow_ref = arguments["workflow_ref"] + if not isinstance(workflow_ref, str) or re.fullmatch( + r"workflow_[a-z]+", workflow_ref + ) is None: + raise ValueError("workflow_ref is malformed") + if registry.get("pattern_fingerprint") != pattern.fingerprint: + raise ValueError("workflow registry pattern does not match the executor") + workflows = registry.get("workflows") + entry = workflows.get(workflow_ref) if isinstance(workflows, dict) else None + if ( + not isinstance(entry, dict) + or entry.get("pattern_fingerprint") != pattern.fingerprint + ): + raise ValueError("workflow_ref is unknown or bound to another pattern") + return validate_items(entry.get("items"), pattern) + + +def execute_branch( + request: dict[str, Any], + pattern: CompiledPattern, + root: dict[str, str], + leaf_executor: LeafExecutor, +) -> dict[str, Any]: + previous: dict[str, Any] | None = None + steps = [] + for index in range(len(pattern.steps)): + call = pattern.instantiate(root, previous, index) + leaf_request = {**request, "call": call} + envelope = leaf_executor(leaf_request) + result = envelope.get("result") if isinstance(envelope, dict) else None + if ( + not isinstance(envelope, dict) + or not envelope.get("ok") + or not isinstance(result, dict) + ): + raise RuntimeError("leaf tool returned an invalid result") + if ( + result.get("call_sha256") != call_sha256(call) + or result.get("call_ref") != call_ref(call) + or result.get("tool_name") != call["name"] + or result.get("side_effects") is not False + ): + raise RuntimeError("leaf tool result did not match its compiled call") + previous = result + steps.append({"call": call, "tool_result": result}) + return {"root": root, "steps": steps, "final_ref": steps[-1]["tool_result"]["call_ref"]} + + +def execute_macro( + request: dict[str, Any], + pattern: CompiledPattern, + leaf_executor: LeafExecutor = execute_leaf, + registry: dict[str, Any] | None = None, +) -> dict[str, Any]: + call = request.get("call") + if not isinstance(call, dict) or call.get("name") != pattern.macro_name: + raise ValueError("request is not for the compiled workflow") + items = resolve_items( + call.get("arguments"), pattern, registry if registry is not None else load_registry() + ) + started = time.perf_counter() + with ThreadPoolExecutor(max_workers=len(items), thread_name_prefix="compiled-workflow") as pool: + futures = [ + pool.submit(execute_branch, request, pattern, root, leaf_executor) + for root in items + ] + branches = [future.result() for future in futures] + elapsed_ms = (time.perf_counter() - started) * 1_000.0 + affinity = sorted(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else [] + return { + "ok": True, + "result": { + "call_sha256": call_sha256(call), + "call_ref": call_ref(call), + "tool_name": pattern.macro_name, + "workflow_fingerprint": pattern.fingerprint, + "branches": branches, + "call_count": len(items) * len(pattern.steps), + "elapsed_ms": elapsed_ms, + "cpu_affinity": affinity, + "side_effects": False, + }, + } + + +def execute(request: dict[str, Any], pattern: CompiledPattern) -> dict[str, Any]: + call = request.get("call") + if isinstance(call, dict) and call.get("name") == pattern.macro_name: + return execute_macro(request, pattern) + return execute_leaf(request) + + +def main() -> int: + if sys.argv[1:] != ["--dflash-tool-spec-v1"]: + print("expected --dflash-tool-spec-v1", file=sys.stderr) + return 2 + try: + line = sys.stdin.readline() + if not line: + raise ValueError("missing request") + request = json.loads(line) + if not isinstance(request, dict) or request.get("protocol") != PROTOCOL: + raise ValueError("unsupported tool-speculation request") + print(json.dumps(execute(request, load_pattern()), separators=(",", ":")), flush=True) + return 0 + except (OSError, RuntimeError, ValueError, json.JSONDecodeError) as error: + print(str(error), file=sys.stderr) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/server/CMakeLists.txt b/server/CMakeLists.txt index 46ecfda81..d8bd8170b 100644 --- a/server/CMakeLists.txt +++ b/server/CMakeLists.txt @@ -446,6 +446,7 @@ add_library(dflash_common STATIC src/common/dflash_draft_ipc.cpp src/common/dflash_draft_ipc_daemon.cpp src/common/pflash_drafter_ipc.cpp + src/common/qwen3_tool_predictor_ipc.cpp src/common/dflash_draft_graph.cpp src/common/dflash_draft_kv.cpp src/common/dflash_spec_decode.cpp @@ -497,6 +498,8 @@ add_library(dflash_common STATIC src/server/chat_template.cpp src/server/tool_parser.cpp src/server/tool_hint.cpp + src/server/semantic_tool_hint.cpp + src/server/native_semantic_tool_predictor.cpp src/server/reasoning.cpp src/server/tool_memory.cpp src/server/sse_emitter.cpp @@ -1308,6 +1311,14 @@ if(DFLASH27B_TESTS) target_include_directories(smoke_qwen3_forward PRIVATE ${DFLASH27B_SRC_INCLUDE_DIRS}) target_link_libraries(smoke_qwen3_forward PRIVATE dflash_common ggml ${DFLASH27B_GGML_BACKEND_TARGET}) endif() + if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/test/smoke_qwen3_tool_predictor_ipc.cpp") + add_executable(smoke_qwen3_tool_predictor_ipc + test/smoke_qwen3_tool_predictor_ipc.cpp) + target_include_directories(smoke_qwen3_tool_predictor_ipc PRIVATE + ${DFLASH27B_SRC_INCLUDE_DIRS}) + target_link_libraries(smoke_qwen3_tool_predictor_ipc PRIVATE + dflash_common ggml ${DFLASH27B_GGML_BACKEND_TARGET}) + endif() if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/test/test_vs_oracle.cpp") add_executable(test_vs_oracle test/test_vs_oracle.cpp) target_include_directories(test_vs_oracle PRIVATE ${DFLASH27B_SRC_INCLUDE_DIRS}) @@ -1522,6 +1533,8 @@ if(DFLASH27B_TESTS) set(_server_unit_sources test/test_unit_main.cpp test/test_server_unit.cpp + test/test_tool_speculation.cpp + test/test_semantic_tool_hint.cpp test/test_anchor_params.cpp test/test_derived_scalars.cpp test/test_adaptive_keep_ratio.cpp @@ -1554,6 +1567,7 @@ if(DFLASH27B_TESTS) add_executable(test_server_unit ${_server_unit_sources}) target_sources(test_server_unit PRIVATE src/server/http_server.cpp + src/server/tool_speculation.cpp src/server/scheduler.cpp src/server/model_card.cpp src/server/prompt_normalize.cpp @@ -1925,6 +1939,7 @@ if(DFLASH27B_SERVER) add_executable(dflash_server src/server/server_main.cpp src/server/http_server.cpp + src/server/tool_speculation.cpp src/server/scheduler.cpp src/server/model_card.cpp src/server/prompt_normalize.cpp @@ -1980,6 +1995,7 @@ if(DFLASH27B_SERVER) add_executable(backend_ipc_daemon src/ipc/backend_ipc_main.cpp src/common/pflash_drafter_ipc_daemon.cpp + src/common/qwen3_tool_predictor_ipc_daemon.cpp ) target_include_directories(backend_ipc_daemon PRIVATE ${DFLASH27B_SRC_INCLUDE_DIRS}) if(DFLASH27B_GPU_BACKEND STREQUAL "hip") diff --git a/server/src/common/backend_ipc.cpp b/server/src/common/backend_ipc.cpp index 2a98b8e9c..3551f184d 100644 --- a/server/src/common/backend_ipc.cpp +++ b/server/src/common/backend_ipc.cpp @@ -3,16 +3,24 @@ #include "backend_ipc.h" #include "io_utils.h" +#include +#include #include #include #include #include +#include +#include +#include +#include #include +#include #include #if !defined(_WIN32) # include # include +# include # include # include # include @@ -24,11 +32,154 @@ namespace dflash::common { +namespace { + +#if !defined(_WIN32) +bool close_descriptor_range(unsigned int first, unsigned int last) { + if (first > last) return true; +#if defined(__linux__) && defined(SYS_close_range) + int rc = -1; + do { + rc = static_cast(::syscall(SYS_close_range, first, last, 0)); + } while (rc != 0 && errno == EINTR); + if (rc == 0) return true; +#endif + return false; +} + +bool descriptor_is_kept(const std::array & keep, int fd) { + return fd == keep[0] || fd == keep[1] || fd == keep[2]; +} + +// close_range can be unavailable on an older kernel or denied by seccomp. +// Enumerate the descriptors that actually exist instead of trusting the +// current RLIMIT_NOFILE: a process may have opened a high descriptor before +// lowering its soft limit. Raw getdents64 keeps this post-fork path free of +// libc directory-stream allocation and turns an unavailable /proc into a +// fail-closed launch. +bool close_unlisted_descriptors(const std::array & keep) { +#if defined(__linux__) && defined(SYS_getdents64) + struct LinuxDirent64 { + uint64_t inode; + int64_t offset; + unsigned short record_bytes; + unsigned char type; + char name[1]; + }; + + const int directory_fd = ::open( + "/proc/self/fd", O_RDONLY | O_DIRECTORY | O_CLOEXEC | O_NOFOLLOW); + if (directory_fd < 0) return false; + + alignas(long) char entries[4096]; + bool ok = true; + int failure = 0; + while (ok) { + long bytes = -1; + do { + bytes = ::syscall( + SYS_getdents64, directory_fd, entries, sizeof(entries)); + } while (bytes < 0 && errno == EINTR); + if (bytes == 0) break; + if (bytes < 0) { + ok = false; + failure = errno; + break; + } + + size_t cursor = 0; + while (cursor < static_cast(bytes)) { + auto * entry = reinterpret_cast(entries + cursor); + constexpr size_t name_offset = offsetof(LinuxDirent64, name); + if (static_cast(bytes) - cursor <= name_offset || + entry->record_bytes <= name_offset || + entry->record_bytes > static_cast(bytes) - cursor) { + ok = false; + failure = EIO; + break; + } + const size_t name_bytes = entry->record_bytes - name_offset; + const char * terminator = static_cast( + std::memchr(entry->name, '\0', name_bytes)); + if (!terminator) { + ok = false; + failure = EIO; + break; + } + + uint64_t parsed = 0; + bool numeric = terminator != entry->name; + for (const char * digit = entry->name; digit < terminator; ++digit) { + if (*digit < '0' || *digit > '9') { + numeric = false; + break; + } + parsed = parsed * 10U + static_cast(*digit - '0'); + if (parsed > static_cast(INT_MAX)) { + numeric = false; + break; + } + } + if (numeric) { + const int fd = static_cast(parsed); + if (fd > STDERR_FILENO && fd != directory_fd && + !descriptor_is_kept(keep, fd) && + ::close(fd) != 0 && errno != EBADF) { + const int close_error = errno; + errno = 0; + if (!(::fcntl(fd, F_GETFD) < 0 && errno == EBADF)) { + ok = false; + failure = close_error; + break; + } + } + } + cursor += entry->record_bytes; + } + } + + if (::close(directory_fd) != 0 && ok) { + ok = false; + failure = errno; + } + if (!ok) errno = failure == 0 ? EIO : failure; + return ok; +#else + (void)keep; + errno = ENOTSUP; + return false; +#endif +} + +bool isolate_child_descriptors( + int payload_fd, int stream_fd, int shared_fd) { + std::array keep{payload_fd, stream_fd, shared_fd}; + std::sort(keep.begin(), keep.end()); + unsigned int first = STDERR_FILENO + 1; + int previous = -1; + bool ranges_closed = true; + for (const int fd : keep) { + if (fd < static_cast(first) || fd == previous) continue; + if (!close_descriptor_range(first, static_cast(fd - 1))) { + ranges_closed = false; + break; + } + first = static_cast(fd) + 1U; + previous = fd; + } + if (ranges_closed && close_descriptor_range(first, UINT_MAX)) return true; + return close_unlisted_descriptors(keep); +} +#endif + +} // namespace + const char * backend_ipc_mode_name(BackendIpcMode mode) { switch (mode) { case BackendIpcMode::Invalid: return "invalid"; case BackendIpcMode::DFlashDraft: return "dflash-draft"; case BackendIpcMode::PFlashCompress: return "pflash-compress"; + case BackendIpcMode::Qwen3ToolPredict: return "qwen3-tool-predict"; case BackendIpcMode::Qwen35TargetShard: return "qwen35-target-shard"; case BackendIpcMode::Gemma4TargetShard: return "gemma4-target-shard"; case BackendIpcMode::LagunaTargetShard: return "laguna-target-shard"; @@ -47,6 +198,10 @@ bool parse_backend_ipc_mode(const std::string & value, BackendIpcMode & out) { out = BackendIpcMode::PFlashCompress; return true; } + if (value == "qwen3-tool-predict") { + out = BackendIpcMode::Qwen3ToolPredict; + return true; + } if (value == "qwen35-target-shard") { out = BackendIpcMode::Qwen35TargetShard; return true; @@ -104,8 +259,7 @@ bool BackendIpcProcess::start(const BackendIpcLaunchConfig & cfg) { #else close(); if (cfg.bin.empty() || cfg.payload_path.empty()) return false; - if (!init_work_dir(cfg.work_dir)) return false; - + if (!init_work_dir(cfg.work_dir, cfg.require_private_work_dir)) return false; int cmd_pipe[2] = {-1, -1}; int payload_pipe[2] = {-1, -1}; int stream_pipe[2] = {-1, -1}; @@ -117,6 +271,7 @@ bool BackendIpcProcess::start(const BackendIpcLaunchConfig & cfg) { if (payload_pipe[1] >= 0) ::close(payload_pipe[1]); if (stream_pipe[0] >= 0) ::close(stream_pipe[0]); if (stream_pipe[1] >= 0) ::close(stream_pipe[1]); + close(); return false; } const bool shared_required = @@ -129,6 +284,7 @@ bool BackendIpcProcess::start(const BackendIpcLaunchConfig & cfg) { ::close(cmd_pipe[0]); ::close(cmd_pipe[1]); ::close(payload_pipe[0]); ::close(payload_pipe[1]); ::close(stream_pipe[0]); ::close(stream_pipe[1]); + close(); return false; } if (shared_requested && cfg.shared_payload_bytes > 0) { @@ -159,6 +315,13 @@ bool BackendIpcProcess::start(const BackendIpcLaunchConfig & cfg) { return false; } if (pid_ == 0) { + if (cfg.require_private_work_dir && + (work_dir_fd_ < 0 || ::fchdir(work_dir_fd_) != 0)) { + std::fprintf(stderr, + "backend-ipc private work_dir chdir failed: %s\n", + std::strerror(errno)); + _exit(127); + } if (cmd_pipe[0] != STDIN_FILENO && ::dup2(cmd_pipe[0], STDIN_FILENO) < 0) { std::fprintf(stderr, "backend-ipc dup2 failed: %s\n", std::strerror(errno)); _exit(127); @@ -190,6 +353,14 @@ bool BackendIpcProcess::start(const BackendIpcLaunchConfig & cfg) { argv.reserve(argv_storage.size() + 1); for (std::string & arg : argv_storage) argv.push_back(arg.data()); argv.push_back(nullptr); + if (cfg.isolate_inherited_fds && + !isolate_child_descriptors( + payload_pipe[0], stream_pipe[1], shared_payload_fd_)) { + std::fprintf(stderr, + "backend-ipc descriptor isolation failed: %s\n", + std::strerror(errno)); + _exit(127); + } ::execv(exec_bin.c_str(), argv.data()); std::fprintf(stderr, "backend-ipc exec failed: %s: %s\n", exec_bin.c_str(), std::strerror(errno)); @@ -209,7 +380,14 @@ bool BackendIpcProcess::start(const BackendIpcLaunchConfig & cfg) { return false; } int32_t status = -1; - if (!read_exact_fd(stream_fd_, &status, sizeof(status)) || status != 0) { + bool readiness_timed_out = false; + const auto readiness_deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds((std::max)(0, cfg.readiness_timeout_ms)); + const bool status_read = cfg.readiness_timeout_ms > 0 + ? read_exact_fd_until(stream_fd_, &status, sizeof(status), + readiness_deadline, readiness_timed_out) + : read_exact_fd(stream_fd_, &status, sizeof(status)); + if (!status_read || status != 0) { int child_status = 0; const pid_t exited = ::waitpid(pid_, &child_status, WNOHANG); if (exited == pid_) { @@ -228,9 +406,17 @@ bool BackendIpcProcess::start(const BackendIpcLaunchConfig & cfg) { } pid_ = -1; } else { - std::fprintf(stderr, "backend-ipc daemon did not become ready (status=%d)\n", status); + std::fprintf(stderr, + readiness_timed_out + ? "backend-ipc daemon readiness timed out after %d ms\n" + : "backend-ipc daemon did not become ready (status=%d)\n", + readiness_timed_out ? cfg.readiness_timeout_ms : status); + } + if (pid_ > 0 && cfg.readiness_timeout_ms > 0) { + terminate(); + } else { + close(); } - close(); return false; } active_ = true; @@ -243,7 +429,19 @@ bool BackendIpcProcess::start(const BackendIpcLaunchConfig & cfg) { } void BackendIpcProcess::close() { + close_impl(false); +} + +void BackendIpcProcess::terminate() { + close_impl(true); +} + +void BackendIpcProcess::close_impl(bool force_terminate) { #if !defined(_WIN32) + const pid_t child = pid_; + if (force_terminate && child > 0) { + (void)::kill(child, SIGTERM); + } if (cmd_) { std::fclose(cmd_); cmd_ = nullptr; @@ -264,14 +462,47 @@ void BackendIpcProcess::close() { ::close(shared_payload_fd_); shared_payload_fd_ = -1; } - if (pid_ > 0) { + if (child > 0) { int status = 0; - ::waitpid(pid_, &status, 0); + if (force_terminate) { + bool reaped = false; + const auto deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(100); + while (std::chrono::steady_clock::now() < deadline) { + const pid_t waited = ::waitpid(child, &status, WNOHANG); + if (waited == child || + (waited < 0 && errno == ECHILD)) { + reaped = true; + break; + } + if (waited < 0 && errno != EINTR) break; + std::this_thread::sleep_for(std::chrono::milliseconds(2)); + } + if (!reaped) { + (void)::kill(child, SIGKILL); + while (::waitpid(child, &status, 0) < 0 && errno == EINTR) {} + } + } else { + while (::waitpid(child, &status, 0) < 0 && errno == EINTR) {} + } pid_ = -1; } - if (owns_work_dir_ && !work_dir_.empty()) { - ::rmdir(work_dir_.c_str()); + bool remove_owned_work_dir = owns_work_dir_ && !work_dir_.empty(); + if (work_dir_fd_ >= 0) { + struct stat opened {}; + struct stat named {}; + remove_owned_work_dir = remove_owned_work_dir && + ::fstat(work_dir_fd_, &opened) == 0 && + ::lstat(work_dir_.c_str(), &named) == 0 && + opened.st_dev == named.st_dev && opened.st_ino == named.st_ino; + ::close(work_dir_fd_); + work_dir_fd_ = -1; + } + if (remove_owned_work_dir) { + (void)::rmdir(work_dir_.c_str()); } +#else + (void)force_terminate; #endif active_ = false; owns_work_dir_ = false; @@ -287,6 +518,62 @@ std::string BackendIpcProcess::next_path(const char * prefix) { return work_dir_ + "/" + prefix + "_" + std::to_string(seq_++) + ".bin"; } +bool BackendIpcProcess::write_private_file( + const char * prefix, + const void * data, + size_t bytes, + std::string & name) { + name.clear(); +#if defined(_WIN32) + (void)prefix; (void)data; (void)bytes; + return false; +#else + if (work_dir_fd_ < 0 || !prefix || !*prefix || + (bytes > 0 && !data)) { + return false; + } + for (const char * cursor = prefix; *cursor; ++cursor) { + const unsigned char character = static_cast(*cursor); + if (!std::isalnum(character) && character != '_' && character != '-') { + return false; + } + } + for (int attempt = 0; attempt < 16; ++attempt) { + name = std::string(prefix) + "_" + std::to_string(::getpid()) + "_" + + std::to_string(seq_++) + ".bin"; + const int fd = ::openat( + work_dir_fd_, name.c_str(), + O_WRONLY | O_CREAT | O_EXCL | O_NOFOLLOW | O_CLOEXEC, 0600); + if (fd < 0) { + if (errno == EEXIST) continue; + name.clear(); + return false; + } + const bool written = write_exact_fd(fd, data, bytes); + const bool closed = ::close(fd) == 0; + if (written && closed) return true; + (void)::unlinkat(work_dir_fd_, name.c_str(), 0); + name.clear(); + return false; + } + name.clear(); + return false; +#endif +} + +bool BackendIpcProcess::remove_private_file(const std::string & name) { +#if defined(_WIN32) + (void)name; + return false; +#else + if (work_dir_fd_ < 0 || name.empty() || + name.find('/') != std::string::npos) { + return false; + } + return ::unlinkat(work_dir_fd_, name.c_str(), 0) == 0 || errno == ENOENT; +#endif +} + bool BackendIpcProcess::write_shared_payload(const void * data, size_t bytes, uint64_t & seq) { BackendIpcPayloadSegment segment{data, bytes}; return write_shared_payload_segments(&segment, 1, seq); @@ -388,7 +675,63 @@ bool BackendIpcProcess::init_shared_payload(size_t bytes) { return true; } -bool BackendIpcProcess::init_work_dir(const std::string & requested) { +bool BackendIpcProcess::init_work_dir(const std::string & requested, + bool require_private) { + if (require_private) { + std::string parent = "/tmp"; + if (!requested.empty()) { + parent = requested; + if (::mkdir(parent.c_str(), 0700) != 0 && errno != EEXIST) { + std::fprintf(stderr, "backend-ipc mkdir failed: %s: %s\n", + parent.c_str(), std::strerror(errno)); + return false; + } + struct stat parent_stat {}; + if (::lstat(parent.c_str(), &parent_stat) != 0 || + !S_ISDIR(parent_stat.st_mode) || + parent_stat.st_uid != ::geteuid() || + (parent_stat.st_mode & 0777) != 0700) { + std::fprintf(stderr, + "backend-ipc private work base must be an owned, " + "non-symlink mode-0700 directory: %s\n", + parent.c_str()); + return false; + } + } + + std::string templ = parent + "/backend-ipc-private-XXXXXX"; + std::vector buf(templ.begin(), templ.end()); + buf.push_back('\0'); + char * dir = ::mkdtemp(buf.data()); + if (!dir) { + std::fprintf(stderr, + "backend-ipc private mkdtemp failed: %s\n", + std::strerror(errno)); + return false; + } + work_dir_ = dir; + work_dir_fd_ = ::open( + work_dir_.c_str(), + O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC); + struct stat opened {}; + if (work_dir_fd_ < 0 || ::fstat(work_dir_fd_, &opened) != 0 || + !S_ISDIR(opened.st_mode) || opened.st_uid != ::geteuid() || + (opened.st_mode & 0777) != 0700) { + std::fprintf(stderr, + "backend-ipc cannot retain private work directory: %s\n", + std::strerror(errno)); + if (work_dir_fd_ >= 0) { + ::close(work_dir_fd_); + work_dir_fd_ = -1; + } + (void)::rmdir(work_dir_.c_str()); + work_dir_.clear(); + return false; + } + owns_work_dir_ = true; + return true; + } + if (!requested.empty()) { work_dir_ = requested; owns_work_dir_ = false; @@ -398,12 +741,13 @@ bool BackendIpcProcess::init_work_dir(const std::string & requested) { work_dir_.c_str(), std::strerror(errno)); return false; } - struct stat st; - if (::stat(work_dir_.c_str(), &st) != 0 || !S_ISDIR(st.st_mode)) { - std::fprintf(stderr, "backend-ipc work_dir is not a directory: %s\n", - work_dir_.c_str()); - return false; - } + } + struct stat st; + if (::stat(work_dir_.c_str(), &st) != 0 || !S_ISDIR(st.st_mode)) { + std::fprintf(stderr, + "backend-ipc work_dir is not a directory: %s\n", + work_dir_.c_str()); + return false; } return true; } diff --git a/server/src/common/backend_ipc.h b/server/src/common/backend_ipc.h index d995759ac..50ddfdc78 100644 --- a/server/src/common/backend_ipc.h +++ b/server/src/common/backend_ipc.h @@ -25,6 +25,7 @@ enum class BackendIpcMode { Invalid, DFlashDraft, PFlashCompress, + Qwen3ToolPredict, Qwen35TargetShard, Gemma4TargetShard, LagunaTargetShard, @@ -123,6 +124,15 @@ struct BackendIpcLaunchConfig { std::string work_dir; BackendIpcPayloadTransport payload_transport = BackendIpcPayloadTransport::Auto; size_t shared_payload_bytes = 0; + // Optional sidecars can fail closed instead of waiting forever for their + // initial status word. Zero preserves the legacy unbounded startup wait. + int readiness_timeout_ms = 0; + // Security-sensitive sidecars inherit only stdin/stdout/stderr and the + // explicitly configured payload/response descriptors. + bool isolate_inherited_fds = false; + // Keep legacy backend work directories compatible while allowing private + // sidecars to require a fresh 0700 child of an owned, non-symlink base. + bool require_private_work_dir = false; }; struct BackendIpcPayloadSegment { @@ -139,6 +149,9 @@ class BackendIpcProcess { bool start(const BackendIpcLaunchConfig & cfg); void close(); + // Stop a wedged daemon without waiting indefinitely for graceful EOF + // handling. Used by optional sidecar lanes with hard deadlines. + void terminate(); bool active() const { return active_; } FILE * command_stream() const { return cmd_; } @@ -152,6 +165,13 @@ class BackendIpcProcess { const std::string & work_dir() const { return work_dir_; } std::string next_path(const char * prefix); + // Create/remove a regular file relative to the retained private directory + // descriptor. The returned name is relative to the daemon's private cwd. + bool write_private_file(const char * prefix, + const void * data, + size_t bytes, + std::string & name); + bool remove_private_file(const std::string & name); bool write_shared_payload(const void * data, size_t bytes, uint64_t & seq); bool write_shared_payload_segments(const BackendIpcPayloadSegment * segments, size_t n_segments, @@ -159,11 +179,13 @@ class BackendIpcProcess { bool read_shared_payload(void * data, size_t bytes, uint64_t seq) const; private: + void close_impl(bool force_terminate); #if !defined(_WIN32) - bool init_work_dir(const std::string & requested); + bool init_work_dir(const std::string & requested, bool require_private); bool init_shared_payload(size_t bytes); pid_t pid_ = -1; + int work_dir_fd_ = -1; #endif FILE * cmd_ = nullptr; int payload_fd_ = -1; diff --git a/server/src/common/io_utils.h b/server/src/common/io_utils.h index a4818b4f5..6097c3616 100644 --- a/server/src/common/io_utils.h +++ b/server/src/common/io_utils.h @@ -3,6 +3,9 @@ #pragma once +#include +#include +#include #include #include #include @@ -18,6 +21,7 @@ # include #else # include +# include # include #endif @@ -110,6 +114,62 @@ static inline bool read_exact_fd(int fd, void * data, size_t bytes) { return true; } +// Read a complete protocol field under one wall-clock deadline. The helper is +// shared by startup handshakes and request/response IPC so partial reads and +// timeout behavior cannot drift between sidecars. +static inline bool read_exact_fd_until( + int fd, + void * data, + size_t bytes, + const std::chrono::steady_clock::time_point & deadline, + bool & timed_out) { + char * cursor = static_cast(data); + size_t received = 0; + timed_out = false; + while (received < bytes) { + const auto now = std::chrono::steady_clock::now(); + if (now >= deadline) { + timed_out = true; + return false; + } + const int64_t remaining = + std::chrono::duration_cast( + deadline - now).count(); + pollfd descriptor{fd, POLLIN | POLLHUP, 0}; + const int wait_ms = static_cast((std::min)( + int64_t{INT_MAX}, (std::max)(int64_t{0}, remaining))); + const int polled = ::poll(&descriptor, 1, wait_ms); + if (polled == 0) { + if (std::chrono::steady_clock::now() >= deadline) { + timed_out = true; + return false; + } + continue; + } + if (polled < 0) { + if (errno == EINTR) continue; + return false; + } + if (descriptor.revents & (POLLERR | POLLNVAL)) return false; + // Readiness does not extend the wall-clock budget. In particular, a + // descriptor that becomes readable during poll's final millisecond + // must not be consumed after the deadline. + if (std::chrono::steady_clock::now() >= deadline) { + timed_out = true; + return false; + } + const ssize_t count = ::read( + fd, cursor + received, bytes - received); + if (count == 0) return false; + if (count < 0) { + if (errno == EINTR) continue; + return false; + } + received += static_cast(count); + } + return true; +} + static inline bool write_exact_fd(int fd, const void * data, size_t bytes) { const char * p = (const char *)data; size_t done = 0; diff --git a/server/src/common/qwen3_tool_predictor_ipc.cpp b/server/src/common/qwen3_tool_predictor_ipc.cpp new file mode 100644 index 000000000..421f82b9f --- /dev/null +++ b/server/src/common/qwen3_tool_predictor_ipc.cpp @@ -0,0 +1,228 @@ +#include "qwen3_tool_predictor_ipc.h" + +#include "io_utils.h" + +#include +#include +#include +#include +#include + +namespace dflash::common { + +bool read_qwen3_tool_predictor_response( + int stream_fd, + int max_tokens, + int timeout_ms, + std::vector & output_ids, + std::string & error) { + output_ids.clear(); + error.clear(); +#if !defined(__linux__) + (void)stream_fd; + (void)max_tokens; + (void)timeout_ms; + error = "native_predictor_ipc_unsupported"; + return false; +#else + if (stream_fd < 0 || max_tokens <= 0 || timeout_ms <= 0) { + error = "native_predictor_invalid_request"; + return false; + } + const auto deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(timeout_ms); + bool timed_out = false; + int32_t status = -1; + if (!read_exact_fd_until( + stream_fd, &status, sizeof(status), deadline, timed_out)) { + error = timed_out + ? "native_predictor_timeout" + : "native_predictor_invalid_response"; + return false; + } + if (status != 0) { + error = "native_predictor_generation_failed"; + return false; + } + + int32_t count = -1; + if (!read_exact_fd_until( + stream_fd, &count, sizeof(count), deadline, timed_out) || + count <= 0 || count > max_tokens) { + error = timed_out + ? "native_predictor_timeout" + : "native_predictor_invalid_response"; + return false; + } + output_ids.assign(static_cast(count), 0); + if (!read_exact_fd_until( + stream_fd, output_ids.data(), + output_ids.size() * sizeof(int32_t), deadline, timed_out)) { + output_ids.clear(); + error = timed_out + ? "native_predictor_timeout" + : "native_predictor_invalid_response"; + return false; + } + return true; +#endif +} + +bool Qwen3ToolPredictorIpcClient::start( + const std::string & bin, + const std::string & model_path, + int gpu, + int max_ctx, + const std::string & work_dir, + int readiness_timeout_ms) { +#if !defined(__linux__) + (void)bin; (void)model_path; (void)gpu; (void)max_ctx; (void)work_dir; + (void)readiness_timeout_ms; + std::fprintf(stderr, + "Qwen3 tool-predictor IPC is only implemented on Linux hosts\n"); + return false; +#else + std::lock_guard lock(mutex_); + close_locked(); + if (bin.empty() || model_path.empty() || max_ctx <= 0 || + readiness_timeout_ms <= 0) return false; + + std::error_code path_error; + const std::string resolved_bin = + std::filesystem::canonical(bin, path_error).string(); + if (path_error) { + std::fprintf(stderr, + "[tool-predictor-ipc] cannot resolve IPC binary: %s\n", + path_error.message().c_str()); + return false; + } + const std::string resolved_model = + std::filesystem::canonical(model_path, path_error).string(); + if (path_error) { + std::fprintf(stderr, + "[tool-predictor-ipc] cannot resolve model: %s\n", + path_error.message().c_str()); + return false; + } + BackendIpcLaunchConfig launch; + launch.bin = resolved_bin; + launch.mode = BackendIpcMode::Qwen3ToolPredict; + launch.payload_path = resolved_model; + launch.work_dir = work_dir; + launch.args.push_back("--target-gpu=" + std::to_string(std::max(0, gpu))); + launch.args.push_back("--max-ctx=" + std::to_string(max_ctx)); + launch.readiness_timeout_ms = readiness_timeout_ms; + launch.isolate_inherited_fds = true; + launch.require_private_work_dir = true; + if (!process_.start(launch)) { + std::fprintf(stderr, "[tool-predictor-ipc] backend process start failed\n"); + return false; + } + active_ = true; + std::fprintf(stderr, + "[tool-predictor-ipc] ready model=%s gpu=%d max_ctx=%d work_dir=%s\n", + model_path.c_str(), std::max(0, gpu), max_ctx, + process_.work_dir().c_str()); + return true; +#endif +} + +bool Qwen3ToolPredictorIpcClient::predict( + const std::vector & prompt_ids, + int max_tokens, + int timeout_ms, + std::vector & output_ids, + std::string & error) { + output_ids.clear(); + error.clear(); +#if !defined(__linux__) + (void)prompt_ids; (void)max_tokens; (void)timeout_ms; + error = "native_predictor_ipc_unsupported"; + return false; +#else + if (prompt_ids.empty() || max_tokens <= 0 || timeout_ms <= 0) { + error = "native_predictor_invalid_request"; + return false; + } + const auto deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(timeout_ms); + std::unique_lock lock(mutex_, std::defer_lock); + if (!lock.try_lock_until(deadline)) { + error = "native_predictor_timeout"; + return false; + } + FILE * command = process_.command_stream(); + const int stream_fd = process_.stream_fd(); + if (!active_.load(std::memory_order_acquire) || + !command || stream_fd < 0) { + error = "native_predictor_not_active"; + return false; + } + + std::string prompt_name; + if (!process_.write_private_file( + "tool_predictor_prompt", prompt_ids.data(), + prompt_ids.size() * sizeof(int32_t), prompt_name)) { + error = "native_predictor_prompt_write_failed"; + return false; + } + + if (std::chrono::steady_clock::now() >= deadline) { + (void)process_.remove_private_file(prompt_name); + error = "native_predictor_timeout"; + return false; + } + + if (std::fprintf( + command, "predict %d %s\n", max_tokens, prompt_name.c_str()) < 0 || + std::fflush(command) != 0) { + (void)process_.remove_private_file(prompt_name); + error = "native_predictor_command_write_failed"; + process_.terminate(); + active_ = false; + return false; + } + const auto response_started = std::chrono::steady_clock::now(); + if (response_started >= deadline) { + (void)process_.remove_private_file(prompt_name); + error = "native_predictor_timeout"; + process_.terminate(); + active_ = false; + return false; + } + const int response_timeout_ms = std::max(1, static_cast( + std::chrono::duration_cast( + deadline - response_started).count())); + const bool ok = read_qwen3_tool_predictor_response( + stream_fd, max_tokens, response_timeout_ms, output_ids, error); + (void)process_.remove_private_file(prompt_name); + if (!ok) { + output_ids.clear(); + process_.terminate(); + active_ = false; + return false; + } + if (std::chrono::steady_clock::now() >= deadline) { + output_ids.clear(); + error = "native_predictor_timeout"; + return false; + } + return true; +#endif +} + +bool Qwen3ToolPredictorIpcClient::active() const { + return active_.load(std::memory_order_acquire); +} + +void Qwen3ToolPredictorIpcClient::close_locked() { + process_.close(); + active_ = false; +} + +void Qwen3ToolPredictorIpcClient::close() { + std::lock_guard lock(mutex_); + close_locked(); +} + +} // namespace dflash::common diff --git a/server/src/common/qwen3_tool_predictor_ipc.h b/server/src/common/qwen3_tool_predictor_ipc.h new file mode 100644 index 000000000..6b401b0ed --- /dev/null +++ b/server/src/common/qwen3_tool_predictor_ipc.h @@ -0,0 +1,70 @@ +// Persistent Qwen3 tool-predictor IPC lane. +// +// The HTTP server tokenizes the predictor prompt with the predictor's own +// vocabulary, then sends token IDs to a small out-of-process Qwen3 backend. +// Keeping this lane behind BackendIpcProcess isolates the target decoder from +// predictor crashes and lets heterogeneous deployments choose a different GPU. + +#pragma once + +#include "backend_ipc.h" + +#include +#include +#include +#include +#include +#include + +namespace dflash::common { + +class Qwen3ToolPredictorIpcClient { +public: + Qwen3ToolPredictorIpcClient() = default; + Qwen3ToolPredictorIpcClient(const Qwen3ToolPredictorIpcClient &) = delete; + Qwen3ToolPredictorIpcClient & operator=( + const Qwen3ToolPredictorIpcClient &) = delete; + ~Qwen3ToolPredictorIpcClient() { close(); } + + bool start(const std::string & bin, + const std::string & model_path, + int gpu, + int max_ctx, + const std::string & work_dir, + int readiness_timeout_ms); + + // Requests are serialized: one compact predictor model owns one KV cache. + // On transport or generation failure the lane closes and fails shut. + bool predict(const std::vector & prompt_ids, + int max_tokens, + int timeout_ms, + std::vector & output_ids, + std::string & error); + + bool active() const; + void close(); + +private: + void close_locked(); + + mutable std::timed_mutex mutex_; + BackendIpcProcess process_; + std::atomic active_{false}; +}; + +// Read one daemon response under a single wall-clock deadline. Kept outside +// the client so the timeout and partial-response behavior can be unit tested +// without loading a model. +bool read_qwen3_tool_predictor_response( + int stream_fd, + int max_tokens, + int timeout_ms, + std::vector & output_ids, + std::string & error); + +int run_qwen3_tool_predictor_ipc_daemon(const char * model_path, + int gpu, + int max_ctx, + int stream_fd); + +} // namespace dflash::common diff --git a/server/src/common/qwen3_tool_predictor_ipc_daemon.cpp b/server/src/common/qwen3_tool_predictor_ipc_daemon.cpp new file mode 100644 index 000000000..9ebe96fd3 --- /dev/null +++ b/server/src/common/qwen3_tool_predictor_ipc_daemon.cpp @@ -0,0 +1,120 @@ +#include "qwen3_tool_predictor_ipc.h" + +#include "io_utils.h" +#include "model_backend.h" +#include "qwen3/qwen3_backend.h" + +#include +#include +#include +#include +#include + +namespace dflash::common { +namespace { + +bool send_status(int stream_fd, int32_t status) { + return write_exact_fd(stream_fd, &status, sizeof(status)); +} + +} // namespace + +int run_qwen3_tool_predictor_ipc_daemon( + const char * model_path, + int gpu, + int max_ctx, + int stream_fd) { +#if defined(_WIN32) + (void)model_path; (void)gpu; (void)max_ctx; (void)stream_fd; + return 2; +#else + if (!model_path || !*model_path || stream_fd < 0 || max_ctx <= 0) { + std::fprintf(stderr, + "usage: backend_ipc_daemon --backend-ipc-mode=qwen3-tool-predict " + " --stream-fd=FD --target-gpu=N --max-ctx=N\n"); + return 2; + } + + Qwen3BackendConfig config; + config.model_path = model_path; + config.device.backend = PlacementBackend::Auto; + config.device.gpu = std::max(0, gpu); + config.device.max_ctx = max_ctx; + config.chunk = 512; + + Qwen3Backend backend(config); + if (!backend.init()) { + std::fprintf(stderr, "[tool-predictor-daemon] Qwen3 init failed\n"); + send_status(stream_fd, -1); + return 1; + } + std::fprintf(stderr, + "[tool-predictor-daemon] ready gpu=%d max_ctx=%d\n", + std::max(0, gpu), max_ctx); + send_status(stream_fd, 0); + + std::string line; + while (std::getline(std::cin, line)) { + std::istringstream input(line); + std::string command; + input >> command; + if (command == "quit" || command == "exit") break; + if (command != "predict") { + std::fprintf(stderr, + "[tool-predictor-daemon] unknown command: %s\n", + line.c_str()); + send_status(stream_fd, -1); + continue; + } + + int max_tokens = 0; + input >> max_tokens; + const std::string path = read_line_tail(input); + if (max_tokens <= 0 || path.empty()) { + send_status(stream_fd, -1); + continue; + } + const auto prompt = read_int32_file(path); + if (prompt.empty() || + prompt.size() + static_cast(max_tokens) > + static_cast(max_ctx)) { + std::fprintf(stderr, + "[tool-predictor-daemon] invalid context prompt=%zu max_tokens=%d max_ctx=%d\n", + prompt.size(), max_tokens, max_ctx); + send_status(stream_fd, -1); + continue; + } + + GenerateRequest request; + request.prompt = prompt; + request.n_gen = max_tokens; + request.do_sample = false; + request.stream = false; + DaemonIO io; + const GenerateResult result = backend.generate(request, io); + if (!result.ok() || result.tokens.empty()) { + std::fprintf(stderr, + "[tool-predictor-daemon] generation failed code=%s\n", + result.error_code().data()); + send_status(stream_fd, -1); + continue; + } + + const int32_t count = static_cast(result.tokens.size()); + if (!send_status(stream_fd, 0) || + !write_exact_fd(stream_fd, &count, sizeof(count)) || + !write_exact_fd(stream_fd, result.tokens.data(), + result.tokens.size() * sizeof(int32_t))) { + std::fprintf(stderr, + "[tool-predictor-daemon] response write failed\n"); + break; + } + } + + backend.shutdown(); + std::fprintf(stderr, "[tool-predictor-daemon] stopped\n"); + return 0; +#endif +} + +} // namespace dflash::common diff --git a/server/src/ipc/backend_ipc_main.cpp b/server/src/ipc/backend_ipc_main.cpp index b2fae7791..e2764ef9d 100644 --- a/server/src/ipc/backend_ipc_main.cpp +++ b/server/src/ipc/backend_ipc_main.cpp @@ -8,6 +8,7 @@ #include "gemma4/gemma4_layer_split_adapter.h" #include "laguna/laguna_layer_split_adapter.h" #include "pflash_drafter_ipc.h" +#include "qwen3_tool_predictor_ipc.h" #include "common/platform_env.h" #include "qwen35_target_shard_ipc.h" @@ -117,6 +118,8 @@ int main(int argc, char ** argv) { "[--shared-payload-fd=FD --shared-payload-bytes=N] [--draft-gpu=N]\n" " or: %s --backend-ipc-mode=pflash-compress " "--stream-fd=FD [--draft-gpu=N]\n" + " or: %s --backend-ipc-mode=qwen3-tool-predict " + "--stream-fd=FD --target-gpu=N --max-ctx=N\n" " or: %s --backend-ipc-mode=qwen35-target-shard " "--stream-fd=FD --target-gpu=N --layer-begin=N --layer-end=N " "--max-ctx=N [--hidden=N --vocab=N --max-tokens=N]\n" @@ -138,6 +141,8 @@ int main(int argc, char ** argv) { argv[0], argv[0], argv[0], + argv[0], + argv[0], argv[0]); return 2; } @@ -315,6 +320,9 @@ int main(int argc, char ** argv) { shared_payload_bytes); case BackendIpcMode::PFlashCompress: return run_pflash_drafter_ipc_daemon(payload_path, draft_gpu, stream_fd); + case BackendIpcMode::Qwen3ToolPredict: + return run_qwen3_tool_predictor_ipc_daemon( + payload_path, target_gpu, max_ctx, stream_fd); case BackendIpcMode::Qwen35TargetShard: if (target_gpus.empty()) target_gpus.push_back(target_gpu); if (layer_begins.empty()) layer_begins.push_back(layer_begin); diff --git a/server/src/qwen3/qwen3_backend.cpp b/server/src/qwen3/qwen3_backend.cpp index 454bd5e00..f7c798938 100644 --- a/server/src/qwen3/qwen3_backend.cpp +++ b/server/src/qwen3/qwen3_backend.cpp @@ -502,7 +502,9 @@ bool Qwen3Backend::do_decode(int committed, int n_gen, ggml_free(ectx); } - if (!do_step(embed_buf.data(), 1, committed, logits)) { + // `committed` was advanced when `next` was accepted. Write that token + // at its zero-based KV position instead of skipping one cache slot. + if (!do_step(embed_buf.data(), 1, committed - 1, logits)) { return false; } } diff --git a/server/src/qwen3/qwen3_loader.cpp b/server/src/qwen3/qwen3_loader.cpp index 583261992..49e5f9226 100644 --- a/server/src/qwen3/qwen3_loader.cpp +++ b/server/src/qwen3/qwen3_loader.cpp @@ -108,6 +108,25 @@ float get_f32(gguf_context * g, const char * key, float def) { return gguf_get_val_f32(g, k); } +bool supported_weight_storage(ggml_type type) { + return type == GGML_TYPE_Q8_0 || type == GGML_TYPE_BF16 || + type == GGML_TYPE_F16; +} + +ggml_type tensor_storage_type(gguf_context * gctx, const char * name, + ggml_type fallback, bool & ok) { + const int64_t index = gguf_find_tensor(gctx, name); + if (index < 0) return fallback; + const ggml_type type = gguf_get_tensor_type(gctx, index); + if (!supported_weight_storage(type)) { + set_last_error(std::string("unsupported Qwen3-0.6B storage type for ") + + name + ": " + ggml_type_name(type)); + ok = false; + return fallback; + } + return type; +} + } // namespace bool load_qwen3_drafter_model(const std::string & path, @@ -134,7 +153,11 @@ bool load_qwen3_drafter_model(const std::string & path, out.head_dim = (int)get_u32(gctx, "qwen3.attention.key_length", 128); out.rope_theta = get_f32(gctx, "qwen3.rope.freq_base", 1000000.0f); - // Detect weight quant type from blk.0.attn_q.weight; support BF16 and Q8_0. + // Preserve Q8_0 storage when the predictor reuses the production compact + // GGUF. Activations/KV still use the backend precision policy; ggml's + // mul_mat and get_rows kernels dequantize Q8_0 weights as they are read. + // This avoids expanding a 0.6B sidecar to BF16 merely to use the native + // IPC lane. ggml_type wtype = GGML_TYPE_BF16; { int64_t tidx = gguf_find_tensor(gctx, "blk.0.attn_q.weight"); @@ -142,8 +165,16 @@ bool load_qwen3_drafter_model(const std::string & path, wtype = gguf_get_tensor_type(gctx, tidx); } } + if (supported_weight_storage(wtype)) { + out.weight_type = wtype; + } else { + set_last_error(std::string("unsupported Qwen3-0.6B weight type: ") + + ggml_type_name(wtype)); + gguf_free(gctx); + return false; + } std::fprintf(stderr, "[qwen3-0.6b] detected weight type: %s\n", - wtype == GGML_TYPE_Q8_0 ? "Q8_0" : "BF16"); + ggml_type_name(wtype)); std::fflush(stderr); // Compute total tensor metadata size for context allocation. @@ -166,30 +197,76 @@ bool load_qwen3_drafter_model(const std::string & path, const int n_vocab = out.n_vocab; const int q_dim = n_head * head_dim; const int kv_dim = n_head_kv * head_dim; - const ggml_type weight_type = out.weight_type; + bool storage_types_ok = true; + const ggml_type token_type = tensor_storage_type( + gctx, "token_embd.weight", out.weight_type, storage_types_ok); + const ggml_type output_type = tensor_storage_type( + gctx, "output.weight", token_type, storage_types_ok); // Top-level tensors. - out.tok_embd = ggml_new_tensor_2d(out.ctx, weight_type, n_embd, n_vocab); + out.tok_embd = ggml_new_tensor_2d(out.ctx, token_type, n_embd, n_vocab); out.out_norm = ggml_new_tensor_1d(out.ctx, GGML_TYPE_F32, n_embd); - out.output = ggml_new_tensor_2d(out.ctx, weight_type, n_embd, n_vocab); + out.output = ggml_new_tensor_2d(out.ctx, output_type, n_embd, n_vocab); ggml_set_name(out.tok_embd, "token_embd.weight"); ggml_set_name(out.out_norm, "output_norm.weight"); ggml_set_name(out.output, "output.weight"); out.layers.resize(n_layer); + char tensor_name[128]; for (int il = 0; il < n_layer; ++il) { auto & L = out.layers[il]; L.attn_norm = ggml_new_tensor_1d(out.ctx, GGML_TYPE_F32, n_embd); - L.wq = ggml_new_tensor_2d(out.ctx, weight_type, n_embd, q_dim); - L.wk = ggml_new_tensor_2d(out.ctx, weight_type, n_embd, kv_dim); - L.wv = ggml_new_tensor_2d(out.ctx, weight_type, n_embd, kv_dim); - L.wo = ggml_new_tensor_2d(out.ctx, weight_type, q_dim, n_embd); + std::snprintf(tensor_name, sizeof(tensor_name), + "blk.%d.attn_q.weight", il); + L.wq = ggml_new_tensor_2d( + out.ctx, tensor_storage_type(gctx, tensor_name, out.weight_type, + storage_types_ok), + n_embd, q_dim); + std::snprintf(tensor_name, sizeof(tensor_name), + "blk.%d.attn_k.weight", il); + L.wk = ggml_new_tensor_2d( + out.ctx, tensor_storage_type(gctx, tensor_name, out.weight_type, + storage_types_ok), + n_embd, kv_dim); + std::snprintf(tensor_name, sizeof(tensor_name), + "blk.%d.attn_v.weight", il); + L.wv = ggml_new_tensor_2d( + out.ctx, tensor_storage_type(gctx, tensor_name, out.weight_type, + storage_types_ok), + n_embd, kv_dim); + std::snprintf(tensor_name, sizeof(tensor_name), + "blk.%d.attn_output.weight", il); + L.wo = ggml_new_tensor_2d( + out.ctx, tensor_storage_type(gctx, tensor_name, out.weight_type, + storage_types_ok), + q_dim, n_embd); L.q_norm = ggml_new_tensor_1d(out.ctx, GGML_TYPE_F32, head_dim); L.k_norm = ggml_new_tensor_1d(out.ctx, GGML_TYPE_F32, head_dim); L.ffn_norm = ggml_new_tensor_1d(out.ctx, GGML_TYPE_F32, n_embd); - L.ffn_gate = ggml_new_tensor_2d(out.ctx, weight_type, n_embd, n_ff); - L.ffn_up = ggml_new_tensor_2d(out.ctx, weight_type, n_embd, n_ff); - L.ffn_down = ggml_new_tensor_2d(out.ctx, weight_type, n_ff, n_embd); + std::snprintf(tensor_name, sizeof(tensor_name), + "blk.%d.ffn_gate.weight", il); + L.ffn_gate = ggml_new_tensor_2d( + out.ctx, tensor_storage_type(gctx, tensor_name, out.weight_type, + storage_types_ok), + n_embd, n_ff); + std::snprintf(tensor_name, sizeof(tensor_name), + "blk.%d.ffn_up.weight", il); + L.ffn_up = ggml_new_tensor_2d( + out.ctx, tensor_storage_type(gctx, tensor_name, out.weight_type, + storage_types_ok), + n_embd, n_ff); + std::snprintf(tensor_name, sizeof(tensor_name), + "blk.%d.ffn_down.weight", il); + L.ffn_down = ggml_new_tensor_2d( + out.ctx, tensor_storage_type(gctx, tensor_name, out.weight_type, + storage_types_ok), + n_ff, n_embd); + } + if (!storage_types_ok) { + gguf_free(gctx); + ggml_free(out.ctx); + out.ctx = nullptr; + return false; } out.buf = ggml_backend_alloc_ctx_tensors(out.ctx, backend); diff --git a/server/src/server/http_server.cpp b/server/src/server/http_server.cpp index e92f495bf..1efeddbea 100644 --- a/server/src/server/http_server.cpp +++ b/server/src/server/http_server.cpp @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -57,7 +58,7 @@ typedef long ssize_t; #define poll(fds,nfds,timeout) WSAPoll(fds,nfds,timeout) // Replace fcntl(F_GETFL) / fcntl(F_SETFL, O_NONBLOCK) with ioctlsocket static inline int sock_get_flags(SocketHandle fd) { (void)fd; return 0; /* stub */ } -static inline void sock_set_nonblock(SocketHandle fd) { u_long m = 1; ioctlsocket(fd, FIONBIO, &m); } +static inline bool sock_set_nonblock(SocketHandle fd) { u_long m = 1; return ioctlsocket(fd, FIONBIO, &m) == 0; } static inline void sock_set_block(SocketHandle fd) { u_long m = 0; ioctlsocket(fd, FIONBIO, &m); } static inline void socket_close(SocketHandle fd) { closesocket(fd); } #define SETSOCKOPT_CAST (const char *) @@ -72,9 +73,10 @@ static inline bool sock_is_eintr (int e) { return e == WSAEINTR; } static inline bool sock_is_eagain(int e) { return e == WSAEWOULDBLOCK; } #else #include +#include #include static inline int sock_get_flags(SocketHandle fd) { return fcntl(fd, F_GETFL, 0); } -static inline void sock_set_nonblock(SocketHandle fd) { int f = fcntl(fd, F_GETFL, 0); if (f >= 0) fcntl(fd, F_SETFL, f | O_NONBLOCK); } +static inline bool sock_set_nonblock(SocketHandle fd) { int f = fcntl(fd, F_GETFL, 0); return f >= 0 && fcntl(fd, F_SETFL, f | O_NONBLOCK) == 0; } static inline void sock_set_block(SocketHandle fd) { int f = fcntl(fd, F_GETFL, 0); if (f >= 0) fcntl(fd, F_SETFL, f & ~O_NONBLOCK); } static inline void socket_close(SocketHandle fd) { ::close(fd); } #define SETSOCKOPT_CAST /* empty on POSIX */ @@ -251,6 +253,14 @@ bool should_clamp_flowkv_disk_cache( return flowkv && policy.compress; } +bool tool_choice_disables_tool_calls(const json & tool_choice) { + if (tool_choice.is_string()) { + return tool_choice.get() == "none"; + } + return tool_choice.is_object() && + tool_choice.value("type", "") == "none"; +} + } // namespace http_detail // ─── curl helpers for upstream proxy ───────────────────────────────────── @@ -465,6 +475,412 @@ static bool curl_forward(const std::string & url, } #endif // DFLASH_HAS_CURL +namespace { + +struct SemanticSidecarUrl { + std::string host; + std::string port; + std::string path; +}; + +bool parse_semantic_sidecar_url( + const std::string & value, SemanticSidecarUrl & out) { + constexpr char kHttpPrefix[] = "http://"; + if (value.rfind(kHttpPrefix, 0) != 0) return false; + const size_t authority_begin = sizeof(kHttpPrefix) - 1; + const size_t path_begin = value.find('/', authority_begin); + const std::string authority = value.substr( + authority_begin, + path_begin == std::string::npos + ? std::string::npos + : path_begin - authority_begin); + if (authority.empty() || authority.find('@') != std::string::npos) { + return false; + } + + const size_t colon = authority.rfind(':'); + if (colon == std::string::npos) { + out.host = authority; + out.port = "80"; + } else { + out.host = authority.substr(0, colon); + out.port = authority.substr(colon + 1); + } + out.path = path_begin == std::string::npos + ? "/" : value.substr(path_begin); + if (out.host.empty() || out.port.empty() || out.path.empty()) return false; + if (out.host.front() == '[' || out.host.find(':') != std::string::npos) { + // Reject ambiguous IPv6 authority parsing instead of silently + // connecting to the wrong host. + return false; + } + if (!std::all_of(out.port.begin(), out.port.end(), [](unsigned char ch) { + return std::isdigit(ch) != 0; + })) { + return false; + } + try { + const unsigned long port = std::stoul(out.port); + if (port == 0 || port > 65535) return false; + } catch (...) { + return false; + } + // A numeric address makes the configured wall-clock deadline independent + // of an unbounded platform DNS resolver. Keep localhost as the ergonomic + // production spelling for an on-host predictor. + if (out.host == "localhost") out.host = "127.0.0.1"; + in_addr address{}; + return ::inet_pton(AF_INET, out.host.c_str(), &address) == 1; +} + +bool semantic_sidecar_send_all( + SocketHandle fd, + const char * data, + size_t size, + const std::chrono::steady_clock::time_point & deadline) { + size_t sent = 0; + while (sent < size) { + const auto now = std::chrono::steady_clock::now(); + if (now >= deadline) return false; + const int remaining_ms = std::max(1, static_cast( + std::chrono::duration_cast( + deadline - now).count())); + struct pollfd descriptor{fd, POLLOUT, 0}; + const int polled = poll(&descriptor, 1, remaining_ms); + if (polled < 0 && sock_is_eintr(sock_errno())) continue; + if (polled <= 0 || + !(descriptor.revents & POLLOUT) || + (descriptor.revents & (POLLERR | POLLHUP | POLLNVAL))) { + return false; + } + if (std::chrono::steady_clock::now() >= deadline) return false; +#if defined(_WIN32) + const int n = ::send( + fd, data + sent, + static_cast(std::min(size - sent, INT_MAX)), 0); +#else + const ssize_t n = ::send( + fd, data + sent, size - sent, MSG_NOSIGNAL); +#endif + if (n < 0 && (sock_is_eintr(sock_errno()) || + sock_is_eagain(sock_errno()))) { + continue; + } + if (n <= 0) return false; + sent += static_cast(n); + } + return true; +} + +bool semantic_sidecar_connect( + const SemanticSidecarUrl & url, + const std::chrono::steady_clock::time_point & deadline, + SocketHandle & fd) { + fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); + if (!socket_is_valid(fd)) return false; + if (!sock_set_nonblock(fd)) { + socket_close(fd); + fd = kInvalidSocket; + return false; + } + + sockaddr_in address{}; + address.sin_family = AF_INET; + address.sin_port = htons(static_cast(std::stoul(url.port))); + if (::inet_pton(AF_INET, url.host.c_str(), &address.sin_addr) != 1) { + socket_close(fd); + fd = kInvalidSocket; + return false; + } + if (::connect(fd, reinterpret_cast(&address), + static_cast(sizeof(address))) == 0) { + return true; + } + const int connect_error = sock_errno(); +#if defined(_WIN32) + const bool pending = connect_error == WSAEWOULDBLOCK || + connect_error == WSAEINPROGRESS || + connect_error == WSAEINVAL; +#else + const bool pending = connect_error == EINPROGRESS || + connect_error == EWOULDBLOCK; +#endif + if (!pending) { + socket_close(fd); + fd = kInvalidSocket; + return false; + } + + struct pollfd descriptor{fd, POLLOUT, 0}; + int polled = -1; + while (true) { + const auto now = std::chrono::steady_clock::now(); + if (now >= deadline) { + socket_close(fd); + fd = kInvalidSocket; + return false; + } + const int remaining_ms = std::max(1, static_cast( + std::chrono::duration_cast( + deadline - now).count())); + descriptor.revents = 0; + polled = poll(&descriptor, 1, remaining_ms); + if (polled < 0 && sock_is_eintr(sock_errno())) continue; + break; + } + int socket_error = 0; + socklen_t error_size = static_cast(sizeof(socket_error)); + if (std::chrono::steady_clock::now() >= deadline || + polled <= 0 || !(descriptor.revents & POLLOUT) || + getsockopt(fd, SOL_SOCKET, SO_ERROR, + reinterpret_cast(&socket_error), &error_size) != 0 || + socket_error != 0) { + socket_close(fd); + fd = kInvalidSocket; + return false; + } + return true; +} + +std::string lowercase_ascii(std::string value) { + std::transform(value.begin(), value.end(), value.begin(), + [](unsigned char ch) { return static_cast(std::tolower(ch)); }); + return value; +} + +bool decode_chunked_http_body( + const std::string & encoded, std::string & decoded) { + size_t offset = 0; + while (true) { + const size_t line_end = encoded.find("\r\n", offset); + if (line_end == std::string::npos) return false; + const std::string size_text = encoded.substr(offset, line_end - offset); + const size_t extension = size_text.find(';'); + const std::string hex = size_text.substr(0, extension); + size_t parsed = 0; + unsigned long chunk_size = 0; + try { + chunk_size = std::stoul(hex, &parsed, 16); + } catch (...) { + return false; + } + if (parsed != hex.size()) return false; + offset = line_end + 2; + if (chunk_size == 0) return true; + if (chunk_size > encoded.size() - std::min(offset, encoded.size())) { + return false; + } + decoded.append(encoded, offset, static_cast(chunk_size)); + offset += static_cast(chunk_size); + if (offset + 2 > encoded.size() || + encoded.compare(offset, 2, "\r\n") != 0) { + return false; + } + offset += 2; + } +} + +SemanticToolPrediction request_semantic_tool_prediction( + const SemanticToolPredictorConfig & config, + const json & payload, + const json & request_tools) { + const auto started = std::chrono::steady_clock::now(); + const auto deadline = started + + std::chrono::milliseconds(std::max(1, config.timeout_ms)); + SemanticToolPrediction prediction; + prediction.source = config.model; + auto finish = [&]() { + prediction.wall_ms = std::chrono::duration( + std::chrono::steady_clock::now() - started).count(); + return prediction; + }; + + SemanticSidecarUrl url; + if (!parse_semantic_sidecar_url(config.url, url)) { + prediction.error = "predictor_url_must_use_http_numeric_ipv4"; + return finish(); + } + + SocketHandle fd = kInvalidSocket; + if (!semantic_sidecar_connect(url, deadline, fd)) { + prediction.error = "predictor_connect_failed"; + return finish(); + } + + const std::string body = payload.dump(); + const std::string request = + "POST " + url.path + " HTTP/1.1\r\n" + + "Host: " + url.host + ":" + url.port + "\r\n" + + "Content-Type: application/json\r\n" + + "Accept: application/json\r\n" + + "Connection: close\r\n" + + "Content-Length: " + std::to_string(body.size()) + "\r\n\r\n" + + body; + if (!semantic_sidecar_send_all( + fd, request.data(), request.size(), deadline)) { + socket_close(fd); + prediction.error = std::chrono::steady_clock::now() >= deadline + ? "predictor_timeout" : "predictor_send_failed"; + return finish(); + } + + constexpr size_t kMaxResponseBytes = 1024 * 1024; + std::string response; + std::array buffer{}; + while (response.size() < kMaxResponseBytes) { + const auto now = std::chrono::steady_clock::now(); + if (now >= deadline) { + socket_close(fd); + prediction.error = "predictor_timeout"; + return finish(); + } + const int remaining_ms = std::max(1, static_cast( + std::chrono::duration_cast( + deadline - now).count())); + struct pollfd descriptor{fd, POLLIN | POLLHUP, 0}; + const int polled = poll(&descriptor, 1, remaining_ms); + if (polled < 0 && sock_is_eintr(sock_errno())) continue; + if (polled == 0) { + socket_close(fd); + prediction.error = "predictor_timeout"; + return finish(); + } + if (polled < 0 || + (descriptor.revents & (POLLERR | POLLNVAL))) { + socket_close(fd); + prediction.error = "predictor_receive_failed"; + return finish(); + } + if (std::chrono::steady_clock::now() >= deadline) { + socket_close(fd); + prediction.error = "predictor_timeout"; + return finish(); + } +#if defined(_WIN32) + const int received = recv( + fd, buffer.data(), static_cast(buffer.size()), 0); +#else + const ssize_t received = recv(fd, buffer.data(), buffer.size(), 0); +#endif + if (received == 0) break; + if (received < 0) { + if (sock_is_eintr(sock_errno()) || + sock_is_eagain(sock_errno())) { + continue; + } + socket_close(fd); + prediction.error = "predictor_receive_failed"; + return finish(); + } + response.append(buffer.data(), static_cast(received)); + } + socket_close(fd); + if (response.size() >= kMaxResponseBytes) { + prediction.error = "predictor_response_too_large"; + return finish(); + } + + const size_t header_end = response.find("\r\n\r\n"); + const size_t status_end = response.find("\r\n"); + if (header_end == std::string::npos || status_end == std::string::npos) { + prediction.error = "predictor_malformed_http_response"; + return finish(); + } + const std::string status_line = response.substr(0, status_end); + const size_t status_space = status_line.find(' '); + if (status_space == std::string::npos || + status_line.size() < status_space + 4) { + prediction.error = "predictor_malformed_http_status"; + return finish(); + } + const int status = std::atoi(status_line.c_str() + status_space + 1); + if (status < 200 || status >= 300) { + prediction.error = "predictor_http_status_" + std::to_string(status); + return finish(); + } + + const std::string headers = lowercase_ascii( + response.substr(status_end + 2, header_end - status_end - 2)); + const std::string encoded_body = response.substr(header_end + 4); + std::string response_body; + if (headers.find("transfer-encoding: chunked") != std::string::npos) { + if (!decode_chunked_http_body(encoded_body, response_body)) { + prediction.error = "predictor_invalid_chunked_response"; + return finish(); + } + } else { + response_body = encoded_body; + } + + json response_json; + try { + response_json = json::parse(response_body); + } catch (...) { + prediction.error = "predictor_response_invalid_json"; + return finish(); + } + if (!parse_semantic_tool_prediction( + response_json, request_tools, prediction.call, + prediction.error)) { + return finish(); + } + if (!materialize_declared_tool_defaults( + request_tools, prediction.call, prediction.error)) { + return finish(); + } + prediction.ok = true; + return finish(); +} + +SemanticToolPrediction predict_semantic_tool_call( + const SemanticToolPredictorConfig & config, + const SemanticToolPredictorRequest & request, + const json & request_tools, + const std::shared_ptr & native) { + const auto started = std::chrono::steady_clock::now(); + SemanticToolPrediction native_result; + if (native && native->active()) { + native_result = native->predict(request, request_tools); + if (native_result.ok || !config.http_enabled()) { + return native_result; + } + } + if (!config.http_enabled()) { + if (native_result.error.empty()) { + native_result.error = "native_predictor_not_initialized"; + } + return native_result; + } + + const double elapsed_ms = std::chrono::duration( + std::chrono::steady_clock::now() - started).count(); + const int remaining_ms = config.timeout_ms - + static_cast(std::ceil(elapsed_ms)); + if (remaining_ms <= 0) { + native_result.source = config.model; + native_result.ok = false; + native_result.error = native_result.error.empty() + ? "predictor_timeout" + : "native=" + native_result.error + ";http=predictor_timeout"; + native_result.wall_ms = elapsed_ms; + return native_result; + } + + SemanticToolPredictorConfig fallback_config = config; + fallback_config.timeout_ms = remaining_ms; + SemanticToolPrediction fallback = request_semantic_tool_prediction( + fallback_config, request.payload(), request_tools); + if (!fallback.ok && !native_result.error.empty()) { + fallback.error = "native=" + native_result.error + + ";http=" + fallback.error; + } + fallback.wall_ms = std::chrono::duration( + std::chrono::steady_clock::now() - started).count(); + return fallback; +} + +} // namespace + // ─── /props constants ─────────────────────────────────────────────────── // // SERVER_NAME / SERVER_VERSION mirror the Python server's identity strings @@ -750,6 +1166,16 @@ json build_props_body(const ServerConfig & config, // benchmarks to silently run at temp=0 (degenerate-decode collapse) // when the model card specifies temp=1.0/top_p=0.95/top_k=64. const auto & smp = config.sampler_defaults; + json tool_spec_lanes = json::array(); + for (const auto & lane : config.tool_speculation.policy.lanes()) { + tool_spec_lanes.push_back({ + {"resource_percentage", lane.resource_percentage}, + {"model_slowdown_ratio", lane.model_slowdown_ratio}, + {"decode_interference_qualified", + lane.decode_interference_qualified}, + {"accelerator_relation", lane.accelerator_relation}, + }); + } json body = { {"default_generation_settings", { {"n_ctx", config.max_ctx}, @@ -827,6 +1253,57 @@ json build_props_body(const ServerConfig & config, {"ddtree_budget", config.speculative_enabled ? json(config.ddtree_budget) : json(nullptr)}, }}, + {"tool_speculation", { + {"enabled", config.tool_speculation.enabled()}, + {"automatic_prediction_enabled", + config.tool_speculation.enabled() && + config.semantic_tool_predictor.enabled()}, + {"prediction_source", + config.semantic_tool_predictor.native_enabled() + ? json("native-qwen3") + : config.semantic_tool_predictor.http_enabled() + ? json(config.semantic_tool_predictor.model) + : json(nullptr)}, + {"prediction_confidence", + config.semantic_tool_predictor.enabled() + ? json(config.semantic_tool_predictor.execution_confidence) + : json(nullptr)}, + {"predictor_schedule", + config.semantic_tool_predictor.enabled() + ? json("before-model") : json(nullptr)}, + {"predictor_decode_isolated", + config.semantic_tool_predictor.enabled()}, + {"execution_mode", config.tool_speculation.execution_mode()}, + {"profile_status", + config.tool_speculation.policy.empty() + ? json(nullptr) + : json(config.tool_speculation.policy.profile_status())}, + {"executor_contract", + config.tool_speculation.policy.executor_contract().empty() + ? json(nullptr) + : json(config.tool_speculation.policy.executor_contract())}, + {"protocol", "dflash.tool-speculation.v1"}, + {"client_prediction_required", + config.tool_speculation.enabled() && + !config.semantic_tool_predictor.enabled()}, + {"client_result_handling_required", + config.tool_speculation.enabled()}, + {"preserves_token_speculation", true}, + {"unqualified_lane_policy", "defer"}, + {"allowed_tools", config.tool_speculation.allowed_tools}, + {"max_model_slowdown_ratio", + config.tool_speculation.max_model_slowdown_ratio}, + {"compute_isolation", + config.tool_speculation.cpu_affinity_isolated + ? "disjoint_cpu_affinity" : "none"}, + {"cpu_affinity_isolated", + config.tool_speculation.cpu_affinity_isolated}, + {"tool_cpu_affinity", + config.tool_speculation.cpu_affinity}, + {"model_cpu_affinity", + config.tool_speculation.model_cpu_affinity}, + {"profile_lanes", tool_spec_lanes}, + }}, {"sampling", { {"capabilities", { {"supports_temperature", true}, @@ -1012,6 +1489,25 @@ std::vector normalize_chat_messages( return chat_msgs; } +namespace http_detail { + +json canonical_predictor_messages(std::vector messages) { + json canonical = json::array(); + for (ChatMessage & message : messages) { + json item = { + {"role", std::move(message.role)}, + {"content", std::move(message.content)}, + }; + if (!message.tool_call_id.empty()) { + item["tool_call_id"] = std::move(message.tool_call_id); + } + canonical.push_back(std::move(item)); + } + return canonical; +} + +} // namespace http_detail + // ─── Disk-cache identity salt ─────────────────────────────────────────── // Compute a 16-byte salt from inputs that affect KV cache validity: // model path + stat(size + mtime) [covers rope/yarn — GGUF-derived], @@ -1264,6 +1760,24 @@ HttpServer::~HttpServer() { #endif } +bool HttpServer::init_semantic_tool_predictor(std::string & error) { + error.clear(); + if (!config_.semantic_tool_predictor.native_enabled()) return true; + native_semantic_predictor_ = NativeSemanticToolPredictor::create( + config_.semantic_tool_predictor, error); + if (native_semantic_predictor_) return true; + // An explicitly configured HTTP lane is a valid fail-open fallback. The + // target remains authoritative and still verifies every prediction. + if (config_.semantic_tool_predictor.http_enabled()) { + std::fprintf(stderr, + "[tool-hint] native predictor unavailable (%s); using HTTP fallback\n", + error.c_str()); + error.clear(); + return true; + } + return false; +} + void HttpServer::shutdown() { // Signal worker and accept loop to stop. stopping_.store(true); @@ -1664,6 +2178,32 @@ bool HttpServer::parse_common_request_fields( if (body.contains("tools")) req.tools = body["tools"]; // Tool choice constraint for hint generation. if (body.contains("tool_choice")) req.tool_choice = body["tool_choice"]; + if (body.contains("automatic_tool_speculation")) { + if (!body["automatic_tool_speculation"].is_boolean()) { + send_error(fd, 400, + "automatic_tool_speculation must be a boolean"); + return false; + } + req.automatic_tool_speculation_enabled = + body["automatic_tool_speculation"].get(); + } + + if (body.contains("tool_speculation")) { + if (http_detail::tool_choice_disables_tool_calls(req.tool_choice)) { + send_error(fd, 400, + "tool_speculation cannot be used when tool_choice is none"); + return false; + } + ToolSpeculationPrediction prediction; + std::string prediction_error; + if (!parse_tool_speculation_prediction( + body["tool_speculation"], req.tools, prediction, + prediction_error)) { + send_error(fd, 400, prediction_error); + return false; + } + req.tool_speculation = std::move(prediction); + } if (body.contains("prefix_cache") && body["prefix_cache"].is_object()) { const auto & prefix_cache = body["prefix_cache"]; @@ -1969,6 +2509,80 @@ void HttpServer::log_parsed_request(const ParsedRequest & req) const { req.stop_sequences.size(), req.model.c_str()); } +void HttpServer::start_automatic_tool_speculation(ParsedRequest & req) const { + if (req.automatic_tool_speculation.has_value() || + !req.automatic_tool_speculation_enabled || + req.tool_speculation.has_value() || + !config_.tool_speculation.enabled() || + !config_.semantic_tool_predictor.enabled() || + http_detail::tool_choice_disables_tool_calls(req.tool_choice) || + req.tools.empty()) return; + + req.automatic_tool_speculation.emplace(); + auto & launch = *req.automatic_tool_speculation; + const SemanticToolPredictorConfig predictor = + config_.semantic_tool_predictor; + const auto native = native_semantic_predictor_; + auto log_completion = [&]() { + std::fprintf(stderr, + "[tool-hint] predictor complete transport=%s%s tools=%zu " + "execute=%s wall_ms=%.1f\n", + native ? "native-qwen3" : "http", + native && predictor.http_enabled() ? "+http-fallback" : "", + json_array_size(req.tools), launch.attempt ? "true" : "false", + launch.predictor_wall_ms); + }; + const auto preflight_started = std::chrono::steady_clock::now(); + std::string payload_error; + const json & predictor_messages = req.predictor_messages.is_array() + ? req.predictor_messages : req.messages; + const auto payload = build_semantic_tool_predictor_request( + predictor_messages, req.tools, req.tool_choice, + predictor.model.empty() ? "native-qwen3" : predictor.model, + predictor.max_tokens, payload_error); + if (!payload.has_value()) { + launch.predictor_error = payload_error.empty() + ? "predictor_request_invalid" : std::move(payload_error); + launch.prediction_source = "request-preflight"; + launch.predictor_wall_ms = std::chrono::duration( + std::chrono::steady_clock::now() - preflight_started).count(); + log_completion(); + return; + } + try { + const SemanticToolPrediction semantic = predict_semantic_tool_call( + predictor, *payload, req.tools, native); + launch.predictor_wall_ms = semantic.wall_ms; + launch.prediction_source = semantic.source; + if (!semantic.ok) { + launch.predictor_error = semantic.error.empty() + ? "predictor_unavailable" : semantic.error; + } else { + ToolSpeculationPrediction prediction; + std::string error; + const json arguments = json::parse(semantic.call.arguments.dump()); + if (!build_tool_speculation_prediction( + semantic.call.name, arguments, + predictor.execution_confidence, prediction, error)) { + launch.predictor_error = std::move(error); + } else { + auto attempt = ToolSpeculationAttempt::create( + config_.tool_speculation, prediction, req.response_id); + launch.attempt = std::shared_ptr( + std::move(attempt)); + launch.attempt->start(); + } + } + } catch (const std::exception & error) { + launch.predictor_error = + std::string("automatic_prediction_failed: ") + error.what(); + } catch (...) { + launch.predictor_error = + "automatic_prediction_failed: unknown error"; + } + log_completion(); +} + void HttpServer::enqueue_request_and_wait(SocketHandle fd, ParsedRequest req) { // Set socket non-blocking for send() stall detection during streaming. sock_set_nonblock(fd); @@ -2050,6 +2664,18 @@ bool HttpServer::route_request(SocketHandle fd, const HttpRequest & hr) { if (!render_and_tokenize_request(fd, render_messages, req)) return true; + if (config_.semantic_tool_predictor.enabled() && + config_.tool_speculation.enabled() && + config_.pflash_upstream_base.empty() && + req.automatic_tool_speculation_enabled && + !req.tool_speculation.has_value() && + !http_detail::tool_choice_disables_tool_calls(req.tool_choice) && + !req.tools.empty()) { + req.predictor_messages = + http_detail::canonical_predictor_messages( + std::move(render_messages)); + } + // count_tokens: short-circuit after tokenization. Skip generation // entirely — Anthropic's contract is just {"input_tokens": N}. if (count_tokens_only) { @@ -3460,6 +4086,9 @@ void HttpServer::prepare_generation_inputs( inputs.request.n_gen = inputs.generation_cap; inputs.request.sampler = req.sampler; inputs.request.do_sample = req.sampler.needs_logit_processing(); + // External tool prediction is deliberately absent from GenerateRequest. + // It must not change the target decoder, including the backend's normal + // empty-speculation recovery policy. // Tokens are delivered through DaemonIO so all API formats share the // same disconnect and streaming state machine. inputs.request.stream = false; @@ -3647,7 +4276,7 @@ void HttpServer::worker_loop() { void HttpServer::process_job(ServerJob * job) { SocketHandle fd = job->fd; - const auto & req = job->req; + auto & req = job->req; auto started_at = std::chrono::steady_clock::now(); // Track live status for /status page. RAII guard ensures idle on all paths. @@ -3753,6 +4382,19 @@ void HttpServer::process_job(ServerJob * job) { return; } + // The only qualified schedule predicts before target compute. This keeps + // shared-GPU deployments deterministic; the admitted external tool then + // overlaps target generation. + start_automatic_tool_speculation(req); + + std::unique_ptr tool_speculation; + if (req.tool_speculation.has_value()) { + tool_speculation = ToolSpeculationAttempt::create( + config_.tool_speculation, *req.tool_speculation, + req.response_id); + tool_speculation->start(); + } + auto & effective_prompt = prepared.tokens; const bool pflash_compressed = prepared.compressed; @@ -3882,8 +4524,56 @@ void HttpServer::process_job(ServerJob * job) { if (job->client_disconnected.load(std::memory_order_acquire)) { client_disconnected = true; } + auto finish_tool_speculation = [ + &](const char * cancel_reason) -> std::optional { + if (tool_speculation) { + json metadata = cancel_reason + ? tool_speculation->cancel(cancel_reason) + : tool_speculation->resolve(emitter.tool_calls()); + metadata["prediction_source"] = "client"; + return metadata; + } + if (!req.automatic_tool_speculation.has_value()) return std::nullopt; + const ParsedRequest::AutomaticToolSpeculationLaunch & launch = + *req.automatic_tool_speculation; + json metadata; + if (launch.attempt) { + metadata = cancel_reason + ? launch.attempt->cancel(cancel_reason) + : launch.attempt->resolve(emitter.tool_calls()); + } else { + metadata = { + {"protocol", "dflash.tool-speculation.v1"}, + {"status", "deferred"}, + {"reason", "predictor_unavailable"}, + }; + if (!launch.predictor_error.empty()) { + metadata["detail"] = launch.predictor_error; + } + } + metadata["prediction_source"] = launch.prediction_source.empty() + ? "predictor" : launch.prediction_source; + metadata["predictor_wall_ms"] = launch.predictor_wall_ms; + return metadata; + }; + // A partial tool call from a failed generation is not authoritative. + // Keep its speculative result private just as we do on disconnect. + const char * generation_cancel_reason = + result.ok() ? nullptr : "generation_failed"; if (req.stream && !client_disconnected) { auto final_chunks = emitter.emit_finish(completion_tokens, &gen_timings); + if (auto metadata = finish_tool_speculation( + generation_cancel_reason)) { + const std::string extension = render_tool_speculation_sse( + req.format, req.response_id, req.model, *metadata); + // Keep the standard terminal event last: [DONE], message_stop, + // or response.completed. Only opted-in clients see this extension. + if (final_chunks.empty()) { + final_chunks.push_back(extension); + } else { + final_chunks.insert(final_chunks.end() - 1, extension); + } + } for (const auto & chunk : final_chunks) { if (!send_job_bytes(job, chunk.data(), chunk.size())) { client_disconnected = true; @@ -3891,13 +4581,19 @@ void HttpServer::process_job(ServerJob * job) { } } } else if (!req.stream && !client_disconnected) { - const json response = build_non_streaming_response( + json response = build_non_streaming_response( req, result, n_gen_cap, gen_timings, tokenizer_, emitter); + if (auto metadata = finish_tool_speculation( + generation_cancel_reason)) { + response["dflash_tool_speculation"] = std::move(*metadata); + } // Streaming uses non-blocking sends; restore blocking mode before // writing a complete JSON response on this shared socket path. sock_set_block(fd); send_response(fd, 200, "application/json", response.dump() + "\n"); + } else { + finish_tool_speculation("client_disconnected"); } if (client_disconnected) { diff --git a/server/src/server/http_server.h b/server/src/server/http_server.h index 90e02412e..900c7757e 100644 --- a/server/src/server/http_server.h +++ b/server/src/server/http_server.h @@ -19,6 +19,9 @@ #include "tokenizer.h" #include "chat_template.h" #include "tool_memory.h" +#include "tool_speculation.h" +#include "semantic_tool_hint.h" +#include "native_semantic_tool_predictor.h" #include "prefix_cache.h" #include "disk_prefix_cache.h" #include "freeze_history.h" @@ -40,6 +43,7 @@ #include #include #include +#include #include #include #if !defined(_WIN32) @@ -232,6 +236,15 @@ struct ServerConfig { // Routing data collection (--collect-routing ): write binary per-token // routing data (hidden states + expert selections) for predictor training. std::string collect_routing_path; + + // Lossless external-tool speculation. Off unless the operator configures + // an executor, an empirical interference profile, and an explicit + // read-only/idempotent tool allowlist. + ToolSpeculationConfig tool_speculation; + // Model-agnostic tool-call prediction. Predictors run before the target; + // only the allowlisted external tool overlaps target generation. This is + // the qualified schedule on shared and single-accelerator deployments. + SemanticToolPredictorConfig semantic_tool_predictor; }; namespace http_detail { @@ -248,6 +261,14 @@ float resolve_pflash_keep_ratio(float configured_ratio, const HttpServerSessions & sessions); bool should_clamp_flowkv_disk_cache( bool flowkv, const DiskPrefixCachePolicy & policy); +// True for API dialects that explicitly prohibit a tool call. Used before +// either caller-supplied or automatic speculative execution can start. +bool tool_choice_disables_tool_calls(const json & tool_choice); +// Convert the exact normalized dialogue rendered for the target into the +// OpenAI-compatible message shape consumed by semantic predictors. The input +// is taken by value so callers can move the rendered message storage into the +// predictor payload without another full content copy. +json canonical_predictor_messages(std::vector messages); } // namespace http_detail @@ -266,8 +287,28 @@ struct ParsedRequest { json tool_choice; // Original messages (for response formatting) json messages; + // Canonical dialogue used by the semantic predictor. Responses API + // function_call/function_call_output items are represented here as the + // assistant/tool turns seen by the target instead of disappearing during + // predictor prompt construction. + json predictor_messages; // Original request body (for upstream proxy forwarding) json raw_body; + // Concrete invocation predicted by a caller or future semantic sidecar. + // The engine may execute it privately, but never exposes its result until + // the model emits the exact canonical invocation. + std::optional tool_speculation; + // Engine-side prediction may start a private external tool before target + // generation. The model's eventual canonical call remains authoritative. + struct AutomaticToolSpeculationLaunch { + std::shared_ptr attempt; + double predictor_wall_ms = 0.0; + std::string prediction_source; + std::string predictor_error; + }; + bool automatic_tool_speculation_enabled = true; + std::optional + automatic_tool_speculation; // Response ID std::string response_id; // Thinking/reasoning state @@ -335,6 +376,10 @@ class HttpServer { // Set the chat template format (detected from model arch). void set_chat_format(ChatFormat fmt) { chat_format_ = fmt; } + // Start the optional native Qwen predictor after target construction. + // HTTP-only predictor configurations need no persistent initialization. + bool init_semantic_tool_predictor(std::string & error); + // Start listening. Blocks until shutdown() is called. int run(); @@ -483,6 +528,7 @@ class HttpServer { ParsedRequest & req); bool validate_request_context(SocketHandle fd, const ParsedRequest & req); void log_parsed_request(const ParsedRequest & req) const; + void start_automatic_tool_speculation(ParsedRequest & req) const; void enqueue_request_and_wait(SocketHandle fd, ParsedRequest req); // Send HTTP response helpers. @@ -510,6 +556,7 @@ class HttpServer { ServerConfig config_; ChatFormat chat_format_; PFlashDrafterIpcClient pflash_remote_; + std::shared_ptr native_semantic_predictor_; ToolMemory tool_memory_; PrefixCache prefix_cache_; DiskPrefixCache disk_cache_; diff --git a/server/src/server/native_semantic_tool_predictor.cpp b/server/src/server/native_semantic_tool_predictor.cpp new file mode 100644 index 000000000..3e7f55085 --- /dev/null +++ b/server/src/server/native_semantic_tool_predictor.cpp @@ -0,0 +1,112 @@ +#include "native_semantic_tool_predictor.h" + +#include +#include +#include +#include + +namespace dflash::common { + +namespace { +constexpr int kNativePredictorStartupTimeoutMs = 60000; +} + +std::shared_ptr +NativeSemanticToolPredictor::create( + const SemanticToolPredictorConfig & config, + std::string & error) { + error.clear(); + if (!config.native_enabled()) { + error = "native_predictor_config_incomplete"; + return nullptr; + } + auto predictor = std::shared_ptr( + new NativeSemanticToolPredictor(config)); + if (!predictor->tokenizer_.load_from_gguf( + config.native_model_path.c_str())) { + error = "native_predictor_tokenizer_load_failed"; + return nullptr; + } + if (!predictor->ipc_.start( + config.native_ipc_bin, config.native_model_path, + config.native_gpu, config.native_max_ctx, + config.native_work_dir, kNativePredictorStartupTimeoutMs)) { + error = "native_predictor_ipc_start_failed"; + return nullptr; + } + return predictor; +} + +SemanticToolPrediction NativeSemanticToolPredictor::predict( + const SemanticToolPredictorRequest & predictor_request, + const json & request_tools, + std::string * generated_text) { + const auto started = std::chrono::steady_clock::now(); + const auto deadline = started + + std::chrono::milliseconds(std::max(1, config_.timeout_ms)); + SemanticToolPrediction prediction; + prediction.source = "native-qwen3"; + auto finish = [&]() { + prediction.wall_ms = std::chrono::duration( + std::chrono::steady_clock::now() - started).count(); + return prediction; + }; + + std::string prompt_error; + const std::string prompt = build_native_semantic_tool_predictor_prompt( + predictor_request, prompt_error, &deadline); + if (prompt.empty()) { + prediction.error = std::move(prompt_error); + return finish(); + } + if (std::chrono::steady_clock::now() >= deadline) { + prediction.error = "native_predictor_timeout"; + return finish(); + } + std::vector prompt_ids; + if (!tokenizer_.encode_until(prompt, deadline, prompt_ids)) { + prediction.error = "native_predictor_timeout"; + return finish(); + } + if (prompt_ids.empty()) { + prediction.error = "native_predictor_prompt_tokenization_failed"; + return finish(); + } + if (prompt_ids.size() + static_cast(config_.max_tokens) > + static_cast(config_.native_max_ctx)) { + prediction.error = "native_predictor_context_overflow"; + return finish(); + } + + const auto now = std::chrono::steady_clock::now(); + if (now >= deadline) { + prediction.error = "native_predictor_timeout"; + return finish(); + } + const int remaining_ms = std::max(1, static_cast( + std::chrono::duration_cast( + deadline - now).count())); + std::vector output_ids; + if (!ipc_.predict(prompt_ids, config_.max_tokens, remaining_ms, + output_ids, prediction.error)) { + return finish(); + } + if (std::chrono::steady_clock::now() >= deadline) { + prediction.error = "native_predictor_timeout"; + return finish(); + } + const std::string generated = tokenizer_.decode(output_ids); + if (generated_text) *generated_text = generated; + if (!parse_native_semantic_tool_prediction( + generated, request_tools, prediction.call, prediction.error)) { + return finish(); + } + if (!materialize_declared_tool_defaults( + request_tools, prediction.call, prediction.error)) { + return finish(); + } + prediction.ok = true; + return finish(); +} + +} // namespace dflash::common diff --git a/server/src/server/native_semantic_tool_predictor.h b/server/src/server/native_semantic_tool_predictor.h new file mode 100644 index 000000000..125ac38cd --- /dev/null +++ b/server/src/server/native_semantic_tool_predictor.h @@ -0,0 +1,41 @@ +// Native Qwen semantic tool predictor built from the PFlash/Qwen runtime. + +#pragma once + +#include "semantic_tool_hint.h" + +#include "common/qwen3_tool_predictor_ipc.h" +#include "tokenizer.h" + +#include +#include + +namespace dflash::common { + +class NativeSemanticToolPredictor { +public: + static std::shared_ptr create( + const SemanticToolPredictorConfig & config, + std::string & error); + + NativeSemanticToolPredictor(const NativeSemanticToolPredictor &) = delete; + NativeSemanticToolPredictor & operator=( + const NativeSemanticToolPredictor &) = delete; + + SemanticToolPrediction predict( + const SemanticToolPredictorRequest & predictor_request, + const json & request_tools, + std::string * generated_text = nullptr); + + bool active() const { return ipc_.active(); } + +private: + explicit NativeSemanticToolPredictor( + const SemanticToolPredictorConfig & config) : config_(config) {} + + SemanticToolPredictorConfig config_; + Tokenizer tokenizer_; + Qwen3ToolPredictorIpcClient ipc_; +}; + +} // namespace dflash::common diff --git a/server/src/server/semantic_tool_hint.cpp b/server/src/server/semantic_tool_hint.cpp new file mode 100644 index 000000000..43c9c7036 --- /dev/null +++ b/server/src/server/semantic_tool_hint.cpp @@ -0,0 +1,691 @@ +#include "semantic_tool_hint.h" + +#include "tool_parser.h" + +#include +#include +#include +#include + +namespace dflash::common { + +namespace { + +std::string string_member(const json & object, const char * key) { + if (!object.is_object()) return {}; + const auto member = object.find(key); + return member != object.end() && member->is_string() + ? member->get() : std::string{}; +} + +bool request_has_function(const json & tools, const std::string & name) { + if (!tools.is_array() || name.empty()) return false; + for (const auto & tool : tools) { + if (!tool.is_object()) continue; + if (string_member(tool, "name") == name) return true; + const auto function = tool.find("function"); + if (function != tool.end() && function->is_object() && + string_member(*function, "name") == name) { + return true; + } + } + return false; +} + +std::string sole_request_function(const json & tools) { + if (!tools.is_array()) return {}; + std::string sole; + for (const auto & tool : tools) { + if (!tool.is_object()) continue; + std::string name = string_member(tool, "name"); + const auto function = tool.find("function"); + if (name.empty() && function != tool.end() && function->is_object()) { + name = string_member(*function, "name"); + } + if (name.empty()) continue; + if (!sole.empty() && sole != name) return {}; + sole = std::move(name); + } + return sole; +} + +bool parse_arguments(const json & value, ordered_json & out) { + try { + if (value.is_string()) { + out = ordered_json::parse(value.get()); + } else if (value.is_object()) { + out = ordered_json::parse(value.dump()); + } else { + return false; + } + } catch (...) { + return false; + } + return out.is_object(); +} + +bool parse_call_object(const json & value, SemanticToolCall & out) { + if (!value.is_object()) return false; + std::string name = string_member(value, "name"); + if (name.empty()) name = string_member(value, "function"); + if (name.empty()) return false; + + const json * arguments = nullptr; + for (const char * key : {"arguments", "parameters", "params"}) { + const auto it = value.find(key); + if (it != value.end()) { + arguments = &*it; + break; + } + } + ordered_json parsed; + if (!arguments || !parse_arguments(*arguments, parsed)) return false; + out.name = name; + out.arguments = std::move(parsed); + return true; +} + +std::string trim_copy(std::string value) { + const auto is_space = [](unsigned char ch) { return std::isspace(ch); }; + value.erase(value.begin(), std::find_if_not( + value.begin(), value.end(), is_space)); + value.erase(std::find_if_not(value.rbegin(), value.rend(), is_space).base(), + value.end()); + return value; +} + +bool parse_content_call(const std::string & content, SemanticToolCall & out) { + const json value = json::parse(trim_copy(content), nullptr, false); + return !value.is_discarded() && parse_call_object(value, out); +} + +bool parse_qwen_tagged_call_repair( + const std::string & generated_text, + SemanticToolCall & out) { + const std::string open = ""; + const size_t open_pos = generated_text.find(open); + if (open_pos == std::string::npos) return false; + const size_t content_pos = open_pos + open.size(); + size_t end_pos = generated_text.find("", content_pos); + if (end_pos == std::string::npos) { + end_pos = generated_text.find("<|im_end|>", content_pos); + } + if (end_pos == std::string::npos) end_pos = generated_text.size(); + + std::string payload = trim_copy( + generated_text.substr(content_pos, end_pos - content_pos)); + if (payload.empty()) return false; + + // Qwen3-0.6B Q8 occasionally drops only the outer opening brace and + // emits a stray quote before the matching closing brace, while keeping + // the name and argument object strict JSON. Repair only that narrow + // envelope error; all semantic fields still pass the normal schema gate. + if (payload.front() != '{') payload.insert(payload.begin(), '{'); + if (payload.size() >= 3 && payload.back() == '}') { + size_t quote = payload.size() - 2; + while (quote > 0 && std::isspace( + static_cast(payload[quote]))) { + --quote; + } + if (payload[quote] == '"') payload.erase(quote, 1); + } + const json value = json::parse(payload, nullptr, false); + return !value.is_discarded() && parse_call_object(value, out); +} + +bool parse_qwen_bare_single_tool_arguments( + const std::string & generated_text, + const json & request_tools, + SemanticToolCall & out) { + const std::string name = sole_request_function(request_tools); + if (name.empty()) return false; + + size_t begin = generated_text.find(""); + begin = begin == std::string::npos + ? 0 : begin + std::string("").size(); + begin = generated_text.find('{', begin); + if (begin == std::string::npos) return false; + size_t end = generated_text.rfind('}'); + if (end == std::string::npos || end < begin) return false; + + const json arguments = json::parse( + generated_text.begin() + static_cast(begin), + generated_text.begin() + static_cast(end + 1), + nullptr, false); + if (arguments.is_discarded() || !arguments.is_object()) return false; + out.name = name; + out.arguments = ordered_json::parse(arguments.dump()); + return true; +} + +bool semantic_deadline_expired( + const std::chrono::steady_clock::time_point * deadline) { + return deadline && std::chrono::steady_clock::now() >= *deadline; +} + +constexpr size_t kMaxNativePredictorRequestBytes = 256U * 1024U; +constexpr size_t kMaxNativePredictorJsonDepth = 64U; + +bool semantic_take_budget(size_t bytes, size_t & remaining) { + if (bytes > remaining) return false; + remaining -= bytes; + return true; +} + +bool semantic_string_within_budget( + const std::string & value, + const std::chrono::steady_clock::time_point * deadline, + size_t & remaining, + size_t & operations, + bool & timed_out) { + if (!semantic_take_budget(2, remaining)) return false; + for (const unsigned char character : value) { + if ((operations++ & 255U) == 0U && + semantic_deadline_expired(deadline)) { + timed_out = true; + return false; + } + const size_t encoded_bytes = character < 0x20U + ? 6U : (character == '"' || character == '\\' ? 2U : 1U); + if (!semantic_take_budget(encoded_bytes, remaining)) return false; + } + return true; +} + +bool semantic_json_within_budget( + const json & value, + const std::chrono::steady_clock::time_point * deadline, + size_t & remaining, + size_t depth, + size_t & operations, + bool & timed_out) { + if (depth > kMaxNativePredictorJsonDepth) return false; + if ((operations++ & 255U) == 0U && + semantic_deadline_expired(deadline)) { + timed_out = true; + return false; + } + if (value.is_null()) return semantic_take_budget(4, remaining); + if (value.is_boolean()) return semantic_take_budget(5, remaining); + if (value.is_number()) return semantic_take_budget(64, remaining); + if (value.is_string()) { + return semantic_string_within_budget( + value.get_ref(), deadline, + remaining, operations, timed_out); + } + if (value.is_array()) { + if (!semantic_take_budget(2, remaining)) return false; + bool first = true; + for (const auto & element : value) { + if (!first && !semantic_take_budget(1, remaining)) return false; + first = false; + if (!semantic_json_within_budget( + element, deadline, remaining, depth + 1, + operations, timed_out)) { + return false; + } + } + return true; + } + if (value.is_object()) { + if (!semantic_take_budget(2, remaining)) return false; + bool first = true; + for (const auto & item : value.items()) { + if (!first && !semantic_take_budget(1, remaining)) return false; + first = false; + if (!semantic_string_within_budget( + item.key(), deadline, remaining, operations, timed_out) || + !semantic_take_budget(1, remaining) || + !semantic_json_within_budget( + item.value(), deadline, remaining, depth + 1, + operations, timed_out)) { + return false; + } + } + return true; + } + return false; +} + +bool semantic_message_content( + const json & message, + const std::chrono::steady_clock::time_point * deadline, + std::string & text) { + text.clear(); + const auto content = message.find("content"); + if (content == message.end() || content->is_null()) return true; + if (content->is_string()) { + text = content->get(); + return !semantic_deadline_expired(deadline); + } + if (!content->is_array()) { + text = content->dump(); + return !semantic_deadline_expired(deadline); + } + + for (const auto & part : *content) { + if (semantic_deadline_expired(deadline)) return false; + if (part.is_string()) { + text += part.get(); + continue; + } + if (!part.is_object()) continue; + const std::string type = string_member(part, "type"); + if (type == "text" || type == "input_text" || + type == "output_text") { + text += string_member(part, "text"); + } + } + return !semantic_deadline_expired(deadline); +} + +std::string forced_tool_name(const json & choice) { + if (!choice.is_object()) return {}; + const auto function = choice.find("function"); + if (function != choice.end() && function->is_object()) { + return string_member(*function, "name"); + } + return string_member(choice, "name"); +} + +json canonical_semantic_tools(const json & tools) { + json canonical = json::array(); + if (!tools.is_array()) return canonical; + for (const auto & tool : tools) { + if (!tool.is_object()) continue; + const json * source = &tool; + const auto wrapped = tool.find("function"); + if (wrapped != tool.end() && wrapped->is_object()) source = &*wrapped; + const std::string name = string_member(*source, "name"); + if (name.empty()) continue; + + json function = {{"name", name}}; + const auto description = source->find("description"); + if (description != source->end() && description->is_string()) { + function["description"] = *description; + } + for (const char * schema_key : {"parameters", "input_schema"}) { + const auto schema = source->find(schema_key); + if (schema != source->end() && schema->is_object()) { + function["parameters"] = *schema; + break; + } + } + const auto strict = source->find("strict"); + if (strict != source->end() && strict->is_boolean()) { + function["strict"] = *strict; + } + canonical.push_back({ + {"type", "function"}, + {"function", std::move(function)}, + }); + } + return canonical; +} + +} // namespace + +bool parse_semantic_tool_prediction( + const json & response, + const json & request_tools, + SemanticToolCall & out, + std::string & error) { + error.clear(); + const auto choices = response.find("choices"); + if (choices == response.end() || !choices->is_array() || + choices->size() != 1 || !(*choices)[0].is_object()) { + error = "predictor_response_missing_single_choice"; + return false; + } + const auto message = (*choices)[0].find("message"); + if (message == (*choices)[0].end() || !message->is_object()) { + error = "predictor_response_missing_message"; + return false; + } + + bool parsed = false; + const auto calls = message->find("tool_calls"); + if (calls != message->end()) { + if (!calls->is_array() || calls->size() != 1 || + !(*calls)[0].is_object()) { + error = "predictor_response_requires_single_tool_call"; + return false; + } + const auto function = (*calls)[0].find("function"); + if (function != (*calls)[0].end()) { + parsed = parse_call_object(*function, out); + } + } else { + const auto content = message->find("content"); + if (content != message->end() && content->is_string()) { + parsed = parse_content_call(content->get(), out); + } + } + if (!parsed) { + error = "predictor_response_has_no_valid_call"; + return false; + } + if (!request_has_function(request_tools, out.name)) { + error = "predictor_selected_unknown_function"; + return false; + } + return true; +} + +bool materialize_declared_tool_defaults( + const json & request_tools, + SemanticToolCall & call, + std::string & error) { + error.clear(); + if (!call.arguments.is_object()) { + error = "predictor_arguments_not_object"; + return false; + } + if (!request_tools.is_array()) { + error = "predictor_tools_not_array"; + return false; + } + + const json * function = nullptr; + for (const auto & tool : request_tools) { + if (!tool.is_object()) continue; + const json & candidate = tool.contains("function") && + tool["function"].is_object() + ? tool["function"] : tool; + if (string_member(candidate, "name") == call.name) { + function = &candidate; + break; + } + } + if (!function) { + error = "predictor_selected_unknown_function"; + return false; + } + + const json * parameters = nullptr; + for (const char * key : {"parameters", "input_schema"}) { + const auto found = function->find(key); + if (found != function->end() && found->is_object()) { + parameters = &*found; + break; + } + } + if (!parameters) return true; + const auto properties = parameters->find("properties"); + if (properties == parameters->end() || !properties->is_object()) { + return true; + } + for (const auto & property : properties->items()) { + if (call.arguments.contains(property.key()) || + !property.value().is_object() || + !property.value().contains("default")) { + continue; + } + call.arguments[property.key()] = property.value()["default"]; + } + return true; +} + +std::optional +build_semantic_tool_predictor_request( + const json & messages, + const json & tools, + const json & tool_choice, + const std::string & sidecar_model, + int max_tokens, + std::string & error) { + error.clear(); + size_t request_budget = kMaxNativePredictorRequestBytes; + size_t budget_operations = 0; + bool budget_timed_out = false; + const json default_choice = "auto"; + const json & effective_choice = tool_choice.is_null() + ? default_choice : tool_choice; + if (!semantic_take_budget(512, request_budget) || + !semantic_string_within_budget( + sidecar_model, nullptr, request_budget, + budget_operations, budget_timed_out) || + !semantic_json_within_budget( + messages, nullptr, request_budget, 0, + budget_operations, budget_timed_out) || + !semantic_json_within_budget( + tools, nullptr, request_budget, 0, + budget_operations, budget_timed_out) || + !semantic_json_within_budget( + effective_choice, nullptr, request_budget, 0, + budget_operations, budget_timed_out)) { + error = "predictor_request_too_large"; + return std::nullopt; + } + json canonical_tools = canonical_semantic_tools(tools); + if (canonical_tools.empty()) { + error = "predictor_request_has_no_valid_tools"; + return std::nullopt; + } + json request = { + {"model", sidecar_model}, + {"stream", false}, + {"temperature", 0}, + {"max_tokens", max_tokens}, + {"messages", messages}, + {"tools", std::move(canonical_tools)}, + {"tool_choice", effective_choice}, + }; + // Tool-schema normalization can add OpenAI wrapper objects. Validate the + // canonical result once before granting the bounded-request type; native + // rendering then relies on that type instead of scanning the same payload + // for a third time. + size_t canonical_budget = kMaxNativePredictorRequestBytes; + size_t canonical_operations = 0; + bool canonical_timed_out = false; + if (!semantic_json_within_budget( + request, nullptr, canonical_budget, 0, + canonical_operations, canonical_timed_out)) { + error = "predictor_request_too_large"; + return std::nullopt; + } + return SemanticToolPredictorRequest(std::move(request)); +} + +std::string build_native_semantic_tool_predictor_prompt( + const SemanticToolPredictorRequest & bounded_request, + std::string & error, + const std::chrono::steady_clock::time_point * deadline) { + error.clear(); + if (semantic_deadline_expired(deadline)) { + error = "native_predictor_timeout"; + return {}; + } + const json & predictor_request = bounded_request.payload(); + const json choice = predictor_request.value("tool_choice", json("auto")); + if ((choice.is_string() && choice.get() == "none") || + (choice.is_object() && string_member(choice, "type") == "none")) { + error = "native_predictor_tool_choice_none"; + return {}; + } + const auto messages = predictor_request.find("messages"); + if (messages == predictor_request.end() || !messages->is_array() || + messages->empty()) { + error = "native_predictor_missing_messages"; + return {}; + } + const json tools = predictor_request.value("tools", json::array()); + if (!tools.is_array() || tools.empty()) { + error = "native_predictor_missing_tools"; + return {}; + } + + struct PredictorMessage { + std::string role; + std::string content; + json tool_calls; + }; + std::vector chat; + chat.reserve(messages->size()); + for (const auto & message : *messages) { + if (semantic_deadline_expired(deadline)) { + error = "native_predictor_timeout"; + return {}; + } + if (!message.is_object()) continue; + std::string role = string_member(message, "role"); + if (role.empty()) role = "user"; + if (role == "developer") role = "system"; + std::string content; + if (!semantic_message_content(message, deadline, content)) { + error = "native_predictor_timeout"; + return {}; + } + chat.push_back({ + std::move(role), std::move(content), + message.value("tool_calls", json::array()), + }); + } + if (chat.empty()) { + error = "native_predictor_empty_messages"; + return {}; + } + + std::string constraint; + if (choice.is_string() && choice.get() == "required") { + constraint = "You must call exactly one available function."; + } else if (const std::string name = forced_tool_name(choice); + !name.empty()) { + constraint = "You must call the function " + name + "."; + } + // Render the exact tokenizer.chat_template contract embedded in the + // Qwen3-0.6B GGUF. PFlash's generic Qwen3.5 renderer uses parameter XML, + // while this model was trained to emit one JSON object inside + // ; using the wrong contract destroys multi-tool accuracy. + size_t begin = 0; + std::string system_content; + if (!chat.empty() && chat.front().role == "system") { + system_content = chat.front().content; + begin = 1; + } + if (!constraint.empty()) { + if (!system_content.empty()) system_content += "\n\n"; + system_content += constraint; + } + + std::string rendered = "<|im_start|>system\n"; + if (!system_content.empty()) { + rendered += system_content; + rendered += "\n\n"; + } + rendered += + "# Tools\n\n" + "You may call one or more functions to assist with the user query.\n\n" + "You are provided with function signatures within XML tags:\n" + ""; + for (const auto & tool : tools) { + if (semantic_deadline_expired(deadline)) { + error = "native_predictor_timeout"; + return {}; + } + rendered += tool.dump(); + } + rendered += + "\n\n\n" + "For each function call, return a json object with function name and " + "arguments within XML tags:\n" + "\n" + "{\"name\": , \"arguments\": }\n" + "<|im_end|>\n"; + + bool in_tool_response = false; + for (size_t index = begin; index < chat.size(); ++index) { + if (semantic_deadline_expired(deadline)) { + error = "native_predictor_timeout"; + return {}; + } + const auto & message = chat[index]; + if (message.role == "tool") { + if (!in_tool_response) { + rendered += "<|im_start|>user"; + in_tool_response = true; + } + rendered += "\n\n" + message.content + + "\n"; + const bool next_is_tool = index + 1 < chat.size() && + chat[index + 1].role == "tool"; + if (!next_is_tool) { + rendered += "<|im_end|>\n"; + in_tool_response = false; + } + continue; + } + + rendered += "<|im_start|>" + message.role + "\n" + message.content; + if (message.role == "assistant" && message.tool_calls.is_array()) { + for (const auto & raw_call : message.tool_calls) { + if (semantic_deadline_expired(deadline)) { + error = "native_predictor_timeout"; + return {}; + } + if (!raw_call.is_object()) continue; + const json & call = raw_call.contains("function") && + raw_call["function"].is_object() + ? raw_call["function"] : raw_call; + const std::string name = string_member(call, "name"); + if (name.empty() || !call.contains("arguments")) continue; + if (!message.content.empty()) rendered += "\n"; + rendered += "\n{\"name\": \"" + name + + "\", \"arguments\": "; + rendered += call["arguments"].is_string() + ? call["arguments"].get() + : call["arguments"].dump(); + rendered += "}\n"; + } + } + rendered += "<|im_end|>\n"; + } + rendered += "<|im_start|>assistant\n\n\n\n\n"; + if (semantic_deadline_expired(deadline)) { + error = "native_predictor_timeout"; + return {}; + } + return rendered; +} + +bool parse_native_semantic_tool_prediction( + const std::string & generated_text, + const json & request_tools, + SemanticToolCall & out, + std::string & error) { + error.clear(); + const ToolParseResult parsed = parse_tool_calls( + generated_text, request_tools); + if (parsed.tool_calls.size() == 1) { + try { + ordered_json arguments = ordered_json::parse( + parsed.tool_calls.front().arguments); + if (!arguments.is_object()) { + error = "native_predictor_arguments_not_object"; + return false; + } + out.name = parsed.tool_calls.front().name; + out.arguments = std::move(arguments); + } catch (...) { + error = "native_predictor_arguments_invalid_json"; + return false; + } + } else if (parsed.tool_calls.empty()) { + if (!parse_qwen_tagged_call_repair(generated_text, out) && + !parse_qwen_bare_single_tool_arguments( + generated_text, request_tools, out)) { + error = "native_predictor_response_has_no_valid_call"; + return false; + } + } else { + error = "native_predictor_response_has_multiple_calls"; + return false; + } + if (!request_has_function(request_tools, out.name)) { + error = "predictor_selected_unknown_function"; + return false; + } + return true; +} + +} // namespace dflash::common diff --git a/server/src/server/semantic_tool_hint.h b/server/src/server/semantic_tool_hint.h new file mode 100644 index 000000000..418b3d96d --- /dev/null +++ b/server/src/server/semantic_tool_hint.h @@ -0,0 +1,115 @@ +// Model-agnostic tool-call predictions shared by HTTP and native predictors. + +#pragma once + +#include + +#include +#include +#include +#include + +namespace dflash::common { + +using json = nlohmann::json; +using ordered_json = nlohmann::ordered_json; + +struct SemanticToolPredictorConfig { + std::string url; + std::string model; + std::string native_model_path; + std::string native_ipc_bin; + std::string native_work_dir; + int native_gpu = 0; + int native_max_ctx = 4096; + int timeout_ms = 2000; + int max_tokens = 96; + // Conservative prior used by the measured tool-execution admission + // policy. The base predictor currently emits no calibrated probability. + double execution_confidence = 0.75; + + bool http_enabled() const { return !url.empty() && !model.empty(); } + bool native_enabled() const { + return !native_model_path.empty() && !native_ipc_bin.empty(); + } + bool enabled() const { return native_enabled() || http_enabled(); } +}; + +struct SemanticToolCall { + std::string name; + ordered_json arguments = ordered_json::object(); +}; + +struct SemanticToolPrediction { + bool ok = false; + std::string error; + // Actual predictor used for this result. Native and HTTP fallback paths + // share one execution gate, so response metadata must not guess. + std::string source; + SemanticToolCall call; + double wall_ms = 0.0; +}; + +// A predictor payload that has passed the recursive byte/depth admission gate. +// Keeping construction private makes the native prompt renderer's bounded-input +// invariant explicit and prevents accidental duplicate validation work. +class SemanticToolPredictorRequest { +public: + const json & payload() const { return payload_; } + +private: + explicit SemanticToolPredictorRequest(json payload) + : payload_(std::move(payload)) {} + + friend std::optional + build_semantic_tool_predictor_request( + const json &, const json &, const json &, const std::string &, int, + std::string &); + + json payload_; +}; + +// Parse one OpenAI-compatible sidecar response and reject calls whose +// function name is absent from the request schema. Arguments remain decoded +// JSON values; sidecar token IDs are never accepted by the target. +bool parse_semantic_tool_prediction( + const json & response, + const json & request_tools, + SemanticToolCall & out, + std::string & error); + +// Materialize top-level defaults declared by the selected function before a +// prediction is executed. This turns an omitted optional default into the +// exact explicit invocation the target may emit; the normal exact-match gate +// still rejects the result if the authoritative call differs. +bool materialize_declared_tool_defaults( + const json & request_tools, + SemanticToolCall & call, + std::string & error); + +// Build the bounded OpenAI-compatible request sent to the predictor. Only +// normalized dialogue/tool semantics are copied; target-only extensions are +// omitted. Oversized inputs fail before any full-field copy. +std::optional +build_semantic_tool_predictor_request( + const json & messages, + const json & tools, + const json & tool_choice, + const std::string & sidecar_model, + int max_tokens, + std::string & error); + +// Native predictor bridge. The prompt uses the Qwen tool template and the +// decoded response is parsed semantically before any target token IDs exist. +std::string build_native_semantic_tool_predictor_prompt( + const SemanticToolPredictorRequest & predictor_request, + std::string & error, + const std::chrono::steady_clock::time_point * deadline = nullptr); + +bool parse_native_semantic_tool_prediction( + const std::string & generated_text, + const json & request_tools, + SemanticToolCall & out, + std::string & error); + +} // namespace dflash::common diff --git a/server/src/server/server_main.cpp b/server/src/server/server_main.cpp index 7373aace4..af126bbe2 100644 --- a/server/src/server/server_main.cpp +++ b/server/src/server/server_main.cpp @@ -27,6 +27,8 @@ #include "kvflash_pager.h" #include +#include +#include #include #include #include @@ -37,6 +39,10 @@ #include #include +#if !defined(_WIN32) +#include +#endif + using namespace dflash::common; // Global server pointer for signal handling. @@ -66,6 +72,31 @@ static bool parse_double_list(const char * value, std::vector & out) { return !out.empty(); } +static bool parse_int_strict(const char * value, int & out) { + if (!value || !*value) return false; + const char * end = value + std::strlen(value); + const auto parsed = std::from_chars(value, end, out); + return parsed.ec == std::errc{} && parsed.ptr == end; +} + +static bool parse_double_strict(const char * value, double & out) { + if (!value || !*value) return false; + char * end = nullptr; + errno = 0; + const double parsed = std::strtod(value, &end); + if (errno == ERANGE || end == value || !end || *end != '\0' || + !std::isfinite(parsed)) { + return false; + } + out = parsed; + return true; +} + +static bool environment_flag_enabled(const char * name) { + const char * value = std::getenv(name); + return value && *value && std::strcmp(value, "0") != 0; +} + static void print_usage(const char * prog) { std::fprintf(stderr, "Usage: %s [options]\n" @@ -175,6 +206,45 @@ static void print_usage(const char * prog) { " Drafter lifetime policy (default: auto)\n" " --lazy-draft Legacy alias for --draft-residency=request-scoped\n" "\n" + "Tool-call prediction (lossless exact verification):\n" + " --tool-hint-sidecar-url \n" + " OpenAI-compatible chat-completions endpoint.\n" + " --tool-hint-sidecar-model \n" + " Predictor model served by that endpoint.\n" + " --tool-hint-native-model \n" + " Qwen3-0.6B GGUF for the native PFlash runtime.\n" + " --tool-hint-native-ipc-bin \n" + " Matching backend_ipc_daemon executable.\n" + " --tool-hint-native-gpu Predictor GPU (default: 0).\n" + " --tool-hint-native-max-ctx \n" + " Predictor context capacity (default: 4096).\n" + " --tool-hint-native-work-dir \n" + " Optional private IPC scratch directory.\n" + " --tool-hint-timeout-ms Hard native/HTTP predictor deadline\n" + " (default: 2000).\n" + " --tool-hint-max-tokens Predictor completion cap (default: 96).\n" + " --tool-hint-execution-confidence

\n" + " Calibrated 0..1 prior for automatic external\n" + " tool admission (default: 0.75).\n" + " The target verifies every hint.\n" + "\n" + "Speculative external tools (opt-in, POSIX):\n" + " --tool-spec-executor Trusted executor adapter. Receives one\n" + " dflash.tool-speculation.v1 JSON request\n" + " on stdin; no shell is used.\n" + " --tool-spec-profile Measured resource-lane frontier JSON.\n" + " --tool-spec-allow Allow one read-only/idempotent tool; repeatable.\n" + " --tool-spec-cpu-affinity \n" + " Pin child tools to Linux CPUs/ranges, e.g.\n" + " 14-15,30-31. The model process affinity\n" + " must exclude every listed CPU.\n" + " --tool-spec-timeout-ms Executor result timeout (default: 60000).\n" + " --tool-spec-max-model-slowdown \n" + " Reject lanes slower than this inference\n" + " ratio (default: 1.20).\n" + " Every admitted lane must pass exact-output\n" + " decode-interference qualification.\n" + "\n" "PFlash upstream proxy (forward compressed prompt to a backend):\n" " --prefill-upstream-base OpenAI-compatible upstream. Compressed\n" " requests POST the raw prompt to\n" @@ -552,6 +622,114 @@ int main(int argc, char ** argv) { } else if (std::strcmp(argv[i], "--lazy-draft") == 0) { sconfig.lazy_draft = true; sconfig.draft_residency = DraftResidencyPolicy::RequestScoped; + } else if (std::strcmp(argv[i], "--tool-hint-sidecar-url") == 0 && + i + 1 < argc) { + sconfig.semantic_tool_predictor.url = argv[++i]; + } else if (std::strcmp(argv[i], "--tool-hint-sidecar-model") == 0 && + i + 1 < argc) { + sconfig.semantic_tool_predictor.model = argv[++i]; + } else if (std::strcmp(argv[i], "--tool-hint-native-model") == 0 && + i + 1 < argc) { + sconfig.semantic_tool_predictor.native_model_path = argv[++i]; + } else if (std::strcmp(argv[i], "--tool-hint-native-ipc-bin") == 0 && + i + 1 < argc) { + sconfig.semantic_tool_predictor.native_ipc_bin = argv[++i]; + } else if (std::strcmp(argv[i], "--tool-hint-native-work-dir") == 0 && + i + 1 < argc) { + sconfig.semantic_tool_predictor.native_work_dir = argv[++i]; + } else if (std::strcmp(argv[i], "--tool-hint-native-gpu") == 0 && + i + 1 < argc) { + if (!parse_int_strict( + argv[++i], sconfig.semantic_tool_predictor.native_gpu) || + sconfig.semantic_tool_predictor.native_gpu < 0) { + std::fprintf(stderr, + "[server] --tool-hint-native-gpu must be non-negative\n"); + return 2; + } + } else if (std::strcmp(argv[i], "--tool-hint-native-max-ctx") == 0 && + i + 1 < argc) { + if (!parse_int_strict( + argv[++i], sconfig.semantic_tool_predictor.native_max_ctx) || + sconfig.semantic_tool_predictor.native_max_ctx <= 0) { + std::fprintf(stderr, + "[server] --tool-hint-native-max-ctx must be positive\n"); + return 2; + } + } else if ((std::strcmp(argv[i], "--tool-hint-timeout-ms") == 0 || + std::strcmp(argv[i], "--tool-hint-sidecar-timeout-ms") == 0) && + i + 1 < argc) { + if (!parse_int_strict( + argv[++i], sconfig.semantic_tool_predictor.timeout_ms) || + sconfig.semantic_tool_predictor.timeout_ms <= 0) { + std::fprintf(stderr, + "[server] --tool-hint-timeout-ms must be positive\n"); + return 2; + } + } else if ((std::strcmp(argv[i], "--tool-hint-max-tokens") == 0 || + std::strcmp(argv[i], "--tool-hint-sidecar-max-tokens") == 0) && + i + 1 < argc) { + if (!parse_int_strict( + argv[++i], sconfig.semantic_tool_predictor.max_tokens) || + sconfig.semantic_tool_predictor.max_tokens <= 0) { + std::fprintf(stderr, + "[server] --tool-hint-max-tokens must be positive\n"); + return 2; + } + } else if (std::strcmp( + argv[i], "--tool-hint-execution-confidence") == 0 && + i + 1 < argc) { + if (!parse_double_strict( + argv[++i], + sconfig.semantic_tool_predictor.execution_confidence) || + sconfig.semantic_tool_predictor.execution_confidence < 0.0 || + sconfig.semantic_tool_predictor.execution_confidence > 1.0) { + std::fprintf(stderr, + "[server] --tool-hint-execution-confidence must be between 0 and 1\n"); + return 2; + } + } else if (std::strcmp(argv[i], "--tool-spec-executor") == 0 && + i + 1 < argc) { + sconfig.tool_speculation.executor_path = argv[++i]; + } else if (std::strcmp(argv[i], "--tool-spec-profile") == 0 && + i + 1 < argc) { + sconfig.tool_speculation.profile_path = argv[++i]; + } else if (std::strcmp(argv[i], "--tool-spec-allow") == 0 && + i + 1 < argc) { + const std::string name = argv[++i]; + if (name.empty()) { + std::fprintf(stderr, "[server] --tool-spec-allow needs a name\n"); + return 2; + } + sconfig.tool_speculation.allowed_tools.push_back(name); + } else if (std::strcmp(argv[i], "--tool-spec-cpu-affinity") == 0 && + i + 1 < argc) { + std::string affinity_error; + if (!parse_tool_speculation_cpu_affinity( + argv[++i], sconfig.tool_speculation.cpu_affinity, + affinity_error)) { + std::fprintf(stderr, "[server] %s\n", affinity_error.c_str()); + return 2; + } + } else if (std::strcmp(argv[i], "--tool-spec-timeout-ms") == 0 && + i + 1 < argc) { + if (!parse_int_strict( + argv[++i], sconfig.tool_speculation.timeout_ms) || + sconfig.tool_speculation.timeout_ms <= 0) { + std::fprintf(stderr, + "[server] --tool-spec-timeout-ms must be positive\n"); + return 2; + } + } else if (std::strcmp( + argv[i], "--tool-spec-max-model-slowdown") == 0 && + i + 1 < argc) { + if (!parse_double_strict( + argv[++i], + sconfig.tool_speculation.max_model_slowdown_ratio) || + sconfig.tool_speculation.max_model_slowdown_ratio < 1.0) { + std::fprintf(stderr, + "[server] --tool-spec-max-model-slowdown must be >= 1\n"); + return 2; + } } else if (std::strcmp(argv[i], "--chat-template-file") == 0 && i + 1 < argc) { const char * path = argv[++i]; std::FILE * f = std::fopen(path, "rb"); @@ -616,6 +794,96 @@ int main(int argc, char ** argv) { return 2; } } + + const bool semantic_http_predictor_requested = + !sconfig.semantic_tool_predictor.url.empty() || + !sconfig.semantic_tool_predictor.model.empty(); + if (semantic_http_predictor_requested && + !sconfig.semantic_tool_predictor.http_enabled()) { + std::fprintf(stderr, + "[server] HTTP semantic tool hints require both " + "--tool-hint-sidecar-url and --tool-hint-sidecar-model\n"); + return 2; + } + const bool semantic_native_predictor_requested = + !sconfig.semantic_tool_predictor.native_model_path.empty() || + !sconfig.semantic_tool_predictor.native_ipc_bin.empty() || + !sconfig.semantic_tool_predictor.native_work_dir.empty(); + if (semantic_native_predictor_requested && + !sconfig.semantic_tool_predictor.native_enabled()) { + std::fprintf(stderr, + "[server] native semantic tool hints require both " + "--tool-hint-native-model and --tool-hint-native-ipc-bin\n"); + return 2; + } + const bool tool_speculation_requested = + !sconfig.tool_speculation.executor_path.empty() || + !sconfig.tool_speculation.profile_path.empty() || + !sconfig.tool_speculation.allowed_tools.empty() || + !sconfig.tool_speculation.cpu_affinity.empty(); + if (tool_speculation_requested) { + if (!tool_speculation_executor_isolation_supported()) { + std::fprintf(stderr, + "[server] tool speculation requires Linux glibc >= 2.34 " + "for child descriptor isolation\n"); + return 2; + } + if (sconfig.tool_speculation.executor_path.empty() || + sconfig.tool_speculation.profile_path.empty() || + sconfig.tool_speculation.allowed_tools.empty()) { + std::fprintf(stderr, + "[server] tool speculation requires --tool-spec-executor, " + "--tool-spec-profile, and at least one --tool-spec-allow\n"); + return 2; + } +#if !defined(_WIN32) + if (::access(sconfig.tool_speculation.executor_path.c_str(), X_OK) != 0) { + std::fprintf(stderr, + "[server] tool speculation executor is not executable: %s\n", + sconfig.tool_speculation.executor_path.c_str()); + return 2; + } +#endif + std::sort(sconfig.tool_speculation.allowed_tools.begin(), + sconfig.tool_speculation.allowed_tools.end()); + sconfig.tool_speculation.allowed_tools.erase( + std::unique(sconfig.tool_speculation.allowed_tools.begin(), + sconfig.tool_speculation.allowed_tools.end()), + sconfig.tool_speculation.allowed_tools.end()); + std::string profile_error; + if (!sconfig.tool_speculation.policy.load_file( + sconfig.tool_speculation.profile_path, profile_error)) { + std::fprintf(stderr, "[server] %s\n", profile_error.c_str()); + return 2; + } + std::string cpu_affinity_error; + if (!qualify_tool_speculation_cpu_affinity( + sconfig.tool_speculation, cpu_affinity_error)) { + std::fprintf(stderr, "[server] %s\n", cpu_affinity_error.c_str()); + return 2; + } + if (sconfig.tool_speculation.cpu_affinity_isolated) { + std::fprintf(stderr, + "[server] disjoint CPU tool lane: %zu model logical CPUs, " + "%zu reserved tool logical CPUs\n", + sconfig.tool_speculation.model_cpu_affinity.size(), + sconfig.tool_speculation.cpu_affinity.size()); + } + const std::string & executor_contract = + sconfig.tool_speculation.policy.executor_contract(); + if (!executor_contract.empty() && + executor_contract != + sconfig.tool_speculation.execution_mode()) { + std::fprintf(stderr, + "[server] tool profile requires executor '%s', got '%s'\n", + executor_contract.c_str(), + sconfig.tool_speculation.execution_mode()); + return 2; + } + std::fprintf(stderr, + "[server] tool speculation preserves token speculation; " + "unqualified resource lanes are deferred\n"); + } if (fast_rollback_forced_off) { bargs.fast_rollback = false; target_split_fast_rollback_cli = false; @@ -1117,6 +1385,49 @@ int main(int argc, char ** argv) { std::fprintf(stderr, "[server] │ prefix_cache = %d slots\n", sconfig.prefix_cache_cap); std::fprintf(stderr, "[server] │ prefill_cache = %d slots\n", sconfig.prefill_cache_cap); std::fprintf(stderr, "[server] │ cors = %s\n", sconfig.enable_cors ? "ON" : "off"); + std::fprintf(stderr, "[server] │ tool_speculation= %s\n", + sconfig.tool_speculation.enabled() ? "ON" : "off"); + std::fprintf(stderr, "[server] │ tool_call_predictor= %s\n", + sconfig.semantic_tool_predictor.enabled() ? "ON" : "off"); + if (sconfig.semantic_tool_predictor.enabled()) { + const auto & predictor = sconfig.semantic_tool_predictor; + std::fprintf(stderr, "[server] │ tool_hint_transport= %s%s\n", + predictor.native_enabled() ? "native-qwen3" : "http", + predictor.native_enabled() && predictor.http_enabled() + ? "+http-fallback" : ""); + std::fprintf(stderr, "[server] │ tool_hint_model = %s\n", + predictor.native_enabled() + ? predictor.native_model_path.c_str() + : predictor.model.c_str()); + if (predictor.native_enabled()) { + std::fprintf(stderr, + "[server] │ tool_hint_gpu = %d (max_ctx=%d)\n", + predictor.native_gpu, predictor.native_max_ctx); + } + std::fprintf(stderr, + "[server] │ tool_hint_schedule= before-model\n"); + std::fprintf(stderr, "[server] │ tool_hint_timeout= %d ms\n", + predictor.timeout_ms); + std::fprintf(stderr, "[server] │ tool_hint_execute= %s (confidence=%.3f)\n", + sconfig.tool_speculation.enabled() ? "ON" : "off", + predictor.execution_confidence); + } + if (sconfig.tool_speculation.enabled()) { + std::fprintf(stderr, "[server] │ tool_spec_exec = %s\n", + sconfig.tool_speculation.execution_mode()); + std::fprintf(stderr, "[server] │ tool_spec_profile= %s\n", + sconfig.tool_speculation.profile_path.c_str()); + std::fprintf(stderr, "[server] │ tool_spec_decode = %s\n", + "spec preserved (unqualified lanes deferred)"); + std::fprintf(stderr, "[server] │ tool_spec_lanes ="); + for (const auto & lane : sconfig.tool_speculation.policy.lanes()) { + std::fprintf(stderr, " %d%%:%s", + lane.resource_percentage, + lane.decode_interference_qualified + ? "qualified" : "deferred"); + } + std::fprintf(stderr, "\n"); + } std::fprintf(stderr, "[server] │ cache_type_k = %s\n", #ifdef GGML_USE_HIP cache_type_k.empty() ? "q4_0 (default, HIP)" : cache_type_k.c_str()); @@ -1233,6 +1544,13 @@ int main(int argc, char ** argv) { HttpServer server(*backend, tokenizer, sconfig); server.set_chat_format(chat_format_for_arch(arch)); + std::string semantic_predictor_error; + if (!server.init_semantic_tool_predictor(semantic_predictor_error)) { + std::fprintf(stderr, + "[server] native semantic tool predictor initialization failed: %s\n", + semantic_predictor_error.c_str()); + return 1; + } g_server = &server; std::signal(SIGTERM, signal_handler); std::signal(SIGINT, signal_handler); diff --git a/server/src/server/tokenizer.cpp b/server/src/server/tokenizer.cpp index 5ff4b1a78..a350d3ff3 100644 --- a/server/src/server/tokenizer.cpp +++ b/server/src/server/tokenizer.cpp @@ -14,10 +14,65 @@ #include #include #include +#include #include namespace dflash::common { +namespace { + +constexpr size_t kMaxAddedTokenBytes = 4096; + +bool preprocessing_deadline_expired( + const std::chrono::steady_clock::time_point * deadline, + size_t & operations, + bool & timed_out) { + if (!deadline || (operations++ & 255U) != 0U) return false; + if (std::chrono::steady_clock::now() < *deadline) return false; + timed_out = true; + return true; +} + +bool preprocessing_find_until( + const std::string & text, + const std::string & needle, + size_t start, + const std::chrono::steady_clock::time_point * deadline, + bool & timed_out, + size_t & found) { + found = std::string::npos; + if (needle.empty()) { + found = start <= text.size() ? start : std::string::npos; + return true; + } + if (start > text.size() || needle.size() > text.size() - start) { + return true; + } + + constexpr size_t kSearchPositionsPerDeadlineCheck = 4096; + const size_t last_start = text.size() - needle.size(); + size_t cursor = start; + while (cursor <= last_start) { + if (deadline && std::chrono::steady_clock::now() >= *deadline) { + timed_out = true; + return false; + } + const size_t positions = (std::min)( + kSearchPositionsPerDeadlineCheck, last_start - cursor + 1); + const size_t view_bytes = positions + needle.size() - 1; + const std::string_view window(text.data() + cursor, view_bytes); + const size_t local = window.find(needle); + if (local != std::string_view::npos && local < positions) { + found = cursor + local; + return true; + } + cursor += positions; + } + return true; +} + +} // namespace + // ─── Unicode helpers ──────────────────────────────────────────────────── static int utf8_len(uint8_t c) { @@ -147,11 +202,15 @@ static bool is_newline(uint32_t cp) { // \s+(?!\S) | // \s+ -std::vector Tokenizer::pre_tokenize(const std::string & text) const { +std::vector Tokenizer::pre_tokenize( + const std::string & text, + const std::chrono::steady_clock::time_point * deadline, + bool & timed_out) const { std::vector pieces; const char * s = text.c_str(); const size_t len = text.size(); size_t pos = 0; + size_t operations = 0; auto peek_cp = [&](size_t p, int * cplen) -> uint32_t { if (p >= len) { *cplen = 0; return 0; } @@ -159,6 +218,8 @@ std::vector Tokenizer::pre_tokenize(const std::string & text) const }; while (pos < len) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; size_t start = pos; int cplen = 0; uint32_t cp = peek_cp(pos, &cplen); @@ -204,6 +265,8 @@ std::vector Tokenizer::pre_tokenize(const std::string & text) const // One or more letter/mark chars if (cl > 0 && (is_letter(c) || is_mark(c))) { while (cl > 0 && (is_letter(c) || is_mark(c))) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; p += cl; c = peek_cp(p, &cl); } @@ -234,12 +297,16 @@ std::vector Tokenizer::pre_tokenize(const std::string & text) const size_t punc_start = p; while (cl > 0 && !is_whitespace(c) && !is_letter(c) && !is_mark(c) && !is_digit(c)) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; p += cl; c = peek_cp(p, &cl); } if (p > punc_start) { // Trailing newlines while (cl > 0 && is_newline(c)) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; p += cl; c = peek_cp(p, &cl); } @@ -256,11 +323,15 @@ std::vector Tokenizer::pre_tokenize(const std::string & text) const uint32_t c = peek_cp(p, &cl); // Consume leading whitespace while (cl > 0 && is_whitespace(c) && !is_newline(c)) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; p += cl; c = peek_cp(p, &cl); } if (cl > 0 && is_newline(c)) { while (cl > 0 && is_newline(c)) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; p += cl; c = peek_cp(p, &cl); } @@ -277,6 +348,8 @@ std::vector Tokenizer::pre_tokenize(const std::string & text) const c = peek_cp(p, &cl); size_t prev_p = pos; // position before last whitespace char while (cl > 0 && is_whitespace(c)) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; prev_p = p; p += cl; c = peek_cp(p, &cl); @@ -351,18 +424,30 @@ static std::string byte_to_gpt2_unicode(uint8_t b) { } // Convert a raw UTF-8 text piece to GPT-2 byte-encoded form for BPE lookup. -static std::string encode_gpt2_bpe(const std::string & text) { +static std::string encode_gpt2_bpe( + const std::string & text, + const std::chrono::steady_clock::time_point * deadline, + size_t & operations, + bool & timed_out) { std::string out; out.reserve(text.size() * 2); // GPT-2 encoding may expand for (uint8_t b : text) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; out += byte_to_gpt2_unicode(b); } return out; } // Encode a single pre-tokenized piece using BPE merges. -std::vector Tokenizer::bpe_encode_piece(const std::string & piece) const { +std::vector Tokenizer::bpe_encode_piece( + const std::string & piece, + const std::chrono::steady_clock::time_point * deadline, + bool & timed_out) const { if (piece.empty()) return {}; + size_t operations = 0; + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; std::vector symbols; @@ -380,6 +465,8 @@ std::vector Tokenizer::bpe_encode_piece(const std::string & piece) cons std::string encoded; encoded.reserve(sp_piece.size()); for (char c : sp_piece) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; if (c == ' ') { encoded += "\xe2\x96\x81"; } else { @@ -397,6 +484,8 @@ std::vector Tokenizer::bpe_encode_piece(const std::string & piece) cons const char * p = encoded.c_str(); const char * end = p + encoded.size(); while (p < end) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; int cplen; utf8_decode(p, (size_t)(end - p), &cplen); if (cplen <= 0) cplen = 1; @@ -414,7 +503,9 @@ std::vector Tokenizer::bpe_encode_piece(const std::string & piece) cons } } else { // GPT-2 BPE: convert raw text to GPT-2 byte encoding for vocab lookup. - std::string encoded = encode_gpt2_bpe(piece); + std::string encoded = encode_gpt2_bpe( + piece, deadline, operations, timed_out); + if (timed_out) return {}; // Try to find the encoded piece as a single token first. auto it = token_to_id_.find(encoded); @@ -424,6 +515,8 @@ std::vector Tokenizer::bpe_encode_piece(const std::string & piece) cons // Split into individual GPT-2-encoded bytes as initial BPE symbols. for (size_t i = 0; i < piece.size(); i++) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; std::string sym = byte_to_gpt2_unicode((uint8_t)piece[i]); auto sit = token_to_id_.find(sym); if (sit != token_to_id_.end()) { @@ -446,10 +539,14 @@ std::vector Tokenizer::bpe_encode_piece(const std::string & piece) cons // Iteratively merge the highest-priority pair until no more merges apply. while (symbols.size() > 1) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; int best_rank = std::numeric_limits::max(); size_t best_pos = SIZE_MAX; for (size_t i = 0; i + 1 < symbols.size(); i++) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; std::string pair = symbols[i] + " " + symbols[i + 1]; auto mit = merge_rank_.find(pair); if (mit != merge_rank_.end() && mit->second < best_rank) { @@ -469,6 +566,8 @@ std::vector Tokenizer::bpe_encode_piece(const std::string & piece) cons std::vector ids; ids.reserve(symbols.size()); for (const auto & sym : symbols) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; auto sit = token_to_id_.find(sym); if (sit != token_to_id_.end()) { ids.push_back(sit->second); @@ -493,6 +592,8 @@ std::vector Tokenizer::bpe_encode_piece(const std::string & piece) cons const char * p = sym.c_str(); const char * end = p + sym.size(); while (p < end) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; int cplen; uint32_t cp = utf8_decode(p, (size_t)(end - p), &cplen); uint8_t orig_byte; @@ -567,6 +668,13 @@ bool Tokenizer::load_from_gguf(const char * model_path) { if (ttype == 3 || ttype == 4) { const std::string & tok = id_to_token_[i]; if (!tok.empty()) { + if (tok.size() > kMaxAddedTokenBytes) { + std::fprintf(stderr, + "[tokenizer] special token %d exceeds %zu-byte limit\n", + i, kMaxAddedTokenBytes); + gguf_free(gctx); + return false; + } added_tokens_.push_back({tok, (int32_t)i}); } } @@ -633,13 +741,25 @@ bool Tokenizer::load_from_gguf(const char * model_path) { return true; } -std::vector Tokenizer::encode(const std::string & text) const { +std::vector Tokenizer::encode_impl( + const std::string & text, + const std::chrono::steady_clock::time_point * deadline, + bool & timed_out) const { + size_t operations = 0; + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; // If no added tokens, fast path: pre-tokenize → BPE entire text. if (added_tokens_.empty()) { - std::vector pieces = pre_tokenize(text); + std::vector pieces = pre_tokenize( + text, deadline, timed_out); + if (timed_out) return {}; std::vector ids; for (const auto & piece : pieces) { - auto piece_ids = bpe_encode_piece(piece); + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; + auto piece_ids = bpe_encode_piece( + piece, deadline, timed_out); + if (timed_out) return {}; ids.insert(ids.end(), piece_ids.begin(), piece_ids.end()); } return ids; @@ -650,9 +770,13 @@ std::vector Tokenizer::encode(const std::string & text) const { std::vector ids; size_t pos = 0; while (pos < text.size()) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; // Try to match any added token at current position. bool matched = false; for (const auto & [tok_str, tok_id] : added_tokens_) { + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; if (pos + tok_str.size() <= text.size() && text.compare(pos, tok_str.size(), tok_str) == 0) { ids.push_back(tok_id); @@ -666,7 +790,13 @@ std::vector Tokenizer::encode(const std::string & text) const { // Find the next special token (or end of string). size_t next_special = text.size(); for (const auto & [tok_str, tok_id] : added_tokens_) { - size_t found = text.find(tok_str, pos); + if (preprocessing_deadline_expired( + deadline, operations, timed_out)) return {}; + size_t found = std::string::npos; + if (!preprocessing_find_until( + text, tok_str, pos, deadline, timed_out, found)) { + return {}; + } if (found != std::string::npos && found < next_special) { next_special = found; } @@ -674,9 +804,13 @@ std::vector Tokenizer::encode(const std::string & text) const { // Pre-tokenize + BPE the normal segment. std::string segment = text.substr(pos, next_special - pos); - std::vector pieces = pre_tokenize(segment); + std::vector pieces = pre_tokenize( + segment, deadline, timed_out); + if (timed_out) return {}; for (const auto & piece : pieces) { - auto piece_ids = bpe_encode_piece(piece); + auto piece_ids = bpe_encode_piece( + piece, deadline, timed_out); + if (timed_out) return {}; ids.insert(ids.end(), piece_ids.begin(), piece_ids.end()); } pos = next_special; @@ -684,6 +818,24 @@ std::vector Tokenizer::encode(const std::string & text) const { return ids; } +std::vector Tokenizer::encode(const std::string & text) const { + bool timed_out = false; + return encode_impl(text, nullptr, timed_out); +} + +bool Tokenizer::encode_until( + const std::string & text, + std::chrono::steady_clock::time_point deadline, + std::vector & out) const { + bool timed_out = false; + out = encode_impl(text, &deadline, timed_out); + if (timed_out || std::chrono::steady_clock::now() >= deadline) { + out.clear(); + return false; + } + return true; +} + // GPT-2 byte-level BPE uses a Unicode mapping where each byte 0-255 is // represented by a specific Unicode codepoint. Bytes that already have a // printable representation (33-126, 161-172, 174-255) map to themselves; diff --git a/server/src/server/tokenizer.h b/server/src/server/tokenizer.h index 5484fa472..045fb8aad 100644 --- a/server/src/server/tokenizer.h +++ b/server/src/server/tokenizer.h @@ -9,6 +9,7 @@ #pragma once +#include #include #include #include @@ -31,6 +32,12 @@ class Tokenizer { // ─── Encode ────────────────────────────────────────────────────── // Tokenize a UTF-8 string into token IDs. std::vector encode(const std::string & text) const; + // Predictor preprocessing uses the same tokenizer but must not overrun a + // request deadline. Returns false and clears `out` on expiry. + bool encode_until( + const std::string & text, + std::chrono::steady_clock::time_point deadline, + std::vector & out) const; // ─── Decode ────────────────────────────────────────────────────── // Convert a single token ID to its text representation. @@ -55,10 +62,20 @@ class Tokenizer { private: // Pre-tokenize text into pieces using Qwen3/3.5 regex pattern. - std::vector pre_tokenize(const std::string & text) const; + std::vector pre_tokenize( + const std::string & text, + const std::chrono::steady_clock::time_point * deadline, + bool & timed_out) const; // Apply BPE merges to a single pre-tokenized piece. - std::vector bpe_encode_piece(const std::string & piece) const; + std::vector bpe_encode_piece( + const std::string & piece, + const std::chrono::steady_clock::time_point * deadline, + bool & timed_out) const; + std::vector encode_impl( + const std::string & text, + const std::chrono::steady_clock::time_point * deadline, + bool & timed_out) const; // Vocabulary: id → token string std::vector id_to_token_; diff --git a/server/src/server/tool_speculation.cpp b/server/src/server/tool_speculation.cpp new file mode 100644 index 000000000..0f3ddae9e --- /dev/null +++ b/server/src/server/tool_speculation.cpp @@ -0,0 +1,1234 @@ +#if defined(__linux__) && !defined(_GNU_SOURCE) +# define _GNU_SOURCE +#endif + +#include "tool_speculation.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if !defined(_WIN32) +# include +# include +# include +# include +# include +# include +# include +# if defined(__linux__) +# include +# include +# endif +#endif + +#if !defined(_WIN32) && defined(__GLIBC__) && defined(__GLIBC_PREREQ) +# if __GLIBC_PREREQ(2, 34) +# define DFLASH_TOOL_SPEC_HAS_CLOSEFROM 1 +# endif +#endif +#ifndef DFLASH_TOOL_SPEC_HAS_CLOSEFROM +# define DFLASH_TOOL_SPEC_HAS_CLOSEFROM 0 +#endif + +namespace dflash::common { +namespace { + +constexpr size_t kMaxExecutorRequestBytes = 64 * 1024; + +bool finite_positive(double value) { + return std::isfinite(value) && value > 0.0; +} + +double median(std::vector values) { + if (values.empty()) return 0.0; + std::sort(values.begin(), values.end()); + const size_t middle = values.size() / 2; + if (values.size() % 2 != 0) return values[middle]; + return (values[middle - 1] + values[middle]) / 2.0; +} + +bool request_declares_tool(const json & tools, const std::string & name) { + if (!tools.is_array() || name.empty()) return false; + for (const auto & tool : tools) { + if (!tool.is_object()) continue; + if (tool.value("name", "") == name) return true; + if (tool.contains("function") && tool["function"].is_object() && + tool["function"].value("name", "") == name) { + return true; + } + } + return false; +} + +std::string format_cpu_affinity(const std::vector & cpus) { + std::string value; + for (const int cpu : cpus) { + if (!value.empty()) value.push_back(','); + value += std::to_string(cpu); + } + return value; +} + +#if !defined(_WIN32) +bool send_all_socket(int fd, const void * data, size_t bytes) { + const char * cursor = static_cast(data); + while (bytes > 0) { + int flags = 0; +# if defined(MSG_NOSIGNAL) + flags = MSG_NOSIGNAL; +# endif + const ssize_t written = ::send(fd, cursor, bytes, flags); + if (written < 0) { + if (errno == EINTR) continue; + return false; + } + if (written == 0) return false; + cursor += written; + bytes -= static_cast(written); + } + return true; +} + +void signal_executor_process_group( + pid_t leader, int signal, bool leader_fallback = true) { + if (leader <= 0) return; + if (::kill(-leader, signal) != 0 && errno == ESRCH && leader_fallback) { + // Defensive fallback for older platforms that ignored the requested + // spawn process group. + (void)::kill(leader, signal); + } +} + +std::vector executor_environment( + int resource_percentage, + const std::string & accelerator_relation, + const std::vector & cpu_affinity) { + std::vector values; + // Do not copy the long-running server's environment into a tool process: + // it commonly contains model-provider keys and upstream credentials. Keep + // only the small runtime surface needed by executable/script adapters. + static constexpr const char * kInherited[] = { + "PATH", + "LANG", + "LC_ALL", + "LC_CTYPE", + "TZ", + "LD_LIBRARY_PATH", + "DFLASH_TRACE_TRAINING_REPORT", + "DFLASH_TRACE_WORKFLOW_REGISTRY", + }; + for (const char * name : kInherited) { + if (const char * value = std::getenv(name); value && *value) { + values.push_back(std::string(name) + "=" + value); + } + } + values.push_back("DFLASH_TOOL_SPECULATION=1"); + values.push_back( + "DFLASH_TOOL_SPECULATION_RESOURCE_PERCENTAGE=" + + std::to_string(resource_percentage)); + values.push_back( + "DFLASH_TOOL_SPECULATION_ACCELERATOR_RELATION=" + + accelerator_relation); + if (!cpu_affinity.empty()) { + values.push_back( + "DFLASH_TOOL_SPECULATION_CPU_AFFINITY=" + + format_cpu_affinity(cpu_affinity)); + } + return values; +} + +# if defined(__linux__) +bool wait_for_child_cpu_affinity( + pid_t child, + const std::vector & cpus, + std::string & error) { + if (cpus.empty()) { + error.clear(); + return true; + } + cpu_set_t requested; + CPU_ZERO(&requested); + for (const int cpu : cpus) { + if (cpu < 0 || cpu >= CPU_SETSIZE) { + error = "executor CPU is outside CPU_SETSIZE: " + + std::to_string(cpu); + return false; + } + CPU_SET(cpu, &requested); + } + const auto deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(250); + do { + cpu_set_t observed; + CPU_ZERO(&observed); + if (::sched_getaffinity(child, sizeof(observed), &observed) == 0) { + bool matches = true; + for (int cpu = 0; cpu < CPU_SETSIZE; ++cpu) { + if (CPU_ISSET(cpu, &requested) != CPU_ISSET(cpu, &observed)) { + matches = false; + break; + } + } + if (matches) { + error.clear(); + return true; + } + } else if (errno != EINTR) { + error = std::string("executor sched_getaffinity failed: ") + + std::strerror(errno); + return false; + } + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } while (std::chrono::steady_clock::now() < deadline); + error = "executor CPU affinity verification mismatch"; + return false; +} +# endif +#endif + +} // namespace + +bool tool_speculation_executor_isolation_supported() { + return DFLASH_TOOL_SPEC_HAS_CLOSEFROM != 0; +} + +bool CanonicalToolInvocation::from_parts( + const std::string & name, + const json & arguments, + CanonicalToolInvocation & out, + std::string & error) { + if (name.empty()) { + error = "tool name must not be empty"; + return false; + } + if (!arguments.is_object()) { + error = "tool arguments must be a JSON object"; + return false; + } + out.name = name; + out.arguments = arguments; + // nlohmann::json's default object type is key ordered, so dump() is a + // stable canonical identity independent of input object insertion order. + out.arguments_json = arguments.dump(); + error.clear(); + return true; +} + +bool CanonicalToolInvocation::from_tool_call( + const ToolCall & call, + CanonicalToolInvocation & out, + std::string & error) { + try { + const json arguments = call.arguments.empty() + ? json::object() + : json::parse(call.arguments); + return from_parts(call.name, arguments, out, error); + } catch (const std::exception & exception) { + error = std::string("authoritative tool arguments are invalid JSON: ") + + exception.what(); + return false; + } +} + +bool build_tool_speculation_prediction( + const std::string & name, + const json & arguments, + double confidence, + ToolSpeculationPrediction & out, + std::string & error) { + if (!std::isfinite(confidence) || confidence < 0.0 || confidence > 1.0) { + error = "tool prediction confidence must be between 0 and 1"; + return false; + } + CanonicalToolInvocation invocation; + if (!CanonicalToolInvocation::from_parts( + name, arguments, invocation, error)) { + return false; + } + out.call = std::move(invocation); + out.confidence = confidence; + error.clear(); + return true; +} + +bool parse_tool_speculation_prediction( + const json & value, + const json & tools, + ToolSpeculationPrediction & out, + std::string & error) { + if (!value.is_object()) { + error = "tool_speculation must be an object"; + return false; + } + if (!value.contains("call") || !value["call"].is_object()) { + error = "tool_speculation.call must be an object"; + return false; + } + if (!value.contains("confidence") || !value["confidence"].is_number()) { + error = "tool_speculation.confidence must be a number"; + return false; + } + const double confidence = value["confidence"].get(); + if (!std::isfinite(confidence) || confidence < 0.0 || confidence > 1.0) { + error = "tool_speculation.confidence must be between 0 and 1"; + return false; + } + + const json & call = value["call"]; + if (!call.contains("name") || !call["name"].is_string()) { + error = "tool_speculation.call.name must be a string"; + return false; + } + if (!call.contains("arguments")) { + error = "tool_speculation.call.arguments is required"; + return false; + } + ToolSpeculationPrediction prediction; + if (!build_tool_speculation_prediction( + call["name"].get(), call["arguments"], confidence, + prediction, error)) { + return false; + } + if (!request_declares_tool(tools, prediction.call.name)) { + error = "tool_speculation.call.name is not declared in tools"; + return false; + } + out = std::move(prediction); + error.clear(); + return true; +} + +bool parse_tool_speculation_cpu_affinity( + const std::string & value, + std::vector & out, + std::string & error) { + out.clear(); + if (value.empty()) { + error = "tool CPU affinity must not be empty"; + return false; + } + size_t cursor = 0; + while (cursor < value.size()) { + const size_t comma = value.find(',', cursor); + const size_t end = comma == std::string::npos ? value.size() : comma; + const std::string token = value.substr(cursor, end - cursor); + if (token.empty()) { + error = "tool CPU affinity contains an empty item"; + out.clear(); + return false; + } + const size_t dash = token.find('-'); + auto parse_cpu = [&](const std::string & item, int & cpu) { + if (item.empty() || !std::all_of( + item.begin(), item.end(), [](unsigned char character) { + return character >= '0' && character <= '9'; + })) { + return false; + } + char * parsed_end = nullptr; + errno = 0; + const long parsed = std::strtol(item.c_str(), &parsed_end, 10); + if (errno != 0 || !parsed_end || *parsed_end != '\0' || + parsed < 0 || parsed > std::numeric_limits::max()) { + return false; + } + cpu = static_cast(parsed); + return true; + }; + int first = -1; + int last = -1; + if (dash == std::string::npos) { + if (!parse_cpu(token, first)) { + error = "invalid tool CPU affinity item: " + token; + out.clear(); + return false; + } + last = first; + } else if (token.find('-', dash + 1) != std::string::npos || + !parse_cpu(token.substr(0, dash), first) || + !parse_cpu(token.substr(dash + 1), last) || + first > last) { + error = "invalid tool CPU affinity range: " + token; + out.clear(); + return false; + } + if (static_cast(last) - + static_cast(first) > 65535ULL) { + error = "tool CPU affinity range is too large: " + token; + out.clear(); + return false; + } + for (int cpu = first; cpu <= last; ++cpu) { + out.push_back(cpu); + if (cpu == std::numeric_limits::max()) break; + } + if (comma == std::string::npos) break; + cursor = comma + 1; + } + std::sort(out.begin(), out.end()); + out.erase(std::unique(out.begin(), out.end()), out.end()); + error.clear(); + return true; +} + +bool qualify_tool_speculation_cpu_affinity( + ToolSpeculationConfig & config, + std::string & error) { + config.model_cpu_affinity.clear(); + config.cpu_affinity_isolated = false; + if (config.cpu_affinity.empty()) { + error.clear(); + return true; + } +#if defined(__linux__) + if (::access("/usr/bin/taskset", X_OK) != 0) { + error = "tool CPU affinity requires executable /usr/bin/taskset"; + return false; + } + const long configured_cpus = ::sysconf(_SC_NPROCESSORS_CONF); + if (configured_cpus <= 0) { + error = "cannot determine configured CPU count"; + return false; + } + cpu_set_t model_set; + CPU_ZERO(&model_set); + if (::sched_getaffinity(0, sizeof(model_set), &model_set) != 0) { + error = std::string("model sched_getaffinity failed: ") + + std::strerror(errno); + return false; + } + for (int cpu = 0; cpu < CPU_SETSIZE; ++cpu) { + if (CPU_ISSET(cpu, &model_set)) { + config.model_cpu_affinity.push_back(cpu); + } + } + if (config.model_cpu_affinity.empty()) { + error = "model CPU affinity is empty"; + return false; + } + for (const int cpu : config.cpu_affinity) { + if (cpu < 0 || cpu >= CPU_SETSIZE || cpu >= configured_cpus) { + error = "tool CPU is not configured on this host: " + + std::to_string(cpu); + return false; + } + if (CPU_ISSET(cpu, &model_set)) { + error = "tool CPU affinity overlaps model CPU " + + std::to_string(cpu); + return false; + } + } + config.cpu_affinity_isolated = true; + error.clear(); + return true; +#else + error = "tool CPU affinity isolation is supported only on Linux"; + return false; +#endif +} + +bool ToolSpeculationPolicy::load_file( + const std::string & path, std::string & error) { + std::ifstream input(path); + if (!input) { + error = "cannot open tool-speculation profile: " + path; + lanes_.clear(); + baseline_task_ms_ = 0.0; + profile_status_ = "qualified"; + executor_contract_.clear(); + return false; + } + try { + json report; + input >> report; + return load_json(report, error); + } catch (const std::exception & exception) { + error = std::string("invalid tool-speculation profile JSON: ") + + exception.what(); + lanes_.clear(); + baseline_task_ms_ = 0.0; + profile_status_ = "qualified"; + executor_contract_.clear(); + return false; + } +} + +bool ToolSpeculationPolicy::load_json( + const json & report, std::string & error) { + lanes_.clear(); + baseline_task_ms_ = 0.0; + profile_status_ = "qualified"; + executor_contract_.clear(); + if (!report.is_object() || !report.contains("path_summary") || + !report["path_summary"].is_object() || + report["path_summary"].empty()) { + error = "tool-speculation profile needs a non-empty path_summary"; + return false; + } + + if (!report.contains("profile_status") || + !report["profile_status"].is_string()) { + error = "tool-speculation profile needs profile_status=qualified"; + return false; + } + profile_status_ = report["profile_status"].get(); + if (profile_status_ != "qualified") { + error = "tool-speculation profile_status must be qualified"; + return false; + } + if (!report.contains("executor") || !report["executor"].is_string()) { + error = "tool-speculation profile needs an executor contract"; + return false; + } + executor_contract_ = report["executor"].get(); + if (executor_contract_.empty()) { + error = "tool-speculation executor contract cannot be empty"; + return false; + } + + std::vector controls; + try { + for (auto item = report["path_summary"].begin(); + item != report["path_summary"].end(); ++item) { + size_t parsed = 0; + const int resource_percentage = std::stoi(item.key(), &parsed); + if (parsed != item.key().size() || + resource_percentage < 1 || resource_percentage > 100) { + throw std::runtime_error( + "invalid resource percentage " + item.key()); + } + const json & paths = item.value(); + const json & hit = paths.at("hit"); + const json & miss = paths.at("miss"); + const double hit_control = hit.at("control_task_mean_ms").get(); + const double miss_control = miss.at("control_task_mean_ms").get(); + const double hit_task = hit.at("speculative_task_mean_ms").get(); + const double miss_task = miss.at("speculative_task_mean_ms").get(); + const double slowdown_percent = std::max( + hit.at("model_slowdown_percent").get(), + miss.at("model_slowdown_percent").get()); + bool decode_interference_qualified = false; + if (paths.contains("decode_interference_qualified")) { + if (!paths["decode_interference_qualified"].is_boolean()) { + throw std::runtime_error( + "decode_interference_qualified must be boolean"); + } + decode_interference_qualified = + paths["decode_interference_qualified"].get(); + } + if (!paths.contains("accelerator_relation") || + !paths["accelerator_relation"].is_string()) { + throw std::runtime_error( + "accelerator_relation must be explicit"); + } + const std::string accelerator_relation = + paths["accelerator_relation"].get(); + if (accelerator_relation != "non_accelerator" && + accelerator_relation != "separate_physical_gpu") { + throw std::runtime_error( + "accelerator_relation must be non_accelerator or " + "separate_physical_gpu"); + } + if (!finite_positive(hit_control) || + !finite_positive(miss_control) || + !finite_positive(hit_task) || + !finite_positive(miss_task) || + !std::isfinite(slowdown_percent) || slowdown_percent < -100.0) { + throw std::runtime_error("non-positive or non-finite profile latency"); + } + const double control = (hit_control + miss_control) / 2.0; + lanes_.push_back({ + resource_percentage, + control, + hit_task, + miss_task, + 1.0 + slowdown_percent / 100.0, + decode_interference_qualified, + accelerator_relation, + }); + controls.push_back(control); + } + } catch (const std::exception & exception) { + error = std::string("invalid tool-speculation path_summary: ") + + exception.what(); + lanes_.clear(); + profile_status_ = "qualified"; + executor_contract_.clear(); + return false; + } + + std::sort(lanes_.begin(), lanes_.end(), + [](const auto & left, const auto & right) { + return left.resource_percentage < + right.resource_percentage; + }); + for (size_t index = 1; index < lanes_.size(); ++index) { + if (lanes_[index - 1].resource_percentage == + lanes_[index].resource_percentage) { + error = + "tool-speculation profile has duplicate resource percentages"; + lanes_.clear(); + return false; + } + } + baseline_task_ms_ = median(std::move(controls)); + error.clear(); + return true; +} + +ToolSpeculationAdmission ToolSpeculationPolicy::choose( + double confidence, + double max_model_slowdown_ratio) const { + ToolSpeculationAdmission decision; + decision.expected_task_ms = baseline_task_ms_; + if (lanes_.empty() || !finite_positive(baseline_task_ms_)) { + decision.reason = "profile_unavailable"; + return decision; + } + if (!std::isfinite(confidence) || confidence < 0.0 || confidence > 1.0) { + decision.reason = "invalid_confidence"; + return decision; + } + if (!std::isfinite(max_model_slowdown_ratio) || + max_model_slowdown_ratio < 1.0) { + decision.reason = "invalid_slowdown_guardrail"; + return decision; + } + + bool qualified_lane_available = false; + bool lane_passed_guardrail = false; + double best = baseline_task_ms_; + for (const ToolSpeculationLane & lane : lanes_) { + // Token speculation is an invariant, not a fallback choice. A lane + // may overlap DS4/DSpark only after its exact executor and placement + // passed the output-identity interference gate. + if (!lane.decode_interference_qualified) continue; + qualified_lane_available = true; + if (lane.model_slowdown_ratio > max_model_slowdown_ratio) continue; + lane_passed_guardrail = true; + const double expected = + confidence * lane.hit_task_ms + + (1.0 - confidence) * lane.miss_task_ms; + if (expected < best) { + best = expected; + decision.admitted = true; + decision.resource_percentage = lane.resource_percentage; + decision.expected_task_ms = expected; + decision.decode_interference_qualified = + lane.decode_interference_qualified; + decision.accelerator_relation = lane.accelerator_relation; + } + } + if (!decision.admitted) { + decision.reason = !qualified_lane_available + ? "decode_interference_unqualified" + : lane_passed_guardrail + ? "below_profile_break_even" + : "model_slowdown_guardrail"; + return decision; + } + decision.expected_speedup = baseline_task_ms_ / best; + decision.reason = "expected_latency_gain"; + return decision; +} + +bool ToolSpeculationConfig::allows(const std::string & name) const { + return std::find(allowed_tools.begin(), allowed_tools.end(), name) != + allowed_tools.end(); +} + +ToolSpeculationAttempt::ToolSpeculationAttempt( + const ToolSpeculationConfig & config, + const ToolSpeculationPrediction & prediction, + const std::string & request_id) + : config_(config) + , prediction_(prediction) + , request_id_(request_id) { + if (!config_.enabled()) { + admission_.reason = "engine_disabled"; + } else if (!config_.allows(prediction_.call.name)) { + admission_.reason = "tool_not_allowlisted"; + } else { + admission_ = config_.policy.choose( + prediction_.confidence, config_.max_model_slowdown_ratio); + } +} + +ToolSpeculationAttempt::~ToolSpeculationAttempt() { + if (!resolved_) terminate_executor(); +} + +std::unique_ptr ToolSpeculationAttempt::create( + const ToolSpeculationConfig & config, + const ToolSpeculationPrediction & prediction, + const std::string & request_id) { + return std::unique_ptr( + new ToolSpeculationAttempt(config, prediction, request_id)); +} + +void ToolSpeculationAttempt::start() { + if (started_) return; + started_ = true; + if (!admission_.admitted) return; + const json request = { + {"protocol", "dflash.tool-speculation.v1"}, + {"request_id", request_id_}, + {"mode", "speculative"}, + {"resource_percentage", admission_.resource_percentage}, + {"accelerator_relation", admission_.accelerator_relation}, + {"cpu_affinity", config_.cpu_affinity}, + {"cpu_affinity_isolated", config_.cpu_affinity_isolated}, + {"call", { + {"name", prediction_.call.name}, + {"arguments", prediction_.call.arguments}, + }}, + }; + started_at_ = std::chrono::steady_clock::now(); +#if defined(_WIN32) + launch_error_ = "tool speculation child executors are not implemented on Windows"; + return; +#else + const std::string payload = request.dump() + "\n"; + if (payload.size() > kMaxExecutorRequestBytes) { + launch_error_ = "executor request exceeds 64 KiB"; + return; + } + + int input_socket[2] = {-1, -1}; + if (::socketpair(AF_UNIX, SOCK_STREAM, 0, input_socket) != 0) { + launch_error_ = std::string("executor stdin socket failed: ") + + std::strerror(errno); + return; + } +# if defined(SO_NOSIGPIPE) + int no_sigpipe = 1; + ::setsockopt(input_socket[0], SOL_SOCKET, SO_NOSIGPIPE, + &no_sigpipe, sizeof(no_sigpipe)); +# endif + int output_pipe[2] = {-1, -1}; + if (::pipe(output_pipe) != 0) { + launch_error_ = std::string("executor stdout pipe failed: ") + + std::strerror(errno); + ::close(input_socket[0]); + ::close(input_socket[1]); + return; + } + + posix_spawn_file_actions_t actions; + int spawn_status = posix_spawn_file_actions_init(&actions); + const bool actions_initialized = spawn_status == 0; + posix_spawnattr_t attributes; + const int attributes_status = posix_spawnattr_init(&attributes); + const bool attributes_initialized = attributes_status == 0; + if (spawn_status == 0 && attributes_status != 0) { + spawn_status = attributes_status; + } + if (spawn_status == 0) { + spawn_status = posix_spawnattr_setpgroup(&attributes, 0); + } + if (spawn_status == 0) { + spawn_status = posix_spawnattr_setflags( + &attributes, POSIX_SPAWN_SETPGROUP); + } + if (spawn_status == 0) { + spawn_status = posix_spawn_file_actions_adddup2( + &actions, input_socket[1], STDIN_FILENO); + } + if (spawn_status == 0) { + spawn_status = posix_spawn_file_actions_adddup2( + &actions, output_pipe[1], STDOUT_FILENO); + } + if (spawn_status == 0) { + spawn_status = posix_spawn_file_actions_addclose(&actions, output_pipe[0]); + } + if (spawn_status == 0) { + spawn_status = posix_spawn_file_actions_addclose(&actions, input_socket[0]); + } + if (spawn_status == 0 && input_socket[1] != STDIN_FILENO) { + spawn_status = posix_spawn_file_actions_addclose(&actions, input_socket[1]); + } + if (spawn_status == 0 && output_pipe[1] != STDOUT_FILENO) { + spawn_status = posix_spawn_file_actions_addclose(&actions, output_pipe[1]); + } +# if DFLASH_TOOL_SPEC_HAS_CLOSEFROM + // The executor receives only stdin/stdout/stderr. In particular, it must + // not inherit listening sockets, live client connections, model IPC pipes, + // or accelerator descriptors from the long-running server. + if (spawn_status == 0) { + spawn_status = posix_spawn_file_actions_addclosefrom_np( + &actions, STDERR_FILENO + 1); + } +# endif + + std::vector env_storage = executor_environment( + admission_.resource_percentage, admission_.accelerator_relation, + config_.cpu_affinity); + std::vector env; + env.reserve(env_storage.size() + 1); + for (std::string & value : env_storage) env.push_back(value.data()); + env.push_back(nullptr); + + std::string executable = config_.cpu_affinity.empty() + ? config_.executor_path + : "/usr/bin/taskset"; + std::string protocol_arg = "--dflash-tool-spec-v1"; + std::string affinity_arg = format_cpu_affinity(config_.cpu_affinity); + std::string taskset_cpu_arg = "-c"; + std::vector argv; + argv.push_back(executable.data()); + if (!config_.cpu_affinity.empty()) { + // taskset applies the mask before execve(), so no executor startup + // code can run on the model CPUs. The request payload remains withheld + // until the parent verifies the resulting mask below. + argv.push_back(taskset_cpu_arg.data()); + argv.push_back(affinity_arg.data()); + argv.push_back(config_.executor_path.data()); + } + argv.push_back(protocol_arg.data()); + argv.push_back(nullptr); + pid_t child = -1; + if (spawn_status == 0) { + spawn_status = ::posix_spawn( + &child, executable.c_str(), &actions, &attributes, argv.data(), + env.data()); + } + if (actions_initialized) { + posix_spawn_file_actions_destroy(&actions); + } + if (attributes_initialized) { + posix_spawnattr_destroy(&attributes); + } + ::close(input_socket[1]); + ::close(output_pipe[1]); + if (spawn_status != 0) { + launch_error_ = std::string("executor spawn failed: ") + + std::strerror(spawn_status); + ::close(input_socket[0]); + ::close(output_pipe[0]); + return; + } + +# if defined(__linux__) + if (!config_.cpu_affinity.empty()) { + std::string affinity_error; + if (!wait_for_child_cpu_affinity( + child, config_.cpu_affinity, affinity_error)) { + signal_executor_process_group(child, SIGKILL); + int child_status = 0; + while (::waitpid(child, &child_status, 0) < 0 && errno == EINTR) {} + ::close(input_socket[0]); + ::close(output_pipe[0]); + launch_error_ = std::move(affinity_error); + return; + } + } +# else + if (!config_.cpu_affinity.empty()) { + signal_executor_process_group(child, SIGKILL); + int child_status = 0; + while (::waitpid(child, &child_status, 0) < 0 && errno == EINTR) {} + ::close(input_socket[0]); + ::close(output_pipe[0]); + launch_error_ = "tool CPU affinity isolation is supported only on Linux"; + return; + } +# endif + + child_pid_ = static_cast(child); + child_stdin_fd_ = input_socket[0]; + child_stdout_fd_ = output_pipe[0]; + const int flags = ::fcntl(child_stdout_fd_, F_GETFL, 0); + if (flags >= 0) { + ::fcntl(child_stdout_fd_, F_SETFL, flags | O_NONBLOCK); + } + running_ = true; + if (!send_all_socket(child_stdin_fd_, payload.data(), payload.size())) { + launch_error_ = std::string("executor request write failed: ") + + std::strerror(errno); + terminate_executor(); + return; + } + std::fprintf(stderr, + "[tool-spec] launched request=%s tool=%s confidence=%.3f " + "resource=%d%%\n", + request_id_.c_str(), prediction_.call.name.c_str(), + prediction_.confidence, admission_.resource_percentage); +#endif +} + +json ToolSpeculationAttempt::base_metadata() const { + json metadata = { + {"protocol", "dflash.tool-speculation.v1"}, + {"confidence", prediction_.confidence}, + {"prediction", { + {"name", prediction_.call.name}, + {"arguments", prediction_.call.arguments}, + }}, + {"resource_percentage", + admission_.admitted + ? json(admission_.resource_percentage) + : json(nullptr)}, + {"expected_speedup", + admission_.admitted ? json(admission_.expected_speedup) : json(nullptr)}, + {"decode_interference_qualified", + admission_.admitted + ? json(admission_.decode_interference_qualified) + : json(nullptr)}, + {"accelerator_relation", + admission_.admitted + ? json(admission_.accelerator_relation) + : json(nullptr)}, + {"cpu_affinity", config_.cpu_affinity}, + {"cpu_affinity_isolated", config_.cpu_affinity_isolated}, + }; + return metadata; +} + +bool ToolSpeculationAttempt::send_control(const char * operation) { +#if defined(_WIN32) + (void)operation; + return false; +#else + if (child_stdin_fd_ < 0 || !operation || !*operation) return false; + const std::string command = json({ + {"protocol", "dflash.tool-speculation.v1"}, + {"request_id", request_id_}, + {"op", operation}, + {"authoritative_resource_percentage", 100}, + }).dump() + "\n"; + return send_all_socket( + child_stdin_fd_, command.data(), command.size()); +#endif +} + +void ToolSpeculationAttempt::terminate_executor(bool allow_control_grace) { +#if !defined(_WIN32) + if (child_stdin_fd_ >= 0) { + ::close(child_stdin_fd_); + child_stdin_fd_ = -1; + } + if (child_stdout_fd_ >= 0) { + ::close(child_stdout_fd_); + child_stdout_fd_ = -1; + } + if (child_pid_ > 0) { + const pid_t pid = static_cast(child_pid_); + auto wait_until = [&](std::chrono::steady_clock::time_point deadline) { + int status = 0; + while (std::chrono::steady_clock::now() < deadline) { + const pid_t waited = ::waitpid(pid, &status, WNOHANG); + if (waited == pid || (waited < 0 && errno == ECHILD)) { + child_pid_ = -1; + running_ = false; + return true; + } + if (waited < 0 && errno != EINTR) break; + std::this_thread::sleep_for(std::chrono::milliseconds(2)); + } + return false; + }; + const int grace_ms = std::max(0, config_.cancel_grace_ms); + if (allow_control_grace && wait_until( + std::chrono::steady_clock::now() + + std::chrono::milliseconds(grace_ms))) { + // The executor contract does not permit detached descendants. + // Clean up any process that outlived its group leader. + signal_executor_process_group(pid, SIGKILL, false); + return; + } + signal_executor_process_group(pid, SIGTERM); + const int term_grace_ms = allow_control_grace + ? std::min(20, grace_ms) : grace_ms; + if (wait_until(std::chrono::steady_clock::now() + + std::chrono::milliseconds(term_grace_ms))) { + signal_executor_process_group(pid, SIGKILL, false); + return; + } + int status = 0; + signal_executor_process_group(pid, SIGKILL); + while (::waitpid(pid, &status, 0) < 0 && errno == EINTR) {} + child_pid_ = -1; + } +#endif + running_ = false; +} + +bool ToolSpeculationAttempt::collect_executor_result( + json & result, + double & wait_ms, + std::string & error) { +#if defined(_WIN32) + (void)result; + wait_ms = 0.0; + error = "tool speculation executors are not implemented on Windows"; + return false; +#else + const auto wait_started = std::chrono::steady_clock::now(); + const auto deadline = started_at_ + + std::chrono::milliseconds(std::max(1, config_.timeout_ms)); + if (wait_started >= deadline) { + error = "executor_timeout"; + terminate_executor(); + wait_ms = 0.0; + return false; + } + std::string output; + bool eof = false; + while (!eof) { + const auto now = std::chrono::steady_clock::now(); + if (now >= deadline) { + error = "executor_timeout"; + terminate_executor(); + wait_ms = std::chrono::duration( + std::chrono::steady_clock::now() - wait_started).count(); + return false; + } + const int remaining_ms = std::max(1, static_cast( + std::chrono::duration_cast( + deadline - now).count())); + pollfd descriptor{child_stdout_fd_, POLLIN | POLLHUP, 0}; + const int polled = ::poll(&descriptor, 1, remaining_ms); + if (polled < 0) { + if (errno == EINTR) continue; + error = std::string("executor_poll_failed: ") + std::strerror(errno); + terminate_executor(); + return false; + } + if (polled == 0) continue; + if (std::chrono::steady_clock::now() >= deadline) { + error = "executor_timeout"; + terminate_executor(); + wait_ms = std::chrono::duration( + std::chrono::steady_clock::now() - wait_started).count(); + return false; + } + if (descriptor.revents & (POLLERR | POLLNVAL)) { + error = "executor_stdout_failed"; + terminate_executor(); + return false; + } + if (descriptor.revents & (POLLIN | POLLHUP)) { + char buffer[8192]; + while (true) { + const ssize_t count = ::read( + child_stdout_fd_, buffer, sizeof(buffer)); + if (count > 0) { + if (output.size() + static_cast(count) > + config_.max_result_bytes) { + error = "executor_result_too_large"; + terminate_executor(); + return false; + } + output.append(buffer, static_cast(count)); + continue; + } + if (count == 0) { + eof = true; + break; + } + if (errno == EINTR) continue; + if (errno == EAGAIN || errno == EWOULDBLOCK) break; + error = std::string("executor_read_failed: ") + + std::strerror(errno); + terminate_executor(); + return false; + } + } + } + ::close(child_stdout_fd_); + child_stdout_fd_ = -1; + + int child_status = 0; + while (true) { + const pid_t waited = ::waitpid( + static_cast(child_pid_), &child_status, WNOHANG); + if (waited == static_cast(child_pid_)) break; + if (waited < 0) { + if (errno == EINTR) continue; + error = std::string("executor_wait_failed: ") + + std::strerror(errno); + child_pid_ = -1; + running_ = false; + return false; + } + if (std::chrono::steady_clock::now() >= deadline) { + error = "executor_exit_timeout"; + terminate_executor(); + return false; + } + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + // A successful adapter must not leave detached subprocesses behind. + signal_executor_process_group( + static_cast(child_pid_), SIGKILL, false); + child_pid_ = -1; + running_ = false; + wait_ms = std::chrono::duration( + std::chrono::steady_clock::now() - wait_started).count(); + if (!WIFEXITED(child_status) || WEXITSTATUS(child_status) != 0) { + error = WIFEXITED(child_status) + ? "executor_exit_" + std::to_string(WEXITSTATUS(child_status)) + : "executor_terminated"; + return false; + } + + try { + const json envelope = json::parse(output); + if (!envelope.is_object() || !envelope.value("ok", false) || + !envelope.contains("result")) { + error = "executor_rejected_or_invalid_envelope"; + return false; + } + result = envelope["result"]; + error.clear(); + return true; + } catch (const std::exception & exception) { + error = std::string("executor_invalid_json: ") + exception.what(); + return false; + } +#endif +} + +json ToolSpeculationAttempt::resolve( + const std::vector & authoritative_calls) { + if (resolved_) { + json metadata = base_metadata(); + metadata["status"] = "failed"; + metadata["reason"] = "already_resolved"; + return metadata; + } + resolved_ = true; + json metadata = base_metadata(); + if (!admission_.admitted) { + metadata["status"] = "deferred"; + metadata["reason"] = admission_.reason; + return metadata; + } + if (!launch_error_.empty() || !running_) { + metadata["status"] = "failed"; + metadata["reason"] = "executor_launch_failed"; + metadata["detail"] = launch_error_.empty() + ? "executor did not start" : launch_error_; + terminate_executor(); + return metadata; + } + if (authoritative_calls.size() != 1) { + send_control("cancel"); + terminate_executor(true); + metadata["status"] = "miss"; + metadata["reason"] = "authoritative_call_count"; + return metadata; + } + + CanonicalToolInvocation authoritative; + std::string canonical_error; + if (!CanonicalToolInvocation::from_tool_call( + authoritative_calls[0], authoritative, canonical_error)) { + send_control("cancel"); + terminate_executor(true); + metadata["status"] = "miss"; + metadata["reason"] = "invalid_authoritative_call"; + return metadata; + } + if (!(authoritative == prediction_.call)) { + send_control("cancel"); + terminate_executor(true); + metadata["status"] = "miss"; + metadata["reason"] = "invocation_mismatch"; + return metadata; + } + + json result; + double wait_ms = 0.0; + std::string executor_error; + const bool commit_signal_sent = send_control("commit"); +#if !defined(_WIN32) + if (child_stdin_fd_ >= 0) { + ::close(child_stdin_fd_); + child_stdin_fd_ = -1; + } +#endif + if (!collect_executor_result(result, wait_ms, executor_error)) { + metadata["status"] = "failed"; + metadata["reason"] = "speculative_executor_failure"; + metadata["detail"] = executor_error; + metadata["commit_signal_sent"] = commit_signal_sent; + metadata["commit_wait_ms"] = wait_ms; + return metadata; + } + const double wall_ms = std::chrono::duration( + std::chrono::steady_clock::now() - started_at_).count(); + metadata["status"] = "hit"; + metadata["call_id"] = authoritative_calls[0].id; + metadata["result"] = std::move(result); + metadata["commit_signal_sent"] = commit_signal_sent; + metadata["executor_wall_ms"] = wall_ms; + metadata["commit_wait_ms"] = wait_ms; + std::fprintf(stderr, + "[tool-spec] hit request=%s tool=%s resource=%d%% " + "wall_ms=%.1f wait_ms=%.1f\n", + request_id_.c_str(), prediction_.call.name.c_str(), + admission_.resource_percentage, wall_ms, wait_ms); + return metadata; +} + +json ToolSpeculationAttempt::cancel(const std::string & reason) { + if (!resolved_) { + resolved_ = true; + send_control("cancel"); + terminate_executor(true); + } + json metadata = base_metadata(); + metadata["status"] = "cancelled"; + metadata["reason"] = reason; + return metadata; +} + +std::string render_tool_speculation_sse( + ApiFormat api_format, + const std::string & request_id, + const std::string & model, + const json & metadata) { + switch (api_format) { + case ApiFormat::OPENAI_CHAT: { + const json event = { + {"id", request_id}, + {"object", "chat.completion.chunk"}, + {"model", model}, + {"choices", json::array()}, + {"dflash_tool_speculation", metadata}, + }; + return "data: " + event.dump() + "\n\n"; + } + case ApiFormat::ANTHROPIC: { + const json event = { + {"type", "dflash_tool_speculation"}, + {"dflash_tool_speculation", metadata}, + }; + return "event: dflash_tool_speculation\ndata: " + + event.dump() + "\n\n"; + } + case ApiFormat::RESPONSES: { + const json event = { + {"type", "response.dflash_tool_speculation"}, + {"response_id", request_id}, + {"dflash_tool_speculation", metadata}, + }; + return "event: response.dflash_tool_speculation\ndata: " + + event.dump() + "\n\n"; + } + default: + return "data: " + json({{"dflash_tool_speculation", metadata}}).dump() + + "\n\n"; + } +} + +} // namespace dflash::common diff --git a/server/src/server/tool_speculation.h b/server/src/server/tool_speculation.h new file mode 100644 index 000000000..fd1d890ee --- /dev/null +++ b/server/src/server/tool_speculation.h @@ -0,0 +1,234 @@ +// Lossless, confidence-gated speculative tool execution. +// +// The model remains authoritative. A predicted read-only invocation may run +// while inference is in flight, but its result is returned only when the +// emitted tool name and canonical JSON arguments match exactly. + +#pragma once + +#include "api_types.h" +#include "tool_parser.h" + +#include + +#include +#include +#include +#include +#include + +namespace dflash::common { + +using json = nlohmann::json; + +struct CanonicalToolInvocation { + std::string name; + json arguments = json::object(); + std::string arguments_json; + + static bool from_parts(const std::string & name, + const json & arguments, + CanonicalToolInvocation & out, + std::string & error); + static bool from_tool_call(const ToolCall & call, + CanonicalToolInvocation & out, + std::string & error); + + bool operator==(const CanonicalToolInvocation & other) const { + return name == other.name && arguments_json == other.arguments_json; + } +}; + +struct ToolSpeculationPrediction { + CanonicalToolInvocation call; + double confidence = 0.0; +}; + +// Construct a canonical prediction from an engine-side predictor. This is +// the same validation boundary used for caller-supplied predictions, minus +// the request-schema check performed by the semantic predictor itself. +bool build_tool_speculation_prediction(const std::string & name, + const json & arguments, + double confidence, + ToolSpeculationPrediction & out, + std::string & error); + +// Parse the request extension: +// "tool_speculation": { +// "call": {"name": "...", "arguments": {...}}, +// "confidence": 0.0..1.0 +// } +// The predicted tool must also be present in the request's `tools` array. +bool parse_tool_speculation_prediction(const json & value, + const json & tools, + ToolSpeculationPrediction & out, + std::string & error); + +// Parse a Linux CPU-list such as "14-15,30-31". The result is sorted and +// deduplicated so it can be compared directly with an observed affinity mask. +bool parse_tool_speculation_cpu_affinity(const std::string & value, + std::vector & out, + std::string & error); + +struct ToolSpeculationLane { + // Backend-neutral executor capacity. A CUDA adapter may map this to an + // MPS share; a ROCm, CPU, I/O, or remote adapter may interpret it using + // its own measured resource contract. + int resource_percentage = 0; + double control_task_ms = 0.0; + double hit_task_ms = 0.0; + double miss_task_ms = 0.0; + double model_slowdown_ratio = 1.0; + // True only when this exact executor/resource lane passed the model-output + // interference gate. Missing profile metadata defers tool speculation; + // token speculation is never disabled or replaced with AR decode. + bool decode_interference_qualified = false; + // Physical relationship between the tool accelerator and the model's + // primary accelerator. Production child executors may use CPU/I/O or a + // separate physical accelerator, but never the model's accelerator. + std::string accelerator_relation = "unspecified"; +}; + +struct ToolSpeculationAdmission { + bool admitted = false; + int resource_percentage = 0; + double expected_task_ms = 0.0; + double expected_speedup = 1.0; + bool decode_interference_qualified = false; + std::string accelerator_relation = "unspecified"; + std::string reason; +}; + +// Runtime policy loaded from a qualification report's `path_summary`. This +// keeps backend-specific interference measurements out of hard-coded engine +// heuristics. +class ToolSpeculationPolicy { +public: + bool load_file(const std::string & path, std::string & error); + bool load_json(const json & report, std::string & error); + + ToolSpeculationAdmission choose( + double confidence, + double max_model_slowdown_ratio) const; + + bool empty() const { return lanes_.empty(); } + double baseline_task_ms() const { return baseline_task_ms_; } + const std::vector & lanes() const { return lanes_; } + const std::string & profile_status() const { return profile_status_; } + const std::string & executor_contract() const { return executor_contract_; } + +private: + std::vector lanes_; + double baseline_task_ms_ = 0.0; + std::string profile_status_ = "qualified"; + std::string executor_contract_; +}; + +struct ToolSpeculationConfig { + std::string executor_path; + std::string profile_path; + std::vector allowed_tools; + ToolSpeculationPolicy policy; + int timeout_ms = 60000; + int cancel_grace_ms = 100; + size_t max_result_bytes = 1024 * 1024; + double max_model_slowdown_ratio = 1.20; + // Optional child-process CPU lane. Startup verifies that these logical + // CPUs are disjoint from the model process affinity; every child is pinned + // and re-read before its request payload is released. + std::vector cpu_affinity; + std::vector model_cpu_affinity; + bool cpu_affinity_isolated = false; + bool enabled() const { + return !executor_path.empty() && !allowed_tools.empty() && + !policy.empty(); + } + const char * execution_mode() const { + return executor_path.empty() + ? "disabled" + : cpu_affinity.empty() + ? "child_process" + : "child_process_cpu_affinity"; + } + bool allows(const std::string & name) const; +}; + +// Report whether this build can close every non-protocol descriptor before +// executing an untrusted tool child. Unsupported platforms fail closed. +bool tool_speculation_executor_isolation_supported(); + +// Capture the model process affinity and fail closed unless it is physically +// disjoint from the configured child executor CPUs. No-op when no CPU lane is +// requested. +bool qualify_tool_speculation_cpu_affinity(ToolSpeculationConfig & config, + std::string & error); + +// One request-scoped attempt. The configured executable is invoked without a +// shell and receives one JSON request on stdin. It must emit one JSON envelope +// on stdout: {"ok":true,"result":...}. Stdin remains open for a later +// `commit` (exact match; promote checkpointed remainder to the authoritative +// 100% lane) or `cancel` control record. A thin executable may forward this +// protocol to a persistent warm tool pool, keeping GPU initialization outside +// the request's critical path. +class ToolSpeculationAttempt { +public: + ToolSpeculationAttempt(const ToolSpeculationAttempt &) = delete; + ToolSpeculationAttempt & operator=(const ToolSpeculationAttempt &) = delete; + ~ToolSpeculationAttempt(); + + static std::unique_ptr create( + const ToolSpeculationConfig & config, + const ToolSpeculationPrediction & prediction, + const std::string & request_id); + + // Launch admitted work. Deferred and launch-failed attempts still return + // metadata through resolve(), so an opted-in client can see why it must + // execute the authoritative tool normally. + void start(); + + // Exact-match one authoritative call, expose a successful private result, + // or discard/cancel it. This method is single-use. + json resolve(const std::vector & authoritative_calls); + + // Cancel without exposing a result (disconnect, generation failure, etc.). + json cancel(const std::string & reason); + + bool admitted() const { return admission_.admitted; } + bool running() const { return running_; } +private: + ToolSpeculationAttempt(const ToolSpeculationConfig & config, + const ToolSpeculationPrediction & prediction, + const std::string & request_id); + + json base_metadata() const; + bool send_control(const char * operation); + void terminate_executor(bool allow_control_grace = false); + bool collect_executor_result(json & result, + double & wait_ms, + std::string & error); + + ToolSpeculationConfig config_; + ToolSpeculationPrediction prediction_; + std::string request_id_; + ToolSpeculationAdmission admission_; + std::chrono::steady_clock::time_point started_at_{}; + bool started_ = false; + bool running_ = false; + bool resolved_ = false; + std::string launch_error_; +#if !defined(_WIN32) + int child_stdin_fd_ = -1; + int child_stdout_fd_ = -1; + int child_pid_ = -1; +#endif +}; + +// Custom SSE extension emitted only for requests that supplied +// `tool_speculation`. Non-streaming responses use the same object under the +// top-level `dflash_tool_speculation` key. +std::string render_tool_speculation_sse(ApiFormat api_format, + const std::string & request_id, + const std::string & model, + const json & metadata); + +} // namespace dflash::common diff --git a/server/test/smoke_qwen3_tool_predictor_ipc.cpp b/server/test/smoke_qwen3_tool_predictor_ipc.cpp new file mode 100644 index 000000000..6af577032 --- /dev/null +++ b/server/test/smoke_qwen3_tool_predictor_ipc.cpp @@ -0,0 +1,203 @@ +#include "server/native_semantic_tool_predictor.h" + +#include +#include +#include +#include +#include + +using namespace dflash::common; + +namespace { + +struct Case { + const char * id; + const char * prompt; + const char * expected_name; + ordered_json expected_arguments; +}; + +json production_tools() { + return json::parse(R"json( +[ + {"type":"function","function":{"name":"get_weather","description":"Get current weather for one city.","parameters":{"type":"object","properties":{"city":{"type":"string"},"unit":{"type":"string","enum":["celsius","fahrenheit"]}},"required":["city","unit"],"additionalProperties":false}}}, + {"type":"function","function":{"name":"get_stock_quote","description":"Get the latest market quote for a ticker symbol.","parameters":{"type":"object","properties":{"symbol":{"type":"string"}},"required":["symbol"],"additionalProperties":false}}}, + {"type":"function","function":{"name":"search_documents","description":"Search indexed documents.","parameters":{"type":"object","properties":{"query":{"type":"string"},"limit":{"type":"integer","minimum":1,"maximum":20}},"required":["query","limit"],"additionalProperties":false}}}, + {"type":"function","function":{"name":"calculate","description":"Evaluate one arithmetic expression.","parameters":{"type":"object","properties":{"expression":{"type":"string"}},"required":["expression"],"additionalProperties":false}}}, + {"type":"function","function":{"name":"lookup_order","description":"Look up an order by its identifier.","parameters":{"type":"object","properties":{"order_id":{"type":"string"}},"required":["order_id"],"additionalProperties":false}}}, + {"type":"function","function":{"name":"translate_text","description":"Translate text to a target language.","parameters":{"type":"object","properties":{"text":{"type":"string"},"target_language":{"type":"string"}},"required":["text","target_language"],"additionalProperties":false}}}, + {"type":"function","function":{"name":"plan_route","description":"Plan a route between two places.","parameters":{"type":"object","properties":{"origin":{"type":"string"},"destination":{"type":"string"},"mode":{"type":"string","enum":["car","walk","transit"]}},"required":["origin","destination","mode"],"additionalProperties":false}}}, + {"type":"function","function":{"name":"read_file","description":"Read a UTF-8 text file from the workspace.","parameters":{"type":"object","properties":{"path":{"type":"string"}},"required":["path"],"additionalProperties":false}}} +] +)json"); +} + +std::optional make_request( + const std::string & prompt, + const json & tools, + int32_t max_tokens, + std::string & error) { + return build_semantic_tool_predictor_request( + json::array({{ + {"role", "user"}, + {"content", prompt}, + }}), + tools, "required", "native-qwen3", max_tokens, error); +} + +std::vector production_cases() { + return { + {"weather_rome", "What is the weather in Rome? Use Celsius.", + "get_weather", {{"city", "Rome"}, {"unit", "celsius"}}}, + {"weather_boston", "Check Boston weather in Fahrenheit.", + "get_weather", {{"city", "Boston"}, {"unit", "fahrenheit"}}}, + {"stock_nvda", "Get the latest quote for NVDA.", + "get_stock_quote", {{"symbol", "NVDA"}}}, + {"stock_amd", "Look up AMD's current stock quote.", + "get_stock_quote", {{"symbol", "AMD"}}}, + {"search_rocm", + "Search documents for 'ROCm graph replay' and return at most 5 results.", + "search_documents", {{"query", "ROCm graph replay"}, {"limit", 5}}}, + {"search_tool", + "Find the top 3 documents about speculative tool execution.", + "search_documents", + {{"query", "speculative tool execution"}, {"limit", 3}}}, + {"calculate", "Calculate (73.5 * 4) / 7.", + "calculate", {{"expression", "(73.5 * 4) / 7"}}}, + {"order", "Look up order LBX-2048-A.", + "lookup_order", {{"order_id", "LBX-2048-A"}}}, + {"translate", "Translate 'the server is ready' to Italian.", + "translate_text", + {{"text", "the server is ready"}, {"target_language", "Italian"}}}, + {"route", + "Plan a walking route from Termini Station to the Colosseum.", + "plan_route", + {{"origin", "Termini Station"}, + {"destination", "the Colosseum"}, + {"mode", "walk"}}}, + {"read_file", "Read the file docs/production.md.", + "read_file", {{"path", "docs/production.md"}}}, + {"punctuation", + "Search for the exact phrase 'R9700 + Strix: 0731/DS4' with limit 4.", + "search_documents", + {{"query", "R9700 + Strix: 0731/DS4"}, {"limit", 4}}}, + }; +} + +bool arguments_equal(const ordered_json & left, const ordered_json & right) { + // Object key order is irrelevant to tool-call semantics. Convert both + // ordered objects to the canonical map-backed representation first. + return json::parse(left.dump()) == json::parse(right.dump()); +} + +void print_prediction(const char * id, const SemanticToolPrediction & prediction, + const char * expected_name, + const ordered_json & expected_arguments, + const std::string & generated) { + const json output = { + {"id", id}, + {"ok", prediction.ok}, + {"error", prediction.error}, + {"wall_ms", prediction.wall_ms}, + {"name", prediction.call.name}, + {"arguments", prediction.call.arguments}, + {"generated", generated}, + {"name_match", prediction.ok && prediction.call.name == expected_name}, + {"exact_match", prediction.ok && prediction.call.name == expected_name && + arguments_equal(prediction.call.arguments, + expected_arguments)}, + }; + std::printf("%s\n", output.dump().c_str()); + std::fflush(stdout); +} + +} // namespace + +int main(int argc, char ** argv) { + if (argc < 4) { + std::fprintf(stderr, + "usage: %s [prompt]\n", + argv[0]); + return 2; + } + + SemanticToolPredictorConfig config; + config.native_model_path = argv[1]; + config.native_ipc_bin = argv[2]; + config.native_gpu = std::atoi(argv[3]); + config.native_max_ctx = 4096; + config.max_tokens = 96; + + std::string error; + auto predictor = NativeSemanticToolPredictor::create(config, error); + if (!predictor) { + std::fprintf(stderr, "predictor start failed: %s\n", error.c_str()); + return 1; + } + + const json tools = production_tools(); + if (argc > 4) { + const std::string prompt = argv[4]; + std::string generated; + const auto request = make_request( + prompt, tools, config.max_tokens, error); + if (!request.has_value()) { + std::fprintf(stderr, "request rejected: %s\n", error.c_str()); + return 1; + } + const auto prediction = predictor->predict(*request, tools, &generated); + print_prediction("custom", prediction, "", ordered_json::object(), + generated); + return prediction.ok ? 0 : 1; + } + + const std::vector cases = production_cases(); + constexpr size_t kMinimumExactMatches = 9; + size_t valid = 0; + size_t name_matches = 0; + size_t exact_matches = 0; + std::vector walls; + for (const Case & test_case : cases) { + std::string generated; + const auto request = make_request( + test_case.prompt, tools, config.max_tokens, error); + if (!request.has_value()) { + std::fprintf(stderr, "request rejected: %s\n", error.c_str()); + return 1; + } + const auto prediction = predictor->predict(*request, tools, &generated); + print_prediction(test_case.id, prediction, test_case.expected_name, + test_case.expected_arguments, generated); + valid += prediction.ok ? 1 : 0; + name_matches += prediction.ok && + prediction.call.name == test_case.expected_name ? 1 : 0; + exact_matches += prediction.ok && + prediction.call.name == test_case.expected_name && + arguments_equal(prediction.call.arguments, + test_case.expected_arguments) + ? 1 : 0; + walls.push_back(prediction.wall_ms); + } + std::sort(walls.begin(), walls.end()); + const double wall_p50 = walls.empty() + ? 0.0 : 0.5 * (walls[(walls.size() - 1) / 2] + walls[walls.size() / 2]); + const json summary = { + {"requests", cases.size()}, + {"valid", valid}, + {"name_matches", name_matches}, + {"exact_matches", exact_matches}, + {"minimum_exact_matches", kMinimumExactMatches}, + {"name_accuracy", cases.empty() ? 0.0 + : static_cast(name_matches) / + static_cast(cases.size())}, + {"exact_accuracy", cases.empty() ? 0.0 + : static_cast(exact_matches) / + static_cast(cases.size())}, + {"wall_p50_ms", wall_p50}, + }; + std::printf("%s\n", summary.dump().c_str()); + return valid == cases.size() && name_matches == cases.size() && + exact_matches >= kMinimumExactMatches + ? 0 + : 1; +} diff --git a/server/test/test_semantic_tool_hint.cpp b/server/test/test_semantic_tool_hint.cpp new file mode 100644 index 000000000..2ae4d25a6 --- /dev/null +++ b/server/test/test_semantic_tool_hint.cpp @@ -0,0 +1,435 @@ +#include "CppUnitTestFramework.hpp" + +#include "common/qwen3_tool_predictor_ipc.h" +#include "server/semantic_tool_hint.h" + +#include +#include +#include +#include + +#if !defined(_WIN32) +#include +#endif + +namespace { +struct SemanticToolHintFixture {}; +} + +using namespace dflash::common; + +static json weather_tools() { + return json::array({{ + {"type", "function"}, + {"function", { + {"name", "get_weather"}, + {"parameters", { + {"type", "object"}, + {"properties", { + {"city", {{"type", "string"}}}, + {"unit", {{"type", "string"}}}, + }}, + }}, + }}, + }}); +} + +TEST_CASE(SemanticToolHintFixture, parses_qwen_openai_tool_call_semantics) { + const json response = { + {"choices", json::array({{ + {"message", { + {"role", "assistant"}, + {"content", ""}, + {"tool_calls", json::array({{ + {"type", "function"}, + {"function", { + {"name", "get_weather"}, + {"arguments", "{\"city\":\"Rome\",\"unit\":\"celsius\"}"}, + }}, + }})}, + }}, + }})}, + }; + SemanticToolCall call; + std::string error; + CHECK(parse_semantic_tool_prediction( + response, weather_tools(), call, error)); + CHECK(error.empty()); + CHECK(call.name == "get_weather"); + CHECK(call.arguments.dump() == + "{\"city\":\"Rome\",\"unit\":\"celsius\"}"); +} + +TEST_CASE(SemanticToolHintFixture, rejects_unknown_predicted_function) { + const json response = { + {"choices", json::array({{ + {"message", { + {"tool_calls", json::array({{ + {"function", { + {"name", "delete_everything"}, + {"arguments", "{}"}, + }}, + }})}, + }}, + }})}, + }; + SemanticToolCall call; + std::string error; + CHECK(!parse_semantic_tool_prediction( + response, weather_tools(), call, error)); + CHECK(error == "predictor_selected_unknown_function"); +} + +TEST_CASE(SemanticToolHintFixture, parses_strict_json_content_fallback) { + const json response = { + {"choices", json::array({{ + {"message", { + {"content", + " {\"name\":\"get_weather\",\"arguments\":{\"city\":\"Rome\"}}\n"}, + }}, + }})}, + }; + SemanticToolCall call; + std::string error; + CHECK(parse_semantic_tool_prediction( + response, weather_tools(), call, error)); + CHECK(call.name == "get_weather"); + CHECK(call.arguments["city"] == "Rome"); +} + +TEST_CASE(SemanticToolHintFixture, rejects_ambiguous_sidecar_calls) { + const json function = { + {"name", "get_weather"}, + {"arguments", "{\"city\":\"Rome\"}"}, + }; + const json response = { + {"choices", json::array({{ + {"message", { + {"content", + "{\"name\":\"get_weather\",\"arguments\":{\"city\":\"Rome\"}}"}, + {"tool_calls", json::array({ + {{"function", function}}, + {{"function", function}}, + })}, + }}, + }})}, + }; + SemanticToolCall call; + std::string error; + CHECK(!parse_semantic_tool_prediction( + response, weather_tools(), call, error)); + CHECK(error == "predictor_response_requires_single_tool_call"); +} + +TEST_CASE(SemanticToolHintFixture, rejects_prose_wrapped_json_content) { + const json response = { + {"choices", json::array({{ + {"message", { + {"content", + "call {\"name\":\"get_weather\",\"arguments\":{\"city\":\"Rome\"}}"}, + }}, + }})}, + }; + SemanticToolCall call; + std::string error; + CHECK(!parse_semantic_tool_prediction( + response, weather_tools(), call, error)); + CHECK(error == "predictor_response_has_no_valid_call"); +} + +TEST_CASE(SemanticToolHintFixture, materializes_declared_optional_defaults) { + json tools = weather_tools(); + tools[0]["function"]["parameters"]["properties"]["unit"]["default"] = + "celsius"; + SemanticToolCall call; + call.name = "get_weather"; + call.arguments = ordered_json::parse(R"({"city":"Rome"})"); + std::string error; + + CHECK(materialize_declared_tool_defaults(tools, call, error)); + CHECK(error.empty()); + CHECK(call.arguments.dump() == + R"({"city":"Rome","unit":"celsius"})"); +} + +TEST_CASE(SemanticToolHintFixture, explicit_prediction_beats_schema_default) { + json tools = weather_tools(); + tools[0]["function"]["parameters"]["properties"]["unit"]["default"] = + "celsius"; + SemanticToolCall call; + call.name = "get_weather"; + call.arguments = ordered_json::parse( + R"({"city":"Rome","unit":"fahrenheit"})"); + std::string error; + + CHECK(materialize_declared_tool_defaults(tools, call, error)); + CHECK(call.arguments["unit"] == "fahrenheit"); +} + +TEST_CASE(SemanticToolHintFixture, predictor_request_forwards_only_semantics) { + const json target = { + {"model", "deepseek-v4-flash"}, + {"messages", json::array({{{"role", "user"}, {"content", "weather"}}})}, + {"tools", weather_tools()}, + {"tool_choice", "required"}, + {"tool_speculation", {{"name", "unsafe"}}}, + {"prefix_cache", {{"scope", "full"}}}, + }; + std::string error; + const auto request = build_semantic_tool_predictor_request( + target["messages"], target["tools"], target["tool_choice"], + "Qwen3-0.6B", 32, error); + CHECK(error.empty()); + REQUIRE(request.has_value()); + const json & payload = request->payload(); + CHECK(payload["model"] == "Qwen3-0.6B"); + CHECK(payload["max_tokens"] == 32); + CHECK(payload["tool_choice"] == "required"); + CHECK(!payload.contains("tool_speculation")); + CHECK(!payload.contains("prefix_cache")); +} + +TEST_CASE(SemanticToolHintFixture, predictor_request_rejects_oversized_fields_before_copy) { + const json messages = json::array({{ + {"role", "user"}, + {"content", std::string(300U * 1024U, 'x')}, + }}); + std::string error; + const auto request = build_semantic_tool_predictor_request( + messages, weather_tools(), nullptr, "Qwen3-0.6B", 32, error); + CHECK(!request.has_value()); + CHECK(error == "predictor_request_too_large"); + + const json small_messages = json::array({{ + {"role", "user"}, {"content", "weather"}, + }}); + json oversized_tools = weather_tools(); + oversized_tools[0]["function"]["description"] = + std::string(300U * 1024U, 'x'); + const auto oversized_tool_request = build_semantic_tool_predictor_request( + small_messages, oversized_tools, nullptr, "Qwen3-0.6B", 32, error); + CHECK(!oversized_tool_request.has_value()); + CHECK(error == "predictor_request_too_large"); +} + +TEST_CASE(SemanticToolHintFixture, predictor_request_canonicalizes_anthropic_tools) { + const json tools = json::array({{ + {"name", "get_weather"}, + {"description", "Read weather"}, + {"input_schema", { + {"type", "object"}, + {"properties", {{"city", {{"type", "string"}}}}}, + }}, + }}); + std::string error; + const auto request = build_semantic_tool_predictor_request( + json::array({{{"role", "user"}, {"content", "weather"}}}), + tools, nullptr, "Qwen3-0.6B", 32, error); + REQUIRE(request.has_value()); + const json & canonical = request->payload()["tools"][0]; + CHECK(canonical["type"] == "function"); + CHECK(canonical["function"]["name"] == "get_weather"); + CHECK(canonical["function"].contains("parameters")); + CHECK(!canonical["function"].contains("input_schema")); +} + +TEST_CASE(SemanticToolHintFixture, predictor_request_handles_non_string_tool_names) { + json tools = json::array({ + {{"type", "function"}, {"function", {{"name", 42}}}}, + {{"name", false}, {"input_schema", {{"type", "object"}}}}, + weather_tools()[0], + }); + std::string error; + const auto request = build_semantic_tool_predictor_request( + json::array({{{"role", "user"}, {"content", "weather"}}}), + tools, nullptr, "Qwen3-0.6B", 32, error); + REQUIRE(request.has_value()); + CHECK(error.empty()); + REQUIRE(request->payload()["tools"].size() == 1); + CHECK(request->payload()["tools"][0]["function"]["name"] == + "get_weather"); + + tools = json::array({ + {{"type", "function"}, {"function", {{"name", 42}}}}, + }); + CHECK(!build_semantic_tool_predictor_request( + json::array({{{"role", "user"}, {"content", "weather"}}}), + tools, nullptr, "Qwen3-0.6B", 32, error).has_value()); + CHECK(error == "predictor_request_has_no_valid_tools"); +} + +TEST_CASE(SemanticToolHintFixture, native_predictor_config_is_independent_of_http) { + SemanticToolPredictorConfig config; + config.native_model_path = "/models/qwen3-0.6b.gguf"; + config.native_ipc_bin = "/opt/lucebox/backend_ipc_daemon"; + CHECK(config.native_enabled()); + CHECK(!config.http_enabled()); + CHECK(config.enabled()); +} + +TEST_CASE(SemanticToolHintFixture, native_prompt_honors_tool_choice_none) { + std::string error; + const auto request = build_semantic_tool_predictor_request( + json::array({{{"role", "user"}, {"content", "Do not call a tool"}}}), + weather_tools(), "none", "Qwen3-0.6B", 32, error); + REQUIRE(request.has_value()); + CHECK(build_native_semantic_tool_predictor_prompt(*request, error).empty()); + CHECK(error == "native_predictor_tool_choice_none"); +} + +TEST_CASE(SemanticToolHintFixture, native_prompt_uses_qwen_tool_contract) { + std::string error; + const auto request = build_semantic_tool_predictor_request( + json::array({{{"role", "user"}, + {"content", "What is the weather in Rome?"}}}), + weather_tools(), "required", "Qwen3-0.6B", 32, error); + REQUIRE(request.has_value()); + const std::string prompt = + build_native_semantic_tool_predictor_prompt(*request, error); + CHECK(error.empty()); + CHECK(prompt.find("You must call exactly one available function.") != + std::string::npos); + CHECK(prompt.find("get_weather") != std::string::npos); + CHECK(prompt.find("What is the weather in Rome?") != std::string::npos); + CHECK(prompt.find("{\"name\": , \"arguments\":") != + std::string::npos); + CHECK(prompt.find("") == + std::string::npos); + CHECK(prompt.find("\n\n") != std::string::npos); +} + +TEST_CASE(SemanticToolHintFixture, native_prompt_honors_preprocessing_deadline) { + std::string error; + const auto request = build_semantic_tool_predictor_request( + json::array({{{"role", "user"}, {"content", "weather"}}}), + weather_tools(), nullptr, "Qwen3-0.6B", 32, error); + REQUIRE(request.has_value()); + const auto expired = std::chrono::steady_clock::now() - + std::chrono::milliseconds(1); + CHECK(build_native_semantic_tool_predictor_prompt( + *request, error, &expired).empty()); + CHECK(error == "native_predictor_timeout"); +} + +TEST_CASE(SemanticToolHintFixture, parses_native_qwen_xml_semantics) { + const std::string generated = + "\n" + "\n" + "\nRome\n\n" + "\ncelsius\n\n" + "\n" + ""; + SemanticToolCall call; + std::string error; + CHECK(parse_native_semantic_tool_prediction( + generated, weather_tools(), call, error)); + CHECK(error.empty()); + CHECK(call.name == "get_weather"); + CHECK(call.arguments.dump() == + "{\"city\":\"Rome\",\"unit\":\"celsius\"}"); +} + +TEST_CASE(SemanticToolHintFixture, repairs_qwen_missing_outer_call_brace) { + const json tools = json::array({{ + {"type", "function"}, + {"function", { + {"name", "get_stock_quote"}, + {"parameters", { + {"type", "object"}, + {"properties", {{"symbol", {{"type", "string"}}}}}, + {"required", json::array({"symbol"})}, + }}, + }}, + }}); + const std::string generated = + "\n" + " \"name\": \"get_stock_quote\",\n" + " \"arguments\": {\"symbol\": \"NVDA\"}\n" + "\"}<|im_end|>"; + SemanticToolCall call; + std::string error; + CHECK(parse_native_semantic_tool_prediction( + generated, tools, call, error)); + CHECK(error.empty()); + CHECK(call.name == "get_stock_quote"); + CHECK(call.arguments.dump() == "{\"symbol\":\"NVDA\"}"); +} + +TEST_CASE(SemanticToolHintFixture, maps_bare_arguments_only_for_one_tool) { + const json one_tool = json::array({{ + {"name", "benchmark_cpu_sparse"}, + {"parameters", { + {"type", "object"}, + {"properties", {{"iterations", {{"type", "integer"}}}}}, + {"required", json::array({"iterations"})}, + }}, + }}); + SemanticToolCall call; + std::string error; + CHECK(parse_native_semantic_tool_prediction( + "\n{\"iterations\":172452}\n", + one_tool, call, error)); + CHECK(error.empty()); + CHECK(call.name == "benchmark_cpu_sparse"); + CHECK(call.arguments.dump() == "{\"iterations\":172452}"); + + json two_tools = one_tool; + two_tools.push_back({ + {"name", "other"}, + {"parameters", {{"type", "object"}}}, + }); + CHECK(!parse_native_semantic_tool_prediction( + "{\"iterations\":172452}", two_tools, call, error)); +} + +TEST_CASE(SemanticToolHintFixture, native_parser_rejects_multiple_calls) { + const std::string call = + "Rome" + "celsius"; + SemanticToolCall prediction; + std::string error; + CHECK(!parse_native_semantic_tool_prediction( + call + call, weather_tools(), prediction, error)); + CHECK(error == "native_predictor_response_has_multiple_calls"); +} + +TEST_CASE(SemanticToolHintFixture, native_ipc_response_obeys_hard_deadline) { +#if !defined(_WIN32) + int descriptors[2] = {-1, -1}; + CHECK(::pipe(descriptors) == 0); + const auto started = std::chrono::steady_clock::now(); + std::vector output; + std::string error; + CHECK(!read_qwen3_tool_predictor_response( + descriptors[0], 8, 25, output, error)); + const double elapsed_ms = std::chrono::duration( + std::chrono::steady_clock::now() - started).count(); + ::close(descriptors[0]); + ::close(descriptors[1]); + CHECK(error == "native_predictor_timeout"); + CHECK(output.empty()); + CHECK(elapsed_ms < 500.0); +#else + CHECK(true); +#endif +} + +TEST_CASE(SemanticToolHintFixture, native_ipc_response_reads_complete_payload) { +#if !defined(_WIN32) + int descriptors[2] = {-1, -1}; + CHECK(::pipe(descriptors) == 0); + const int32_t response[] = {0, 3, 17, 18, 19}; + CHECK(::write(descriptors[1], response, sizeof(response)) == + static_cast(sizeof(response))); + ::close(descriptors[1]); + std::vector output; + std::string error; + CHECK(read_qwen3_tool_predictor_response( + descriptors[0], 8, 100, output, error)); + ::close(descriptors[0]); + CHECK(error.empty()); + CHECK(output == std::vector({17, 18, 19})); +#else + CHECK(true); +#endif +} diff --git a/server/test/test_server_unit.cpp b/server/test/test_server_unit.cpp index 8373859dc..a66dfa128 100644 --- a/server/test/test_server_unit.cpp +++ b/server/test/test_server_unit.cpp @@ -2002,6 +2002,40 @@ TEST_CASE(ServerUnitFixture, test_resolve_deepseek_chat_markers) { unlink(path.c_str()); } +TEST_CASE(ServerUnitFixture, test_tokenizer_added_token_search_obeys_deadline) { + const std::string path = write_deepseek_marker_tokenizer_fixture(); + Tokenizer tokenizer; + TEST_ASSERT(tokenizer.load_from_gguf(path.c_str())); + const std::string long_input(32U * 1024U * 1024U, 'x'); + const auto started = std::chrono::steady_clock::now(); + const auto deadline = started + std::chrono::milliseconds(1); + std::vector output; + TEST_ASSERT(!tokenizer.encode_until(long_input, deadline, output)); + const auto elapsed = std::chrono::steady_clock::now() - started; + TEST_ASSERT(elapsed < std::chrono::seconds(1)); + TEST_ASSERT(output.empty()); + unlink(path.c_str()); +} + +TEST_CASE(ServerUnitFixture, test_tokenizer_rejects_oversized_added_token) { + gguf_context * fixture = gguf_init_empty(); + const std::string oversized(4097U, 'x'); + const char * tokens[] = {"x", oversized.c_str()}; + const uint32_t token_types[] = {1, 3}; + gguf_set_arr_str(fixture, "tokenizer.ggml.tokens", tokens, 2); + gguf_set_arr_data(fixture, "tokenizer.ggml.token_type", GGUF_TYPE_UINT32, + token_types, sizeof(token_types)); + gguf_set_val_str(fixture, "tokenizer.ggml.model", "gpt2"); + gguf_set_val_str(fixture, "tokenizer.ggml.pre", "qwen35"); + const std::string path = "/tmp/dflash_test_oversized_special_token.gguf"; + gguf_write_to_file(fixture, path.c_str(), /*only_meta=*/false); + gguf_free(fixture); + + Tokenizer tokenizer; + TEST_ASSERT(!tokenizer.load_from_gguf(path.c_str())); + unlink(path.c_str()); +} + TEST_CASE(ServerUnitFixture, test_hash_prefix_deterministic) { std::vector ids = {100, 200, 300, 400, 500}; auto h1 = hash_prefix(ids.data(), (int)ids.size()); @@ -2486,6 +2520,23 @@ TEST_CASE(ServerUnitFixture, test_pflash_raw_body_preserved) { TEST_ASSERT(req.raw_body["temperature"].get() > 0.6f); } +TEST_CASE(ServerUnitFixture, test_tool_speculation_defaults_to_automatic_prediction) { + ParsedRequest req; + TEST_ASSERT(req.automatic_tool_speculation_enabled); +} + +TEST_CASE(ServerUnitFixture, test_tool_choice_none_disables_speculation) { + TEST_ASSERT(http_detail::tool_choice_disables_tool_calls("none")); + TEST_ASSERT(http_detail::tool_choice_disables_tool_calls( + json{{"type", "none"}})); + TEST_ASSERT(!http_detail::tool_choice_disables_tool_calls(nullptr)); + TEST_ASSERT(!http_detail::tool_choice_disables_tool_calls("auto")); + TEST_ASSERT(!http_detail::tool_choice_disables_tool_calls("required")); + TEST_ASSERT(!http_detail::tool_choice_disables_tool_calls( + json{{"type", "function"}, + {"function", {{"name", "lookup"}}}})); +} + TEST_CASE(ServerUnitFixture, test_parse_request_sampler_applies_defaults_and_overrides) { SamplingDefaults defaults; defaults.has_temperature = true; @@ -2875,6 +2926,39 @@ TEST_CASE(ServerUnitFixture, test_normalize_responses_tool_followup_messages) { TEST_ASSERT(chat_msgs[3].tool_call_id == call_id); TEST_ASSERT(chat_msgs[3].content == "Process exited with code 0"); } + const json predictor_messages = + http_detail::canonical_predictor_messages(std::move(chat_msgs)); + TEST_ASSERT(predictor_messages.size() == 4); + TEST_ASSERT(predictor_messages[2]["role"] == "assistant"); + TEST_ASSERT(predictor_messages[2]["content"] == raw_tool_call); + TEST_ASSERT(predictor_messages[3]["role"] == "tool"); + TEST_ASSERT(predictor_messages[3]["content"] == + "Process exited with code 0"); + TEST_ASSERT(predictor_messages[3]["tool_call_id"] == call_id); + + const json predictor_tools = json::array({{ + {"type", "function"}, + {"function", { + {"name", "exec_command"}, + {"parameters", { + {"type", "object"}, + {"properties", {{"cmd", {{"type", "string"}}}}}, + }}, + }}, + }}); + std::string predictor_error; + const auto predictor_request = build_semantic_tool_predictor_request( + predictor_messages, predictor_tools, "required", "native-qwen3", 32, + predictor_error); + TEST_ASSERT(predictor_request.has_value()); + const std::string predictor_prompt = + build_native_semantic_tool_predictor_prompt( + *predictor_request, predictor_error); + TEST_ASSERT(predictor_error.empty()); + TEST_ASSERT(predictor_prompt.find(raw_tool_call) != std::string::npos); + TEST_ASSERT(predictor_prompt.find( + "\nProcess exited with code 0\n") != + std::string::npos); } // ═══════════════════════════════════════════════════════════════════════ @@ -4084,6 +4168,52 @@ TEST_CASE(ServerUnitFixture, test_backend_ipc_rejects_file_work_dir) { unlink(file_path.c_str()); } +TEST_CASE(ServerUnitFixture, test_backend_ipc_rejects_public_work_dir) { + const std::string dir_path = + "/tmp/dflash_test_backend_ipc_public_work_dir"; + rmdir(dir_path.c_str()); + TEST_ASSERT(mkdir(dir_path.c_str(), 0755) == 0); + TEST_ASSERT(chmod(dir_path.c_str(), 0755) == 0); + + BackendIpcLaunchConfig cfg; + cfg.bin = "/bin/true"; + cfg.payload_path = "/tmp/dflash_test_backend_ipc_payload"; + cfg.work_dir = dir_path; + cfg.require_private_work_dir = true; + + BackendIpcProcess proc; + TEST_ASSERT(!proc.start(cfg)); + TEST_ASSERT(!proc.active()); + rmdir(dir_path.c_str()); +} + +TEST_CASE(ServerUnitFixture, test_backend_ipc_readiness_timeout) { + const std::string script_path = + "/tmp/dflash_test_backend_ipc_readiness_timeout.sh"; + unlink(script_path.c_str()); + int fd = open(script_path.c_str(), O_CREAT | O_TRUNC | O_WRONLY, 0700); + TEST_ASSERT(fd >= 0); + if (fd < 0) return; + const char script[] = "#!/bin/sh\nwhile :; do :; done\n"; + TEST_ASSERT(write(fd, script, sizeof(script) - 1) == + static_cast(sizeof(script) - 1)); + close(fd); + TEST_ASSERT(chmod(script_path.c_str(), 0700) == 0); + + BackendIpcLaunchConfig cfg; + cfg.bin = script_path; + cfg.payload_path = "/tmp/dflash_test_backend_ipc_payload"; + cfg.readiness_timeout_ms = 50; + const auto started = std::chrono::steady_clock::now(); + BackendIpcProcess proc; + TEST_ASSERT(!proc.start(cfg)); + const auto elapsed_ms = std::chrono::duration_cast( + std::chrono::steady_clock::now() - started).count(); + TEST_ASSERT(elapsed_ms < 1000); + TEST_ASSERT(!proc.active()); + unlink(script_path.c_str()); +} + TEST_CASE(ServerUnitFixture, test_backend_ipc_payload_pipe_round_trip) { int payload_pipe[2] = {-1, -1}; int status_pipe[2] = {-1, -1}; @@ -4836,6 +4966,132 @@ TEST_CASE(ServerUnitFixture, test_props_budget_envelope_shape) { TEST_ASSERT(body["server"]["props_schema"].get() == 2); } +TEST_CASE(ServerUnitFixture, test_props_tool_speculation_shape) { + ServerConfig cfg; + Tokenizer tok; + PrefixCache pc(0, tok); + ToolMemory tm; + + json body = build_props_body(cfg, pc, tm); + TEST_ASSERT(body.contains("tool_speculation")); + const json & disabled = body["tool_speculation"]; + TEST_ASSERT(!disabled["enabled"].get()); + TEST_ASSERT(!disabled["automatic_prediction_enabled"].get()); + TEST_ASSERT(disabled["prediction_source"].is_null()); + TEST_ASSERT(disabled["prediction_confidence"].is_null()); + TEST_ASSERT(disabled["profile_status"].is_null()); + TEST_ASSERT(disabled["executor_contract"].is_null()); + TEST_ASSERT(disabled["protocol"].get() == + "dflash.tool-speculation.v1"); + TEST_ASSERT(!disabled["client_prediction_required"].get()); + TEST_ASSERT(!disabled["client_result_handling_required"].get()); + TEST_ASSERT(disabled["preserves_token_speculation"].get()); + TEST_ASSERT(disabled["unqualified_lane_policy"].get() == + "defer"); + TEST_ASSERT(disabled["allowed_tools"].empty()); + TEST_ASSERT(disabled["compute_isolation"].get() == "none"); + TEST_ASSERT(!disabled["cpu_affinity_isolated"].get()); + TEST_ASSERT(disabled["tool_cpu_affinity"].empty()); + TEST_ASSERT(disabled["model_cpu_affinity"].empty()); + TEST_ASSERT(disabled["profile_lanes"].empty()); + + cfg.tool_speculation.executor_path = "/trusted/tool-adapter"; + cfg.tool_speculation.profile_path = "/measured/frontier.json"; + cfg.tool_speculation.allowed_tools = {"lookup"}; + std::string profile_error; + TEST_ASSERT(cfg.tool_speculation.policy.load_json(json{ + {"profile_status", "qualified"}, + {"executor", "child_process"}, + {"path_summary", { + {"25", { + {"accelerator_relation", "non_accelerator"}, + {"decode_interference_qualified", true}, + {"hit", { + {"control_task_mean_ms", 100.0}, + {"speculative_task_mean_ms", 80.0}, + {"model_slowdown_percent", 2.0}, + }}, + {"miss", { + {"control_task_mean_ms", 100.0}, + {"speculative_task_mean_ms", 101.0}, + {"model_slowdown_percent", 2.0}, + }}, + }}, + }}, + }, profile_error)); + + body = build_props_body(cfg, pc, tm); + const json & enabled = body["tool_speculation"]; + TEST_ASSERT(enabled["enabled"].get()); + TEST_ASSERT(!enabled["automatic_prediction_enabled"].get()); + TEST_ASSERT(enabled["client_prediction_required"].get()); + TEST_ASSERT(enabled["client_result_handling_required"].get()); + TEST_ASSERT(!enabled["predictor_decode_isolated"].get()); + TEST_ASSERT(enabled["profile_status"].get() == "qualified"); + TEST_ASSERT(enabled["executor_contract"].get() == + "child_process"); + TEST_ASSERT(enabled["allowed_tools"] == json::array({"lookup"})); + TEST_ASSERT(enabled["preserves_token_speculation"].get()); + TEST_ASSERT(enabled["unqualified_lane_policy"].get() == + "defer"); + TEST_ASSERT(enabled["profile_lanes"].size() == 1); + TEST_ASSERT(enabled["profile_lanes"][0] + ["resource_percentage"].get() == 25); + TEST_ASSERT(std::fabs(enabled["profile_lanes"][0] + ["model_slowdown_ratio"].get() - 1.02) < + 1e-9); + TEST_ASSERT(enabled["profile_lanes"][0] + ["decode_interference_qualified"].get()); + TEST_ASSERT(enabled["profile_lanes"][0] + ["accelerator_relation"].get() == + "non_accelerator"); + + cfg.semantic_tool_predictor.native_model_path = "/models/qwen3-0.6b.gguf"; + cfg.semantic_tool_predictor.native_ipc_bin = "/bin/backend-ipc"; + body = build_props_body(cfg, pc, tm); + const json & automatic = body["tool_speculation"]; + TEST_ASSERT(automatic["automatic_prediction_enabled"].get()); + TEST_ASSERT(!automatic["client_prediction_required"].get()); + TEST_ASSERT(automatic["client_result_handling_required"].get()); + TEST_ASSERT(automatic["prediction_source"].get() == + "native-qwen3"); + TEST_ASSERT(std::fabs( + automatic["prediction_confidence"].get() - 0.75) < 1e-9); + TEST_ASSERT(automatic["predictor_schedule"].get() == + "before-model"); + TEST_ASSERT(automatic["predictor_decode_isolated"].get()); + + cfg.semantic_tool_predictor.native_model_path.clear(); + cfg.semantic_tool_predictor.native_ipc_bin.clear(); + cfg.semantic_tool_predictor.url = "http://127.0.0.1:9000/v1/chat/completions"; + cfg.semantic_tool_predictor.model = "remote-predictor"; + body = build_props_body(cfg, pc, tm); + const json & remote = body["tool_speculation"]; + TEST_ASSERT(remote["automatic_prediction_enabled"].get()); + TEST_ASSERT(remote["prediction_source"].get() == + "remote-predictor"); + TEST_ASSERT(remote["predictor_schedule"].get() == + "before-model"); + TEST_ASSERT(remote["predictor_decode_isolated"].get()); + cfg.semantic_tool_predictor.url.clear(); + cfg.semantic_tool_predictor.model.clear(); + cfg.semantic_tool_predictor.native_model_path = "/models/qwen3-0.6b.gguf"; + cfg.semantic_tool_predictor.native_ipc_bin = "/bin/backend-ipc"; + + cfg.tool_speculation.cpu_affinity = {14, 30}; + cfg.tool_speculation.model_cpu_affinity = {0, 1, 2, 3}; + cfg.tool_speculation.cpu_affinity_isolated = true; + body = build_props_body(cfg, pc, tm); + const json & cpu_isolated = body["tool_speculation"]; + TEST_ASSERT(cpu_isolated["compute_isolation"].get() == + "disjoint_cpu_affinity"); + TEST_ASSERT(cpu_isolated["cpu_affinity_isolated"].get()); + TEST_ASSERT(cpu_isolated["tool_cpu_affinity"] == + json::array({14, 30})); + TEST_ASSERT(cpu_isolated["model_cpu_affinity"] == + json::array({0, 1, 2, 3})); +} + // ─── /props.runtime captures full config (§4.16) ────────────────────── // Snapshot/bench tooling reads /props.runtime wholesale into // result.json.server_info; this test pins the field set so additions diff --git a/server/test/test_tool_speculation.cpp b/server/test/test_tool_speculation.cpp new file mode 100644 index 000000000..118522fb9 --- /dev/null +++ b/server/test/test_tool_speculation.cpp @@ -0,0 +1,632 @@ +#include "CppUnitTestFramework.hpp" +#include "server/tool_speculation.h" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if !defined(_WIN32) +# include +# include +# include +# include +# if defined(__linux__) +# include +# include +# endif +#endif + +using dflash::common::ApiFormat; +using dflash::common::CanonicalToolInvocation; +using dflash::common::ToolCall; +using dflash::common::ToolSpeculationAttempt; +using dflash::common::ToolSpeculationConfig; +using dflash::common::ToolSpeculationPolicy; +using dflash::common::ToolSpeculationPrediction; +using dflash::common::build_tool_speculation_prediction; +using dflash::common::json; +using dflash::common::parse_tool_speculation_prediction; +using dflash::common::parse_tool_speculation_cpu_affinity; +using dflash::common::qualify_tool_speculation_cpu_affinity; +using dflash::common::render_tool_speculation_sse; + +namespace { +struct ToolSpeculationFixture {}; + +json policy_fixture(bool decode_interference_qualified = true) { + auto path = [decode_interference_qualified]( + double hit_task, double miss_task, + double slowdown_percent) { + return json{ + {"decode_interference_qualified", + decode_interference_qualified}, + {"accelerator_relation", "non_accelerator"}, + {"hit", { + {"control_task_mean_ms", 100.0}, + {"speculative_task_mean_ms", hit_task}, + {"model_slowdown_percent", slowdown_percent}, + }}, + {"miss", { + {"control_task_mean_ms", 100.0}, + {"speculative_task_mean_ms", miss_task}, + {"model_slowdown_percent", slowdown_percent}, + }}, + }; + }; + json fixture = { + {"profile_status", "qualified"}, + {"executor", "child_process"}, + {"path_summary", { + {"25", path(80.0, 101.0, 2.0)}, + {"50", path(60.0, 110.0, 7.0)}, + {"100", path(50.0, 130.0, 16.0)}, + }}, + }; + return fixture; +} + +ToolSpeculationConfig test_config(const std::string & executor = {}) { + ToolSpeculationConfig config; + config.executor_path = executor.empty() ? "/unused/executor" : executor; + config.profile_path = "fixture.json"; + config.allowed_tools = {"lookup"}; + config.timeout_ms = 1000; + config.cancel_grace_ms = 20; + config.max_model_slowdown_ratio = 1.20; + std::string error; + if (!config.policy.load_json(policy_fixture(), error)) { + throw std::runtime_error(error); + } + return config; +} + +ToolSpeculationPrediction prediction(double confidence = 0.9) { + ToolSpeculationPrediction value; + std::string error; + if (!CanonicalToolInvocation::from_parts( + "lookup", json{{"a", 1}, {"b", 2}}, value.call, error)) { + throw std::runtime_error(error); + } + value.confidence = confidence; + return value; +} + +#if !defined(_WIN32) +std::string make_executor_script(const std::string & body) { + char path[] = "/tmp/dflash-tool-spec-test-XXXXXX"; + const int fd = ::mkstemp(path); + if (fd < 0) throw std::runtime_error("mkstemp failed"); + const std::string script = "#!/bin/sh\nIFS= read -r request\n" + body; + size_t offset = 0; + while (offset < script.size()) { + const ssize_t count = ::write( + fd, script.data() + offset, script.size() - offset); + if (count <= 0) { + ::close(fd); + ::unlink(path); + throw std::runtime_error("script write failed"); + } + offset += static_cast(count); + } + if (::fchmod(fd, 0700) != 0) { + ::close(fd); + ::unlink(path); + throw std::runtime_error("chmod failed"); + } + ::close(fd); + return path; +} + +std::string make_temp_path() { + char path[] = "/tmp/dflash-tool-spec-observed-XXXXXX"; + const int fd = ::mkstemp(path); + if (fd < 0) throw std::runtime_error("mkstemp failed"); + ::close(fd); + return path; +} + +std::string read_text_file(const std::string & path) { + FILE * file = std::fopen(path.c_str(), "rb"); + if (!file) return {}; + std::string value; + char buffer[256]; + while (const size_t count = std::fread(buffer, 1, sizeof(buffer), file)) { + value.append(buffer, count); + } + std::fclose(file); + return value; +} +#endif +} // namespace + +TEST_CASE(ToolSpeculationFixture, canonical_identity_ignores_argument_order) { + CanonicalToolInvocation first; + CanonicalToolInvocation second; + std::string error; + CHECK(CanonicalToolInvocation::from_parts( + "lookup", json{{"b", 2}, {"a", 1}}, first, error)); + CHECK(CanonicalToolInvocation::from_parts( + "lookup", json{{"a", 1}, {"b", 2}}, second, error)); + CHECK(first == second); + CHECK(first.arguments_json == R"({"a":1,"b":2})"); +} + +TEST_CASE(ToolSpeculationFixture, engine_prediction_uses_canonical_boundary) { + ToolSpeculationPrediction value; + std::string error; + CHECK(build_tool_speculation_prediction( + "lookup", json{{"b", 2}, {"a", 1}}, 0.75, value, error)); + CHECK(error.empty()); + CHECK(value.call.name == "lookup"); + CHECK(value.call.arguments_json == "{\"a\":1,\"b\":2}"); + CHECK(std::fabs(value.confidence - 0.75) < 1e-9); + CHECK(!build_tool_speculation_prediction( + "lookup", json::array(), 0.75, value, error)); + CHECK(!build_tool_speculation_prediction( + "lookup", json::object(), 1.01, value, error)); +} + +TEST_CASE(ToolSpeculationFixture, prediction_requires_declared_tool) { + const json tools = json::array({{ + {"type", "function"}, + {"function", {{"name", "lookup"}}}, + }}); + const json request = { + {"call", { + {"name", "lookup"}, + {"arguments", {{"key", "x"}}}, + }}, + {"confidence", 0.75}, + }; + ToolSpeculationPrediction parsed; + std::string error; + CHECK(parse_tool_speculation_prediction(request, tools, parsed, error)); + CHECK(parsed.call.name == "lookup"); + + json undeclared = request; + undeclared["call"]["name"] = "write_file"; + CHECK(!parse_tool_speculation_prediction( + undeclared, tools, parsed, error)); + CHECK(error.find("not declared") != std::string::npos); +} + +TEST_CASE(ToolSpeculationFixture, cpu_affinity_parser_canonicalizes_ranges) { + std::vector cpus; + std::string error; + CHECK(parse_tool_speculation_cpu_affinity( + "30-31,15,14-15", cpus, error)); + CHECK(cpus == std::vector({14, 15, 30, 31})); + + CHECK(!parse_tool_speculation_cpu_affinity("14,,15", cpus, error)); + CHECK(cpus.empty()); + CHECK(!parse_tool_speculation_cpu_affinity("15-14", cpus, error)); + CHECK(cpus.empty()); +} + +TEST_CASE(ToolSpeculationFixture, empirical_policy_selects_resource_by_confidence) { + ToolSpeculationPolicy policy; + std::string error; + CHECK(policy.load_json(policy_fixture(), error)); + + const auto deferred = policy.choose(0.0, 1.20); + CHECK(!deferred.admitted); + CHECK(deferred.reason == "below_profile_break_even"); + + const auto low = policy.choose(0.10, 1.20); + CHECK(low.admitted); + CHECK(low.resource_percentage == 25); + + const auto medium = policy.choose(0.50, 1.20); + CHECK(medium.admitted); + CHECK(medium.resource_percentage == 50); + + const auto high = policy.choose(0.90, 1.20); + CHECK(high.admitted); + CHECK(high.resource_percentage == 100); + + const auto guarded = policy.choose(0.90, 1.10); + CHECK(guarded.admitted); + CHECK(guarded.resource_percentage == 50); +} + +TEST_CASE(ToolSpeculationFixture, unqualified_resource_lanes_are_deferred) { + ToolSpeculationPolicy policy; + std::string error; + CHECK(policy.load_json(policy_fixture(false), error)); + + const auto decision = policy.choose(1.0, 1.20); + CHECK(!decision.admitted); + CHECK(decision.reason == "decode_interference_unqualified"); +} + +TEST_CASE(ToolSpeculationFixture, profile_metadata_is_fail_closed) { + ToolSpeculationPolicy policy; + std::string error; + json fixture = policy_fixture(); + CHECK(policy.load_json(fixture, error)); + CHECK(policy.profile_status() == "qualified"); + CHECK(policy.executor_contract() == "child_process"); + + fixture["profile_status"] = "provisional_benchmark_only"; + CHECK(!policy.load_json(fixture, error)); + CHECK(policy.empty()); + + fixture["profile_status"] = "unknown"; + CHECK(!policy.load_json(fixture, error)); + CHECK(policy.empty()); + + fixture = policy_fixture(); + fixture.erase("executor"); + CHECK(!policy.load_json(fixture, error)); + CHECK(policy.empty()); + + fixture = policy_fixture(); + fixture.erase("profile_status"); + CHECK(!policy.load_json(fixture, error)); + CHECK(policy.empty()); + + fixture = policy_fixture(); + fixture["path_summary"]["25"].erase("accelerator_relation"); + CHECK(!policy.load_json(fixture, error)); + CHECK(policy.empty()); +} + +TEST_CASE(ToolSpeculationFixture, same_gpu_profile_is_rejected) { + ToolSpeculationPolicy policy; + std::string error; + json fixture = policy_fixture(); + for (auto & lane : fixture["path_summary"]) { + lane["accelerator_relation"] = "same_physical_gpu"; + } + CHECK(!policy.load_json(fixture, error)); + CHECK(error.find("accelerator_relation") != std::string::npos); + CHECK(policy.empty()); +} + +TEST_CASE(ToolSpeculationFixture, non_allowlisted_tool_is_deferred) { + ToolSpeculationConfig config = test_config(); + config.allowed_tools = {"other"}; + auto attempt = ToolSpeculationAttempt::create( + config, prediction(), "request_allowlist"); + attempt->start(); + const json metadata = attempt->resolve({ + ToolCall{"call_1", "lookup", R"({"a":1,"b":2})"}, + }); + CHECK(metadata["status"] == "deferred"); + CHECK(metadata["reason"] == "tool_not_allowlisted"); + CHECK(!metadata.contains("result")); +} + +#if !defined(_WIN32) +TEST_CASE(ToolSpeculationFixture, cpu_affinity_reaches_child_executor) { +#if defined(__linux__) + cpu_set_t allowed; + CPU_ZERO(&allowed); + CHECK(::sched_getaffinity(0, sizeof(allowed), &allowed) == 0); + int selected_cpu = -1; + for (int cpu = 0; cpu < CPU_SETSIZE; ++cpu) { + if (CPU_ISSET(cpu, &allowed)) { + selected_cpu = cpu; + break; + } + } + CHECK(selected_cpu >= 0); + const std::string selected = std::to_string(selected_cpu); + const std::string path = make_executor_script( + "IFS= read -r control\n" + "observed=$(awk '/Cpus_allowed_list/{print $2}' /proc/self/status)\n" + "printf '{\"ok\":true,\"result\":{\"configured\":\"%s\"," + "\"observed\":\"%s\"}}\\n' " + "\"$DFLASH_TOOL_SPECULATION_CPU_AFFINITY\" \"$observed\"\n"); + ToolSpeculationConfig config = test_config(path); + config.cpu_affinity = {selected_cpu}; + config.cpu_affinity_isolated = true; + CHECK(std::string(config.execution_mode()) == + "child_process_cpu_affinity"); + auto attempt = ToolSpeculationAttempt::create( + config, prediction(), "request_cpu_affinity"); + attempt->start(); + const json metadata = attempt->resolve({ + ToolCall{"call_1", "lookup", R"({"a":1,"b":2})"}, + }); + ::unlink(path.c_str()); + CHECK(metadata["status"] == "hit"); + CHECK(metadata["cpu_affinity_isolated"].get()); + CHECK(metadata["result"]["configured"] == selected); + CHECK(metadata["result"]["observed"] == selected); +#else + CHECK(true); +#endif +} + +TEST_CASE(ToolSpeculationFixture, cpu_affinity_qualification_rejects_overlap) { +#if defined(__linux__) + cpu_set_t allowed; + CPU_ZERO(&allowed); + CHECK(::sched_getaffinity(0, sizeof(allowed), &allowed) == 0); + int selected_cpu = -1; + for (int cpu = 0; cpu < CPU_SETSIZE; ++cpu) { + if (CPU_ISSET(cpu, &allowed)) { + selected_cpu = cpu; + break; + } + } + CHECK(selected_cpu >= 0); + ToolSpeculationConfig config = test_config(); + config.cpu_affinity = {selected_cpu}; + std::string error; + CHECK(!qualify_tool_speculation_cpu_affinity(config, error)); + CHECK(error.find("overlaps model CPU") != std::string::npos); + CHECK(!config.cpu_affinity_isolated); +#else + CHECK(true); +#endif +} + +TEST_CASE(ToolSpeculationFixture, exact_match_exposes_result_and_resource_share) { + const std::string control_path = make_temp_path(); + const std::string path = make_executor_script( + "IFS= read -r control\nprintf '%s' \"$control\" > " + control_path + "\n" + "printf '{\"ok\":true,\"result\":{\"resource\":\"%s\"," + "\"relation\":\"%s\",\"value\":42}}\\n' " + "\"$DFLASH_TOOL_SPECULATION_RESOURCE_PERCENTAGE\" " + "\"$DFLASH_TOOL_SPECULATION_ACCELERATOR_RELATION\"\n"); + ToolSpeculationConfig config = test_config(path); + auto attempt = ToolSpeculationAttempt::create( + config, prediction(), "request_hit"); + attempt->start(); + const json metadata = attempt->resolve({ + ToolCall{"call_1", "lookup", R"({"b":2,"a":1})"}, + }); + ::unlink(path.c_str()); + const std::string control = read_text_file(control_path); + ::unlink(control_path.c_str()); + + CHECK(metadata["status"] == "hit"); + CHECK(metadata["resource_percentage"] == 100); + CHECK(metadata["result"]["resource"] == "100"); + CHECK(metadata["result"]["relation"] == "non_accelerator"); + CHECK(metadata["result"]["value"] == 42); + CHECK(control.find("\"op\":\"commit\"") != std::string::npos); + CHECK(control.find("\"authoritative_resource_percentage\":100") != + std::string::npos); +} + +TEST_CASE(ToolSpeculationFixture, unqualified_tool_never_launches_or_changes_decode) { + const std::string path = make_executor_script( + "IFS= read -r control\n" + "exit 0\n"); + ToolSpeculationConfig config = test_config(path); + std::string error; + CHECK(config.policy.load_json(policy_fixture(false), error)); + auto deferred = ToolSpeculationAttempt::create( + config, prediction(), "request_guarded_decode"); + deferred->start(); + CHECK(!deferred->running()); + const json metadata = deferred->resolve({ + ToolCall{"call_1", "lookup", R"({"a":1,"b":2})"}, + }); + CHECK(metadata["status"] == "deferred"); + CHECK(metadata["reason"] == "decode_interference_unqualified"); + ::unlink(path.c_str()); +} + +TEST_CASE(ToolSpeculationFixture, executor_failure_is_private) { + ToolSpeculationConfig config = test_config( + "/definitely/missing/dflash-tool-spec-executor"); + auto attempt = ToolSpeculationAttempt::create( + config, prediction(), "request_executor_failure"); + attempt->start(); + const json metadata = attempt->resolve({ + ToolCall{"call_1", "lookup", R"({"a":1,"b":2})"}, + }); + CHECK(metadata["status"] == "failed"); + CHECK(metadata["reason"] == "executor_launch_failed"); + CHECK(!metadata.contains("result")); +} + +TEST_CASE(ToolSpeculationFixture, executor_timeout_starts_at_launch) { + const std::string path = make_executor_script( + "sleep 1\n" + "printf '{\"ok\":true,\"result\":{\"value\":42}}\\n'\n"); + ToolSpeculationConfig config = test_config(path); + config.timeout_ms = 25; + auto attempt = ToolSpeculationAttempt::create( + config, prediction(), "request_launch_deadline"); + attempt->start(); + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + const auto resolve_started = std::chrono::steady_clock::now(); + const json metadata = attempt->resolve({ + ToolCall{"call_1", "lookup", R"({"a":1,"b":2})"}, + }); + const double resolve_ms = std::chrono::duration( + std::chrono::steady_clock::now() - resolve_started).count(); + ::unlink(path.c_str()); + + CHECK(metadata["status"] == "failed"); + CHECK(metadata["reason"] == "speculative_executor_failure"); + CHECK(metadata["detail"] == "executor_timeout"); + CHECK(!metadata.contains("result")); + CHECK(resolve_ms < 500.0); +} + +TEST_CASE(ToolSpeculationFixture, executor_timeout_terminates_process_group) { +#if defined(__linux__) + const std::string marker_path = make_temp_path(); + const std::string path = make_executor_script( + "sleep 10 &\n" + "printf '%s' \"$!\" > \"" + marker_path + "\"\n" + "IFS= read -r control\n" + "wait\n"); + ToolSpeculationConfig config = test_config(path); + config.timeout_ms = 100; + auto attempt = ToolSpeculationAttempt::create( + config, prediction(), "request_process_group_deadline"); + attempt->start(); + const json metadata = attempt->resolve({ + ToolCall{"call_1", "lookup", R"({"a":1,"b":2})"}, + }); + const std::string child_text = read_text_file(marker_path); + ::unlink(path.c_str()); + ::unlink(marker_path.c_str()); + + CHECK(metadata["status"] == "failed"); + CHECK(metadata["detail"] == "executor_timeout"); + CHECK(!child_text.empty()); + if (child_text.empty()) return; + const pid_t child = static_cast(std::stol(child_text)); + bool gone = false; + for (int retry = 0; retry < 50; ++retry) { + if (::kill(child, 0) != 0 && errno == ESRCH) { + gone = true; + break; + } + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + CHECK(gone); +#else + CHECK(true); +#endif +} + +TEST_CASE(ToolSpeculationFixture, executor_does_not_inherit_server_fds) { +#if defined(__GLIBC__) && defined(__GLIBC_PREREQ) +# if __GLIBC_PREREQ(2, 34) + const std::string marker_path = make_temp_path(); + const int marker_fd = ::open(marker_path.c_str(), O_RDONLY); + CHECK(marker_fd >= 0); + const std::string path = make_executor_script( + "IFS= read -r control\n" + "leaked=false\n" + "for descriptor in /proc/self/fd/*; do\n" + " target=$(readlink \"$descriptor\" 2>/dev/null || true)\n" + " if [ \"$target\" = \"" + marker_path + "\" ]; then leaked=true; fi\n" + "done\n" + "printf '{\"ok\":true,\"result\":{\"leaked\":%s}}\\n' \"$leaked\"\n"); + ToolSpeculationConfig config = test_config(path); + auto attempt = ToolSpeculationAttempt::create( + config, prediction(), "request_fd_isolation"); + attempt->start(); + const json metadata = attempt->resolve({ + ToolCall{"call_1", "lookup", R"({"a":1,"b":2})"}, + }); + ::close(marker_fd); + ::unlink(path.c_str()); + ::unlink(marker_path.c_str()); + + CHECK(metadata["status"] == "hit"); + CHECK(!metadata["result"]["leaked"].get()); +# else + CHECK(true); +# endif +#else + CHECK(true); +#endif +} + +TEST_CASE(ToolSpeculationFixture, executor_environment_is_minimal) { + const char * previous = std::getenv("DFLASH_TOOL_SPECULATION"); + const bool had_previous = previous != nullptr; + const std::string previous_value = previous ? previous : ""; + const char * previous_secret = std::getenv("DFLASH_TEST_SERVER_SECRET"); + const bool had_previous_secret = previous_secret != nullptr; + const std::string previous_secret_value = previous_secret + ? previous_secret : ""; + CHECK(::setenv("DFLASH_TOOL_SPECULATION", "0", 1) == 0); + CHECK(::setenv("DFLASH_TEST_SERVER_SECRET", "must-not-leak", 1) == 0); + const std::string path = make_executor_script( + "IFS= read -r control\n" + "secret=false\n" + "if [ \"${DFLASH_TEST_SERVER_SECRET+x}\" = x ]; then secret=true; fi\n" + "printf '{\"ok\":true,\"result\":{\"enabled\":\"%s\"," + "\"secret_inherited\":%s}}\\n' " + "\"$DFLASH_TOOL_SPECULATION\" \"$secret\"\n"); + ToolSpeculationConfig config = test_config(path); + auto attempt = ToolSpeculationAttempt::create( + config, prediction(), "request_clean_environment"); + attempt->start(); + const json metadata = attempt->resolve({ + ToolCall{"call_1", "lookup", R"({"a":1,"b":2})"}, + }); + ::unlink(path.c_str()); + if (had_previous) { + CHECK(::setenv( + "DFLASH_TOOL_SPECULATION", previous_value.c_str(), 1) == 0); + } else { + CHECK(::unsetenv("DFLASH_TOOL_SPECULATION") == 0); + } + if (had_previous_secret) { + CHECK(::setenv( + "DFLASH_TEST_SERVER_SECRET", previous_secret_value.c_str(), 1) == 0); + } else { + CHECK(::unsetenv("DFLASH_TEST_SERVER_SECRET") == 0); + } + + CHECK(metadata["status"] == "hit"); + CHECK(metadata["result"]["enabled"] == "1"); + CHECK(!metadata["result"]["secret_inherited"].get()); +} + +TEST_CASE(ToolSpeculationFixture, qualified_lane_keeps_speculative_decode) { + const std::string path = make_executor_script( + "IFS= read -r control\n" + "exit 0\n"); + ToolSpeculationConfig config = test_config(path); + std::string error; + CHECK(config.policy.load_json(policy_fixture(), error)); + auto attempt = ToolSpeculationAttempt::create( + config, prediction(), "request_qualified_lane"); + attempt->start(); + CHECK(attempt->running()); + const json metadata = attempt->cancel("test_complete"); + CHECK(metadata["decode_interference_qualified"].get()); + ::unlink(path.c_str()); +} + +TEST_CASE(ToolSpeculationFixture, mismatch_cancels_and_never_exposes_result) { + const std::string control_path = make_temp_path(); + const std::string path = make_executor_script( + "IFS= read -r control\nprintf '%s' \"$control\" > " + control_path + "\n" + "exit 3\n"); + ToolSpeculationConfig config = test_config(path); + auto attempt = ToolSpeculationAttempt::create( + config, prediction(), "request_miss"); + attempt->start(); + const auto started = std::chrono::steady_clock::now(); + const json metadata = attempt->resolve({ + ToolCall{"call_1", "lookup", R"({"a":999})"}, + }); + const double elapsed_ms = std::chrono::duration( + std::chrono::steady_clock::now() - started).count(); + ::unlink(path.c_str()); + const std::string control = read_text_file(control_path); + ::unlink(control_path.c_str()); + + CHECK(metadata["status"] == "miss"); + CHECK(metadata["reason"] == "invocation_mismatch"); + CHECK(!metadata.contains("result")); + CHECK(elapsed_ms < 1000.0); + CHECK(control.find("\"op\":\"cancel\"") != std::string::npos); +} +#endif + +TEST_CASE(ToolSpeculationFixture, streaming_extension_keeps_result_explicit) { + const json metadata = { + {"status", "hit"}, + {"result", {{"value", 42}}}, + }; + const std::string event = render_tool_speculation_sse( + ApiFormat::OPENAI_CHAT, "req_1", "model", metadata); + CHECK(event.find("dflash_tool_speculation") != std::string::npos); + CHECK(event.find("\"value\":42") != std::string::npos); +}