Add a mock-isolated gateway latency report for main CI

[kfallah]

Summary

Adds a lightweight, secret-free CI report that measures Experiential gateway-added latency against one local OpenAI-compatible mock. The same sampler, payload, warmup, concurrency, and request counts hit:

1. the mock directly
2. the Experiential native gateway

Reported overhead is the client-observed difference (gateway minus mock) for p50, p95, and p99. This is not end-to-end model latency. The static research writeup in docs/research/rust-gateway-engine.md is unchanged. README.md is not edited.

Rebased onto current main. The only conflict was native_engine_disconnect_test.py; this branch now keeps main's escalated-alias fallback coverage.

Methodology

• local mock upstream
• discarded warmup requests
• sequential mock-direct then gateway arms
• fixed request and concurrency counts
• nearest-rank percentiles
• success and failure rate plus RPS
• several repeats
• one representative run: the median by gateway non-stream p50

Streaming time-to-first-token is included because the mock emits the first content token immediately, so TTFT is a first-byte measurement rather than a model-generation race.

Routine PR and main pushes stay on ubuntu-latest. workflow_dispatch uses gateway-benchmark-32core so public PRs cannot consume the larger runner.

Schema

exp.gateway.latency_report v1. The JSON records commit SHA, runner OS, CPU count and model, Python, Experiential engine, every repeat, and signed overhead versus the mock baseline. Functional request failures fail the job. Latency differences do not.

Sample result (ubuntu-latest)

Official job: https://github.com/experientiallabs/experiential/actions/runs/32597320092

Commit 45c59dac, GitHub-hosted ubuntu-latest, Linux 6.17 Azure, 4 x AMD EPYC 9V74, Python 3.12.3, Experiential rust, 0 failures. Representative run 1 of 3. Shared-runner tails are noisy: mock-direct p95 can include a one-second hitch, which makes gateway-added p95/p99 negative. Read p50 as the overhead signal.

Non-stream (warmup 10, 40 requests, concurrency 8):

Arm	Failures	RPS	p50 (ms)	p95 (ms)	p99 (ms)	added p50
mock direct	0	37.92	7.62	1032.85	1042.16
experiential	0	209.52	22.20	95.13	189.16	14.58

Streaming TTFT (warmup 5, 20 requests, concurrency 4):

Arm	Failures	RPS	p50 (ms)	added p50
mock direct	0	701.30	5.11
experiential	0	222.57	8.37	3.26

Commands

uv sync --extra dev
uv run python -m exp.runtime.gateway.latency_report \
  --output-json gateway-latency.json

Limitations

• Mock-isolated overhead only. Real-provider TTFT and tokens/s are out of scope.
• ubuntu-latest is a shared 4-vCPU hosted VM. Percentiles on n=40 are coarse, especially p99.
• Negative gateway-added tails mean the mock arm had a larger hitch, not that the gateway is faster than the mock.
• No hard latency regression gate. Functional failures fail CI.

Badge (README approval needed)

The badge above is in this PR description only. A README insertion still needs explicit README approval:

[![Gateway latency](https://github.com/experientiallabs/experiential/actions/workflows/gateway-latency.yml/badge.svg?branch=main)](https://github.com/experientiallabs/experiential/actions/workflows/gateway-latency.yml?query=branch%3Amain)

Review

Rebased onto current main at 12c57212. Greptile last scored the pre-rebase receipt-timeout fix at 5/5 (safe to merge); that unknown engine path is unchanged. Please re-review the rebased tip.

  

[greptile-apps]

Greptile Summary

The PR adds a secret-free CI benchmark that compares a local OpenAI-compatible mock directly against the Experiential gateway and publishes a versioned latency report.

• Adds a pull-request, main-push, and manually dispatched gateway-latency workflow with commit-pinned actions.
• Adds mock-server, gateway-launch, sampling, percentile, streaming-TTFT, and report-generation helpers with focused tests.
• Documents the benchmark methodology, artifact schema, runner selection, and interpretation limits.
• Preserves accurate engine provenance by waiting for launch receipts and reporting unknown when none is observed.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
.github/workflows/gateway-latency.yml	Adds the benchmark workflow with routine jobs on ubuntu-latest, explicit large-runner dispatch, read-only permissions, and full-SHA action pins.
exp/runtime/gateway/latency_measure.py	Adds the loopback mock, concurrent request sampler, gateway configuration and lifecycle helpers, and corrected launch-receipt engine detection.
exp/runtime/gateway/latency_report.py	Adds the versioned report schema, repeated direct-versus-gateway measurements, functional-failure enforcement, output rendering, and CLI entry point.
exp/runtime/gateway/latency_measure_test.py	Covers Rust and Python receipt parsing, delayed receipt publication, missing receipts, and the corrected timeout behavior.
exp/runtime/gateway/latency_report_test.py	Exercises report aggregation, rendering, configuration, output generation, and benchmark integration behavior.
docs/reference/gateway-latency.md	Documents the mock-isolated methodology, execution schedule, artifact contract, workflow behavior, and interpretation limits.

Sequence Diagram

sequenceDiagram
    participant CI as GitHub Actions
    participant R as Latency reporter
    participant M as Local mock
    participant G as Experiential gateway
    CI->>R: Run benchmark
    R->>M: Start loopback mock
    R->>G: Configure and start gateway
    G->>M: Route latency alias
    R->>M: Warm up and sample directly
    R->>G: Warm up and sample through gateway
    G->>M: Forward requests
    R->>R: Compute gateway-minus-mock percentiles
    R-->>CI: JSON artifact and Markdown summary

Loading

_{Reviews (8): Last reviewed commit: "Remove the third-party proxy comparison ..." | Re-trigger Greptile}

[cursor]

Addressed both Greptile findings on 1d1e472d (engine-receipt wait + SHA-pinned Actions) and resolved the review threads. Please re-review that commit. The gate failure is test_dead_fallback_engine_degrades_only_escalated_routes in an untouched native-engine file, not this latency report.

[cursor]

@greptileai Please re-review e6a30a38. The latency report now measures only mock-direct versus the Experiential gateway.

[cursor]

@greptileai Please re-review 12c57212. The branch is rebased onto current main and still measures only mock-direct versus the Experiential gateway.