Add Docker Sandbox staging smoke example [INT-976]#440
Open
AlexanderZ-Band wants to merge 13 commits into
Open
Add Docker Sandbox staging smoke example [INT-976]#440AlexanderZ-Band wants to merge 13 commits into
AlexanderZ-Band wants to merge 13 commits into
Conversation
…eline harness (#402) * refactor(e2e): registry-based adapter discovery + shared agent shapes Migrate the baseline smokes off the hand-maintained ADAPTER_BUILDERS map onto a discovery registry (toolkit/adapters.py) fronted by @with_agents / @across_adapters glue (agents.py), so a test names adapters by typed handle and the requirement gate travels with the choice. Consolidate the reusable agent shapes (TOOL_AGENT, MEMORY_AGENT) into smoke/sample_agents.py so tests spread one shared shape via @with_agents(..., **SHAPE) instead of re-spelling prompt=/features= per test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(e2e): capability filtering + per-cell steering for @across_adapters Add a `without=` capability filter (the complement of `supports=`) plus `prompt=`/`features=` to @across_adapters / adapter_params / specs, threaded to the provisioned_matrix_agent fixture via a MatrixBuild marker. A matrix scenario can now target the memory-capable adapters (or their complement) and enable the matching tools per cell — symmetric with @with_agents. - convert test_processing_barrier to @across_adapters (drops the hand-rolled build_agent + running_provisioned_agent matrix plumbing) - add test_capability_matrix: complementary supports={MEMORY} / without={MEMORY} demonstrations, list-free (driven purely by the registry's capability flags) - drop the redundant no-arg @across_adapters() from the full-matrix test; a bare provisioned_matrix_agent parameter already runs the full matrix Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): rename matrix fixture + give it a MatrixAgent NamedTuple `provisioned_matrix_agent` -> `matrix_agent` (shorter, less verbose), and its yield is now a `MatrixAgent` NamedTuple instead of a bare tuple — so it still unpacks (`adapter_id, agent = matrix_agent`) but also exposes named fields and a real, documented type. Updated @across_adapters' indirect-parametrize target, the conftest fixture, all call sites, and the README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): split matrix cell into adapter_id + matrix_agent; messages.snapshot/since The matrix fixture is now two flat fixtures instead of one tuple-yielding fixture: `matrix_agent` yields the cell's running `ProvisionedAgent` directly and `adapter_id` yields its id, so tests request what they use with no `adapter_id, agent = matrix_agent` unpack and no compound type. `@across_adapters` parametrizes `adapter_id` (indirect); the construction-only test just requests the `adapter_id` fixture. Retires the transitional `MatrixAgent` NamedTuple. - add `Replies.snapshot()` / `.since(cursor)` so a reused capture reads a later turn without a manual `len(...)`/slice (use it in the recall scenario) - capability matrix reuses the `**MEMORY_AGENT` shape instead of re-spelling prompt/features - README + docstrings updated for the two-fixture model, `without=` capability filter, prompt/features steering, and snapshot/since Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(e2e): make the baseline README a coding-agent playbook Lead with "Writing a test" (the 6-step skeleton + a decision table for choosing how to get agents: @with_agents / @across_adapters / bespoke) and a "Rules" do/don't section so a developer can point a coding agent at this file and get lean tests that reuse the toolkit — no reinvented provisioning/waiting/assertions, no sleeps, no magic strings, no boilerplate. Keeps the existing reference (layout, fixtures, "I want to" table, delivery state, inspection, validation policy) below the guidance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(e2e): tools= on @with_agents/@across_adapters; migrate custom-tool smokes Thread a `tools=` (band CustomToolDef) through build_adapter and every builder, and through @with_agents / @across_adapters (+ the agent/agents/matrix_agent fixtures). The 9 adapters whose constructors take CustomToolDef forward it as additional_tools; agno, letta and pydantic-ai can't (own agent / MCP tools / native-callable format), so they reject a non-empty tools= with a clear error — the toolkit's fail-loudly rule, never a silent drop. - migrate the single-agent custom-tool smokes (test_tool_calls + the room/turn isolation tests) to @with_agents(Adapter.ANTHROPIC, tools=[...], prompt=..., **EXECUTION_REPORTING); add the EXECUTION_REPORTING shape to sample_tools - keep test_tool_calls_isolated_per_sender bespoke (two agents, different tools in one room — a uniform @with_agents set can't express that) - README: add a "Design values" section (consistency / simplicity / ease), a custom-tools row + bullet, and drop the "Not here yet" section All 7 tool/isolation smokes pass live; ruff + pyrefly clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(e2e): add howto for wiring a new adapter into the matrix Step-by-step guide for registering a new framework adapter in the baseline matrix (enum member + @adapter builder, optional Dep/settings, custom-tool handling) so every matrix scenario runs against it. Linked from the baseline README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(e2e): add TBD section for NON_AGENT_ADAPTERS Flag the planned rename/formalization of the DENY set as NON_AGENT_ADAPTERS, to be developed later. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(e2e): ToolSpec custom-tool façade across frameworks + doc audit fixes Define a custom tool once as a `ToolSpec` (input model + handler); the builders translate it per framework — band `CustomToolDef` for the tool-loop adapters, a pydantic-ai `RunContext` callable, an agno agent tool. So `@with_agents` / `@across_adapters` take `tools=[LOOKUP_TOOL]` uniformly; only letta (MCP) can't take a local tool and rejects loudly. Normalize tool-call args at the observation layer (pydantic-ai records JSON-string args) so `assert_fired` is adapter-agnostic. - new `toolkit/tools.py` (`ToolSpec`, `as_custom_tool_def`, `as_callable` with a real synthesized signature both pydantic-ai and agno can introspect) - one cross-framework façade test (anthropic/pydantic-ai/agno) + a separate crewai tool test (its dev-crewai lane is mutually exclusive with pydantic-ai) - rename `DENY` -> `NON_AGENT_ADAPTERS`; `include`/`exclude` now take the typed `Adapter` enum (no magic strings) - fix `@across_adapters` dropping a bare `tools=` (now stamps the marker for tools-only too) - doc audit: README/ADDING_AN_ADAPTER builder examples (ToolSpec + _custom_tool_defs), reject list (letta-only), EXECUTION_REPORTING location; new "Dependency lanes" section (crewai mutual exclusion, run/switch, two-lane coverage); AGENTS.md links ADDING_AN_ADAPTER.md; CLAUDE.md E2E_TIMEOUT default 30 -> 60 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): two-lane baseline workflow + document the BAND_E2E_ADAPTERS knob Add `.github/workflows/e2e.yml`: a manually-triggered live baseline run split into two jobs that mirror the dependency lanes — `--extra dev` (key-only adapters) and `--extra dev-crewai` (crewai/crewai_flow) — each scoping the suite with `BAND_E2E_ADAPTERS` so out-of-lane adapters are deselected at collection instead of failing the fail-loud matrix. Their union is the full-matrix coverage. - pyproject: add `anthropic` to the `dev-crewai` extra (the baseline conftest imports it at load and the LLM judge needs it; thin client, no crewai conflict) - README "Dependency lanes": document `BAND_E2E_ADAPTERS` as the lane knob (the implemented `pytest_collection_modifyitems` deselection) and the local run/switch Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(e2e): derive CI lane partition from the adapter registry Replace the hand-listed BAND_E2E_ADAPTERS knob with BAND_E2E_LANE (a uv extra). Each Dep now carries a DepKind (PROVIDER_KEY / INFRA / VENV) in a single DepSpec record, and ci_lanes()/infra_adapters() derive the {extra: [adapters]} partition purely from each adapter's requires. The e2e workflow emits the lane matrix from the registry instead of listing adapters, and assert_every_adapter_has_a_ci_home() fails loudly if a newly-registered adapter (or Dep) lands in no lane. Out-of-lane and infra adapters now skip-with-reason; in-lane adapters keep their fail-loud @requires gate so a missing provider key still fails. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): grant tool permission by selecting an offered allow option A spec-strict ACP agent (e.g. codex-acp) cannot parse a bare {"outcome": {"outcome": "allowed"}} and aborts the turn. Auto-approve by selecting one of the agent's offered allow options (prefer allow_once over allow_always) via select_allow_option_id, and cancel rather than guess when no allow option is offered. Also raise the stdio transport limit on the codex-acp e2e spawns. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): split baseline conftest into lane_selection + _fixtures conftest.py was 418 lines mixing pytest hooks, CI-lane logic, and ~250 lines of fixtures. Separate the concerns so each file does one job: - lane_selection.py: the BAND_E2E_LANE collection logic (_item_target_adapters, _lane_skip_reason, apply_lane_skips); the conftest hook is now a one-line delegate. - _fixtures/{platform,agents,capture}.py: fixtures grouped by concern, re-imported into conftest. pytest_plugins is deprecated in a non-root conftest, so the import-re-export keeps fixtures registered and scoped to the baseline subtree (listed in __all__ so they read as intentional). conftest.py drops to hooks + re-exports (~115 lines). No behavior change: 76 tests still collected, lane scoping intact, registry guards pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(e2e): mark illustrative baseline README fences notest The three python fences in baseline/README.md (the test-writing example and the delivery/tool-call inspection snippets) reference fixtures/decorators and can't run standalone, so they failed `pytest --markdown-docs`. Mark them `notest` (matching ADDING_AN_ADAPTER.md) so the docs gate passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(google_adk): dereference $ref/$defs in tool param schemas Gemini's FunctionDeclaration `parameters` field is the restricted OpenAPI Schema subset and rejects JSON-Schema `$ref`/`$defs`. Pydantic emits exactly those for enum-typed params (band_list_memories'/band_store_memory's scope, system, type), so enabling memory tools made the ADK adapter fail at message time with a validation error — surfaced only at E2E. Build the declaration via `types.Schema.from_json_schema(...)`, which dereferences the refs into an inline, ref-free Schema valid on both the Gemini Developer API and Vertex AI (unlike `parameters_json_schema`, whose $ref support on Vertex is not guaranteed). Also reword the misleading "incompatible google-adk version" wrapper so the real schema error surfaces. Add a unit regression test that builds a declaration for every real platform tool (memory + contacts) and asserts no `$ref`/`$defs` leak — closing the gap that previously only showed up in the live suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(e2e): generalize lanes to CI jobs; wire the codex backend lane (phase 1) A lane was "a uv extra"; codex/opencode/letta each need their own CI job (a different external backend) within the same `dev` extra, so generalize a lane to "a named CI job = a uv extra + backend setup", still derived from the registry. - requirements.py: collapse DepKind+extra into one `lane` field on DepSpec (provider keys ride the shared `dev` lane; CREWAI->dev-crewai, CODEX_*->codex, OPENCODE_SERVER->opencode, LETTA_CLOUD->letta). LANE_EXTRAS maps lane->uv extra. - adapters.py: adapter_lane() + CILane(id, extra, adapters); ci_lanes() groups every adapter into its lane (5 lanes); guard asserts each is placed. Drop adapter_extra/is_infra_adapter/infra_adapters. - lane_selection.py: key on lane id; out-of-lane skips-with-reason, in-lane runs (an unwired backend fails loudly). No infra special-case. - e2e.yml: lanes job emits all 5 lanes; e2e matrix gains per-lane backend setup gated on matrix.lane. The codex lane installs @openai/codex, logs in with the OpenAI key, sets a disposable CODEX_CWD, and runs the matrix codex cell + the codex-acp e2e. opencode/letta lanes are red-by-design until phases 2/3. - docs: README CI-lanes section, AGENTS.md, ADDING_AN_ADAPTER.md updated. Verified: 5 lanes derived, guards pass, per-lane collection works, unknown lane errors loudly, registry smokes 7/7, ruff + markdown-docs clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): typed Lane enum + read backend config from settings Two cleanups on the lane model: - Lanes were magic strings (lane="codex"). Add a typed `Lane` StrEnum (like `Adapter`) and reference it everywhere (DepSpec.lane, LANE_EXTRAS, CILane.id, adapter_lane); json.dumps still emits the string id, so the workflow matrix is unchanged. The explicit LANE_EXTRAS table is kept (its `in` check is the fail-loud guard that a new lane must declare its uv extra). - The requirement predicates and adapter builders read backend config straight from os.environ, bypassing BaselineSettings. Add a `Backends` settings group (codex/opencode/letta config + the Gemini/Vertex fields on LLMCredentials), and read `settings.backends.*` in the predicates and the codex/opencode/letta builders. Defaults (opencode provider/model, letta model/base_url) now live in settings, the single source — no more os.environ in requirements.py. Verified: 5 typed lanes, guards pass, per-lane collection works, unknown lane errors loudly with clean ids, settings reads the backend env vars (incl. the E2E_CODEX_CWD_IS_DISPOSABLE alias), registry smokes 7/7, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(e2e): ruff-format line wraps in baseline tools/test_tool_calls Formatting only (wrap two long lines) — no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(e2e): wire the opencode backend lane (phase 2) Stand up an OpenCode server in the opencode lane so the matrix opencode cell runs live against the OpenCode Zen free model. Validated locally end-to-end (full opencode lane: 14 passed, 0 failed). - e2e.yml: opencode-lane setup (gated on matrix.lane) — install opencode-ai, write ~/.config/opencode/opencode.json with the Zen provider key via {env:OPENCODE_ZEN_API_KEY} (free small_model pinned, since the Zen account is free-tier), start `opencode serve` from an EMPTY throwaway dir, health-wait, export OPENCODE_BASE_URL + a bumped E2E_TIMEOUT. Needs the OPENCODE_ZEN_API_KEY secret. Header flips opencode [TODO] -> [wired]. - settings.py: refresh opencode_model_id default minimax-m2.5-free -> mimo-v2.5-free (the prior id is no longer in Zen's catalogue; provider id `opencode` is stable). - lane_selection.py: render lane ids (not Lane reprs) in the skip-with-reason text. Local validation surfaced the two non-obvious requirements now baked in: opencode is a coding agent with shell/read/grep tools, so serving from the repo checkout made a weak free model wander into the source instead of replying — an empty cwd fixes it; and the free-tier Zen account has no payment method, so the small/title model must also be a free model. No registry/conftest/adapter changes — phase 1 already made opencode a lane and the builder reads settings.backends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): assert barrier's FIFO + reply-before-processed contract The burst smoke only proved wait_for_processed didn't time out. The runtime processes a room's messages one-at-a-time in FIFO (single per-room loop), marking each PROCESSED only after its reply is emitted — there is no turn coalescing. Assert the contract that actually makes waiting on a single id valid: after the barrier on the last burst message, the earlier message is PROCESSED too (FIFO transitivity) and a reply for the round is already buffered (reply-before-processed). Fix the docstring/comments that wrongly claimed the burst batches into one turn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tests): restore --ignore by not vetoing it from pytest_ignore_collect pytest_ignore_collect is a firstresult hook: returning True ignores a path, but returning False *vetoes* pytest's built-in --ignore/--ignore-glob for the whole session. The hook returned the bool from pytest_ignore_collect_in_ci directly, so for every non-integration path it returned False and silently disabled --ignore — making the documented `pytest tests/ --ignore=tests/e2e` collect (and hang on) e2e anyway. Return None instead of False so pytest's own --ignore handling applies; keep returning True to skip integration in CI. Also make the CLI default-arg tests hermetic: the rest_url/auth_mode defaults read os.environ, and importing the e2e settings loads .env.test into the environment, so a bare `uv run pytest` saw a polluted BAND_REST_URL. Clear the vars with monkeypatch.delenv so the tests assert the built-in defaults. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(acp): gate codex-acp e2e tests behind E2E_TESTS_ENABLED These tests each spawn `npx @zed-industries/codex-acp` as a real subprocess (Node + network), so they are genuine E2E — but they were gated only on "is npx installed", which meant they ran on every dev machine with npx during a plain `uv run pytest`. Their subprocess/fd/event-loop pressure starved nearby server tests (tests/runtime/test_mcp_server.py localhost SSE/HTTP) into spurious >30s/>90s timeouts, and the codex-acp tests themselves timed out under that load. Add an E2E_TESTS_ENABLED opt-in to the module pytestmark (keeping the npx check and the e2e marker) so the default unit run skips them — matching the rest of the e2e suite. With them skipped, the documented unit command goes from 4:21 (4 failures) to ~24s (0 failures); the mcp localhost tests recover on their own. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): honor CODEX_COMMAND in the codex adapter builder The CODEX_CLI requirement gate validates the binary named by CODEX_COMMAND, but _build_codex dropped s.backends.codex_command and left CodexAdapterConfig.codex_command unset, so the adapter spawned the default codex binary. Pass the configured command through, parsed the same way the gate parses it, so a validated override is the binary actually spawned. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(e2e): add __future__ annotations import to baseline fixtures package Every module in the codebase carries `from __future__ import annotations`; the fixtures package __init__ was the lone omission. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(letta): add auto-relay mode (run without a Band MCP server) Setting `mcp_server_url=None` (or "") makes LettaAdapter skip MCP server registration entirely: the agent gets no platform tools and the adapter relays the model's plain-text reply to the room itself (the existing `_send_message` fallback). A self-hosted Letta server can't reach an in-process Band MCP bound to a loopback/private IP — its SSRF guard rejects non-public IPs and stdio MCP isn't registrable via the API — so auto-relay is the mode that actually works there. The instruction-block preamble is now mode-aware (`_persona_preamble`): the MCP enforcement text ("your plain text is discarded") would be actively wrong in auto-relay mode, so a relay note is used instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(e2e): wire letta in auto-relay + consolidate backend lanes into one job Wire the Letta adapter into the baseline matrix as a uniform cell: `_build_letta` builds it in auto-relay mode (mcp_server_url=None) so no Band MCP server is needed, and `letta-client` is added to the `dev` extra. The `_letta_available` gate is simplified accordingly (self-hosted base_url, or a Letta Cloud key — no MCP URL). Collapse the three external-backend lanes (codex/opencode/letta) into a single `backends` lane. They already share the `dev` extra and differ only in the backend their job stands up, so one job keeps the reliable `dev` lane uncoupled from the flaky backends while avoiding a job per backend. `Dep.LETTA_CLOUD` is renamed to `Dep.LETTA` (it gates self-hosted too). The workflow gates all backend setup steps (codex CLI, opencode serve, letta docker, codex-acp e2e) on `matrix.lane == 'backends'`, run with a plain `docker run -p 8283:8283` (no --network host/tunnel). Add a drift guard (`assert_workflow_lane_gates_known`): every `matrix.lane ==` literal in e2e.yml must name a lane `ci_lanes()` emits, so a gate on a renamed/removed lane id can never silently skip its step. Wired into the `lanes` job and covered by a unit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): letta showcase smokes + barrier cell + lane-drift guard test Add two Letta-lane showcase smokes (`test_letta.py`): stateful cross-turn recall and per-room isolation, both via auto-relay. Add Letta to the processing-barrier matrix. Add the unit test for the workflow lane-gate drift guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(e2e): fix stale lane id in test_letta docstring (letta -> backends) The showcase smokes run in the consolidated `backends` lane; the docstring's run command and prose still named the old `letta` lane. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): don't default the Codex model to the OpenAI chat model `_build_codex` fell back to `E2E_LLM_MODEL` (gpt-4o-mini) when `CODEX_MODEL` was unset, but the Codex CLI uses its own model catalogue — gpt-4o-mini isn't in it, so a non-empty config.model short-circuited CodexAdapter's model discovery and failed selection on any backends run without CODEX_MODEL set. Now `model` is only passed when CODEX_MODEL is explicitly set (mirroring the CODEX_COMMAND handling); otherwise config.model stays None and the adapter selects a valid Codex model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): pass BAND_API_KEY_USER_2 through to the e2e job env Wire the optional second user key into the matrix job so the upcoming two-user baseline smokes can read it once the secret is set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): rename baseline _fixtures package to fixtures Drop the leading underscore; it's a normal package, not a private one. Update conftest imports and doc comments to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(e2e): fix baseline README layout table for the Dep enum home The layout table attributed the Dep enum to requires.py, but Dep (plus DepSpec, Lane, LANE_EXTRAS, dep_lane) live in toolkit/requirements.py; requires.py only re-exports Dep and holds the @requires decorator. Add the missing requirements.py row and correct the requires.py description, matching the CI-lanes section and requires.py's own docstring. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): harden lane-gate guard + strip env-derived backend config Address code-review findings on the letta/lane work: - Lane-drift guard regex now matches both quote styles (`matrix.lane == "x"` as well as `'x'`); a double-quoted gate previously slipped through, so the guard could pass vacuously — defeating its purpose. - `_build_letta` strips MCP_SERVER_URL before the `or None`, so a whitespace-only value enters auto-relay mode instead of attempting an invalid MCP registration (matches the CODEX_MODEL handling). - `_letta_available` strips LETTA_BASE_URL and requires it non-empty for the self-hosted path, so a blank value falls back to the Cloud (key-required) path instead of passing the gate against an unusable URL. - Reuse: promote `_REPO_ROOT` -> `REPO_ROOT` in requirements.py and use it for the e2e-workflow path in adapters.py, so the checkout-depth assumption lives once. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(letta): skip MCP tool verification in auto-relay mode `_verify_mcp_tools_attached` is called on agent resume (shared and per_room), but in auto-relay mode (no MCP server) there are no tools to verify — it would still make an `agents.tools.list` round-trip and diff against an empty set. Guard it with an early return on `_mcp_enabled` so resume does no pointless API call. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(letta): warn when capabilities requested in auto-relay mode In auto-relay mode (no MCP server) Letta has no platform tools, so a requested MEMORY/CONTACTS capability is silently inert — and the base on_started warning only fires for *over-requested* capabilities (not in SUPPORTED_CAPABILITIES), so it stays quiet here. Add an explicit on_started warning naming the requested capabilities and pointing at mcp_server_url, turning the silent no-op into a clear signal. SUPPORTED_CAPABILITIES is intentionally left as the {MEMORY, CONTACTS} ceiling (Letta's tools are owned by the external MCP server, not gated per-config); a clarifying comment documents that it's a ceiling, not a per-instance guarantee. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): categorize baseline smoke tests into subdirectories Split the flat smoke/ dir by concern: pull the static/infra harness self-tests (registry guard, provisioning, user_ops) out into a sibling guards/ dir, and group the remaining live worked-examples under smoke/ by subject — samples/ (shared helpers), matrix/, behavior/, inspection/, adapters/. Repoint helper imports to smoke.samples, fix self-referential run-with docstrings, and update README + ADDING_AN_ADAPTER paths. No test bodies, fixtures, or toolkit modules change; CI needs no edit (it runs pytest tests/e2e/baseline/ recursively). Collection unchanged at 79 tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): prefix secrets with E2E_ and add temp push trigger Map the E2E_-prefixed GitHub secrets to the unprefixed env var names the code reads, keeping the translation entirely in the workflow. Also add a temporary push trigger scoped to this branch so the matrix can be exercised without a manual dispatch (to be removed before merge). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: trigger E2E re-run (empty) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(crewai): emit tool events with canonical name/args/output schema The crewai EmitExecutionReporter emitted tool_call/tool_result events keyed as {tool, input} / {tool, result|error}, but the shared parse_tool_call/parse_tool_result and the E2E observer key off name/args/output. crewai's tool events were therefore silently dropped on read — the E2E custom-tool inspection saw no tool fire, and tool events never round-tripped into history. Align both payloads to the canonical schema and pin it with unit assertions on the emitted content (the prior test only checked the send_event call count, so this drift was invisible locally). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(crewai): update flow phase3 assertion to canonical event schema This unit test had pinned the old {tool, input} / {tool, result} shape, so it failed once the reporter was aligned to {name, args} / {name, output, is_error}. Update it to assert the canonical schema. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): apply a global 120s turn timeout to all lanes Agent frameworks vary widely in cold-start + round-trip latency (crewai crew construction, self-hosted backends, free/slow models), and the 60s default flakes on the slow end (e.g. crewai's reply turn). Set E2E_TIMEOUT=120 at the job level for every lane and drop the per-step overrides the backends lane used to set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): cancel in-progress runs when a newer one is queued The baseline E2E runs share one live platform, so they must never run concurrently. Switch the concurrency group to cancel-in-progress: true (newest wins) so a quick succession of pushes cancels the older in-flight run instead of queuing behind it — only the latest commit burns the live platform. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): split google + letta into own lanes, add lane selection, rename lanes - Give gemini/google_adk their own `google` lane (isolate Google free-tier rate-limit flakiness) and letta its own `letta` lane (split from backends, which keeps codex + opencode). - Add a workflow_dispatch `lane` input (dropdown, default all) to run one lane on demand, validated against the registry. - Rename lane ids to a consistent content-based scheme (dev->core, dev-crewai->crewai) and decouple lane ids from uv extras via a typed `Extra` enum (LANE_EXTRAS no longer holds magic strings). - Document "Adding a CI lane" in the baseline README; refresh AGENTS.md and ADDING_AN_ADAPTER.md lane references. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): extract shared setup into a composite action (A+B) Both the lanes and e2e jobs repeated git-config + uv + python setup. Fold it into a local composite action (.github/actions/setup-e2e) and drop the separate actions/setup-python step — astral-sh/setup-uv's python-version sets UV_PYTHON, so uv provisions Python for sync/run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): move backend setup into .github/scripts (C) The codex/opencode/letta setup steps embedded heredocs, docker run, and health-check loops inline in the workflow. Extract each into an executable .github/scripts/setup-<backend>.sh (exporting discovered config via $GITHUB_ENV); the workflow steps now just run them, keeping the per-backend steps visible while the YAML stays readable. Document the script convention in the baseline README's "Adding a CI lane". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): config-driven turn budget + additive timeout(extra=) marker The per-test @pytest.mark.timeout(120) markers equaled E2E_TIMEOUT=120 (the internal wait_for_processed deadline), leaving zero headroom: pytest-timeout's hard kill (which spans fixture setup too) beat the wait's own diagnostic, so a slow turn surfaced an opaque "Timeout (>120s)" instead of the actionable "agent did not process message" error. crewai, the slowest framework, tipped over most reliably. Make the e2e turn budget the single source of truth, driven by config: - conftest base = E2E_TIMEOUT (via E2ESettings), overloading the native `timeout` marker — bare/no marker -> base; timeout(extra=n) -> base + n; timeout(n) -> absolute (native). The computed marker is prepended so pytest-timeout reads it, not the extra=/bare form its parser would reject. - Drop the 33 redundant @pytest.mark.timeout(120) decorators (base covers them). - crewai tool test -> timeout(extra=120) = 240, well above its 120 wait deadline. - Default e2e_timeout 30/60 -> 120 in both settings classes. Path-guarded to tests/e2e/, so unit tests keep the global 30s default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): drop redundant E2E_TIMEOUT override The turn budget now defaults to 120s in the e2e settings, and slow frameworks add headroom per-test via @pytest.mark.timeout(extra=...), so setting E2E_TIMEOUT=120 at the job level is redundant. Drop it; leave a pointer comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): clearer empty-capture assert + scoped flaky reruns - _turn_boundary now asserts the prior turn's reply was captured instead of surfacing an opaque IndexError on an empty buffer (slow/missing reply). - Retry the reply-latency flakes via pytest-rerunfailures (--reruns 2), but skip reruns for the google lane: its failures are the Google free-tier quota (429), which every rerun also hits — retrying there only burns another turn budget per attempt for a guaranteed fail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): apply --reruns 2 uniformly across all lanes Drop the google-lane special-case (--reruns 0). It only made sense while google ran on a free-tier key that fails on the daily quota; with a paid key google reruns are no different from any other lane's, so the conditional was just encoding a temporary state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): retry flaky live-reply tests via markers, not lane-wide reruns Drop the workflow-level --reruns (it retried every test in the lane) and instead mark only the live-agent-reply tests that actually flake with @pytest.mark.flaky(reruns=2): the matrix reply test, both capability-matrix turns, and the turn-scope isolation test. Everything else (construction, guards, inspection) keeps failing fast. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): remove temp push trigger; fix stale timeout/trigger docs - Drop the temporary push trigger on the feature branch — E2E is dispatch-only again. Update the concurrency/SELECTED_LANE comments that referenced push. - AGENTS.md: E2E_TIMEOUT default is 120s (E2ESettings.e2e_timeout), not 60. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(e2e): make adapter→lane association explicit in ADDING_AN_ADAPTER The lane an adapter runs in is derived from its `requires` deps, not set on the adapter — but that mental model was implicit. Add a "Which CI lane your adapter lands in" section with the Dep→lane→extra mapping, note it on the `requires` bullet and checklist, and point to README's "Adding a CI lane" for the brand-new-lane case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document markdown-snippet testing + adapter lane verification - AGENTS.md: add a "Documentation Testing (markdown snippets)" section — how `pytest-markdown-docs` runs tracked .md in CI, the python/notest/fixture fence conventions, prefer-runnable guidance, and the gotcha that snippets under tests/e2e/** skip in CI (E2E skip-gate) so they protect nothing there. - ADDING_AN_ADAPTER.md: add a one-liner to verify which lane an adapter resolved into (ci_lanes()), plus the partition guard that enforces it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): repair inert scenario timeouts + fail loud on dead backend servers Review findings from the e2e CI restructure: #1 The e2e_timeout default jumped 30->120, which pinned the agno scenario inner deadlines `min(e2e_timeout * N, CAP)` permanently at their caps — the *3/*2 multipliers went inert and E2E_TIMEOUT stopped scaling them. Drop the dead multiplier: `min(e2e_timeout, CAP)` keeps the default value identical (100/90) while restoring downward scaling for lowered budgets. #2 setup-letta.sh / setup-opencode.sh health-check loops fell through silently when the server never came up (curl is the LHS of `&&`, so set -e is exempt), exporting *_BASE_URL on a dead server and deferring failure to an opaque connection error at test time. Track readiness and exit 1 with the server log if it never healthchecks — this also covers an opencode server that dies on launch (the backgrounded subshell hid that). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(e2e): single-source E2E_TIMEOUT default + fail-fast on missing codex key #6 The 120s E2E_TIMEOUT default was a literal in both E2ESettings and the standalone BaselineSettings (they previously drifted 30 vs 60). Single-source it via a dependency-free tests/e2e/_constants.py (importing tests/e2e/settings into baseline would drag in its load_dotenv()/pytest import side effects). #7 setup-codex.sh: `printenv OPENAI_API_KEY` takes the name as an argument, so `set -u` can't catch an unset key — login failed opaquely. Add an explicit `: "${OPENAI_API_KEY:?...}"` guard for a clear error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): run default_factory in ToolSpec.as_callable signature synthesis field.get_default() does not run a default_factory — it returns None — so an optional model field like `tags: list[str] = Field(default_factory=list)` baked `tags=None` into the synthesized tool signature and forwarded None into the model, failing validation the first time the LLM omitted the arg (pydantic-ai / agno paths). Pass call_default_factory=True so the real default ([]) is baked in. Add pure-Python guard tests covering omitted optionals, supplied values, no cross-call sharing of a factory default, and the ctx-prepended path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(letta): strip Band tool-loop instructions in relay mode In auto-relay mode (mcp_server_url=None) the agent has no Band tools and the adapter relays its plain-text reply to the room. But on_started still rendered _system_prompt with include_base_instructions=True, so the base instructions told the model to call band_send_message and that "plain text output is not delivered" — directly contradicting the relay note's plain-text contract, and referencing delegation/memory tools that don't exist. This hit both the create and resume instruction-block paths. Gate base instructions on _mcp_enabled so relay mode renders identity + custom section only (which also drops the inert MEMORY/CONTACT capability sections), and add the prompt-injection Security note to the relay preamble so that guidance survives. Fixed at the single render site, so both paths inherit it. Tests drive the real render via on_started (not a stubbed prompt) and assert the relay system prompt, created persona block, and resumed instruction block carry none of the tool-loop phrases, plus an MCP-mode regression guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): transport-neutral default prompt + guard mixed-lane tests Three baseline-toolkit fixes from review: - Default _SHORT_PROMPT named band_send_message, which flows through as custom_section and contradicts Letta auto-relay mode (no Band tools; the adapter relays plain text). The recent relay fix only stripped BASE_INSTRUCTIONS, not custom_section. Reword the default to name no delivery mechanism — MCP/native adapters still get band_send_message from BASE_INSTRUCTIONS. - Add assert_no_unschedulable_mixed_lane: a @with_agents test whose adapters span >1 CI lane is skipped in every lane (false green). Fail collection for it (any run), with a @pytest.mark.mixed_lane opt-out for deliberate local-only tests. - Fix two docstrings that said missing keys "skip" — require_dep fails loudly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): collapse duplicate reply-assertion API onto Replies methods The baseline harness had two parallel implementations of the same tolerant reply assertions: free functions in toolkit/assertions.py and the fluent methods on the Replies/Events list subclasses (toolkit/observations/). Both were live across the smoke suite, so a fix to one could silently drift from the other — and the free assert_contains_any used a raw substring test that let an empty option pass vacuously, where the method routes through tolerant_match (empty matches only empty content). Keep the method API as canonical (capture.messages and .since() already return Replies) and delete the free module. Migrate the 8 call sites; the list(...) snapshot copies become Replies(...) to keep both the copy and the methods. assert_at_least/assert_mentions free functions were already dead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): unify memory reader on `limit`, dropping the page_size outlier Memories.read was the only observation reader to name its record cap `page_size` (default 50); Events.read/ToolCalls.read/MemoryToolCalls.read all use `limit` (default 100). Because the two readers diverged, ReplyCapture.memory had to expose both knobs and routed `limit` to the call layer and `page_size` to the store layer — so `capture.memory(agent, limit=N)` silently capped the tool-call layer while the stored-memory read kept its own default. The knob a caller reaches for didn't control the layer they were inspecting. Rename Memories.read's param to `limit` (mapping to the API's page_size internally, as the other readers already hide their wire params), then collapse memory()'s two knobs into one `limit` that caps both layers uniformly. No call-site churn (nobody passed either knob) and the REST contract is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): unify event 'at least one' assertion on assert_present Events.assert_emitted was the lone outlier for the "something showed up" check — Replies and Memories both name it assert_present. Rename Events.assert_emitted -> assert_present (and the unused emitted() predicate -> present()) so all three observation collections share one cross-collection assertion vocabulary; the failure message keeps the event-specific "emitted" verb. Update the one caller. Also clean up README doc-rot left by the earlier assertion dedup: drop the stale references to the deleted free toolkit/assertions.py module and point the examples/strategy notes at the Replies/Events/Memories assertion methods. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): annotate reply_capture fixture with the CaptureFactory alias The reply_capture fixture spelled out its return type as Callable[[str], AbstractAsyncContextManager[ReplyCapture]] while all ~30 consumers annotate the injected param with the CaptureFactory alias (the same type, defined in toolkit/capture.py). Use the alias on the fixture too, so the value's definition and its consumers describe the type one way and a change to the alias propagates to both. Drops the now-unused AbstractAsyncContextManager and ReplyCapture imports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): route BAND_E2E_LANE through BaselineSettings The lane-scoping knob was the lone BAND_E2E_* var read via os.environ directly, bypassing the pydantic-settings layer every other knob uses. Add a `lane` field to BaselineRun (BAND_E2E_ prefix) and have the collection hook pass settings.run.lane into apply_lane_skips, which now takes the lane as an argument instead of reading the environment. Behavior is unchanged: unknown lane -> UsageError, valid lane -> scoped skips, empty -> full matrix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): bump baseline openai_model default to gpt-5.4-mini gpt-4o-mini was too weak for the matrix tool loop — the crewai cell kept calling band_send_message without the required mention and never replied, failing test_matrix_agent_replies[crewai]. A more capable default fixes the cell and benefits the other openai_model cells (langgraph, pydantic_ai). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): dedup matrix params, fix shadowed decorator, gather room adds - _MATRIX_PARAMS now reuses adapter_params() instead of re-spelling the pytest.param comprehension, so matrix-cell gating lives in one place - rename _reject_tools' `adapter` param to `adapter_id` so it no longer shadows the module-level `adapter()` registration decorator - provision_room adds participants concurrently via asyncio.gather rather than sequentially, so setup latency doesn't scale with participant count Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): guard the workflow lane dropdown against registry drift The workflow_dispatch `lane` dropdown is a hand-maintained option list (GitHub choice inputs require a static list, so it can't be registry-derived at runtime). It was the one lane list with no compile-time guard: a newly-registered lane missing from it can't be dispatch-selected, and a stale option only failed when someone happened to pick it. Add workflow_lane_options() + assert_workflow_lane_options_match_registry() next to the existing matrix.lane gate guard, and a unit-suite test, so the dropdown must equal {registry lanes} | {"all"} — drift (missing or stale) now fails on every PR rather than silently. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): single source for the adapter->lane map apply_lane_skips and assert_no_unschedulable_mixed_lane each built the same {adapter: lane} comprehension inline, so the two could drift. Extract a _lane_of(lanes) helper as the one place that flattens the registry partition; apply_lane_skips reuses its already-fetched lanes (no extra ci_lanes() call). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): split ToolSpec.as_callable into focused helpers as_callable interleaved three concerns (build signature strings, exec, attach annotations) with two separate loops over model_fields that had to stay in lockstep. Extract _build_signature() (param strings + matching exec namespace) and _annotations() (the __annotations__ map), leaving as_callable a thin orchestrator with exec in exactly one place. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(crewai): treat empty final answer after a tool-only turn as benign CrewAI's native-tools executor raises ValueError("Invalid response from LLM call - None or empty.") whenever its forced final-answer step yields empty text. The adapter already swallows this as benign when a band_send_message reply went out, but keyed only on `replied` — so a tool-only turn (e.g. the user says "store this memory, do not send a message") that ends with an empty final answer re-raised, marking the message failed even though the agent did its job. Models that emit trailing text (gpt-4o-mini) masked this; gpt-5.4-mini returns a genuinely empty final answer and exposed it. Extend ReplyTracker with `tool_executed` (set when any tool succeeds, alongside the existing `replied` for band_send_message) and broaden the guard to `replied or tool_executed`. Genuine no-response failures — empty answer with no work done, or any non-"Invalid response" error — still surface and propagate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): heterogeneous multi-agent collaboration smokes Add three baseline smokes for a room of different framework types working together, on the lane-safe @with_agents path (core lane: anthropic + pydantic_ai + agno): - coordinator delegates a compound task to two specialists (one per framework), each runs its own tool; asserts the coordinator mentioned both and both opaque results landed. - coordinator alone recruits a specialist mid-conversation via band_add_participant, then delegates the lookup. - three framework types triage concurrent mentions with no cross-talk. Roles are set by the runtime-injected user message (uniform @with_agents prompt/tools), and delegation integrity is proven by sender-scoped tool reads over opaque-value tools rather than tool absence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): agno and crewai adapter showcase smokes Add adapter-focused baseline smokes alongside the existing letta showcase: - test_agno.py (core lane): a ToolSpec becomes a native agno callable that fires — single tool, and two-tool dispatch in one turn. - test_crewai.py (crewai lane): CrewAIAdapter executes a custom tool (verifies the canonical tool-event schema), and CrewAIFlowAdapter delivers a reply. Both bound to @with_agents so they lane-scope and skip-with-reason elsewhere. Verified live: agno in the core venv, crewai in the dev-crewai venv. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pydantic_ai): strip content:null responses on every model request The matrix/tool E2E smokes for pydantic_ai started 400ing under gpt-5.4-mini with `Invalid value for 'content': expected a string, got null`. The model can emit an empty (or thinking-only) response within a single turn; pydantic-ai then replays it on the next mid-run request as an assistant message with content:null and no tool calls, which the provider rejects. gpt-4o-mini tolerated this; the stricter gpt-5.4-mini exposed it. The existing `_is_replayable_history_message` guard only sanitizes history *persisted between turns* (after AgentRunResultEvent), so a fresh single-turn room still hit the null mid-run. Register the same predicate as a pydantic-ai `history_processor`, which runs before *every* model request (mid-run included), closing the within-run gap. The predicate only drops responses that are entirely empty or entirely thinking — tool calls (and their results) are always kept, so call/result pairing is never broken. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * revert(letta): remove auto-relay mode Auto-relay (mcp_server_url=None) let the adapter run without a band-mcp server by relaying the model's plain-text reply to the room. It was added to work around the Letta E2E lane lacking a reachable band-mcp, but it is a production code path bent around a test limitation and silently disables all platform tools. Remove it: the adapter always registers a band-mcp endpoint (mcp_server_url), matching examples/letta/docker-compose.yml. Drops _LETTA_RELAY_NOTE, the _mcp_enabled flag, the relay/else branch and capability warning in on_started, the include_base_instructions gating, and the _verify_mcp_tools_attached early return; mcp_server_url is a plain str again. Keeps the pre-existing plain-text fallback relay (comments retagged so it is not confused with the removed mode). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pydantic_ai): treat empty final answer after tool work as benign With the content:null history processor in place, gpt-5.4-mini's empty final responses no longer 400 — but they surface one layer up instead: pydantic-ai forces a final str output (output_type=str), so when the agent has already acted via a tool this turn (a band_send_message reply, a band_store_memory, ...) and the model returns a genuinely empty final answer, output-validation retries are exhausted and the run raises UnexpectedModelBehavior. The work already went out, so failing the message is wrong. Mirror the crewai adapter (treat empty final answer after a tool-only turn as benign): track whether any tool succeeded this turn and, on UnexpectedModelBehavior whose message names output validation, swallow it when a tool ran. Genuine no-response failures (no tool ran) and any other model error still propagate. gpt-4o-mini emitted trailing text and masked this; gpt-5.4-mini returns an empty final answer and exposed it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): run Letta lane against a real band-mcp server Replaces the removed auto-relay shortcut with the real tool path, mirroring examples/letta/docker-compose.yml: the letta lane now stands up a self-hosted Letta server plus a band-mcp container on a shared Docker network, and Letta calls band-mcp (http://band-mcp:8002/sse — a routable service host, since Letta's SSRF guard rejects loopback) to execute platform tools. band-mcp authenticates as one agent (BAND_API_KEY), so the agent under test must share that identity: the agent fixtures run Letta as that static agent (id resolved via get_agent_me, not reaped) instead of provisioning a fresh one; every other adapter still provisions dynamically. - setup-letta.sh: ghcr login (built-in token), run letta + band-mcp on one network, export LETTA_BASE_URL + MCP_SERVER_URL - e2e.yml: forward E2E_BAND_API_KEY -> BAND_API_KEY; pass GITHUB_TOKEN to the letta step for the GHCR pull - provisioning: static_agent() + running_provisioned_agent(static_agent=...) - fixtures: Letta runs as the static agent (agents + matrix_agent) - _build_letta registers MCP_SERVER_URL; settings/README/docstring drop the auto-relay description Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): correct band-mcp GHCR image owner to band-ai The letta lane setup pulled ghcr.io/band/band-mcp:latest, but the 'band' GitHub org does not exist (the org is band-ai), so the pull failed with 'manifest unknown' and the lane never ran its tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): make the letta lane runnable locally setup-letta.sh now guards its $GITHUB_ENV writes (and prints the URLs), so the same script that brings up the letta + band-mcp stack in CI also works on a developer machine. Adds letta-lane-local.sh: one command that sources .env.test, brings the stack up, runs the Letta smokes with the lane env, and tears it down on exit. README documents it. Also fixes the band-mcp image path to the band-ai org (ghcr.io/band-ai/band-mcp) in the example compose — the old ghcr.io/band/... path would fail to pull. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): defer Letta to a follow-up; keep an empty letta lane Letta E2E is out of scope for this PR (tracked by INT-919): running it live needs a band-mcp server reachable from the Letta server, which isn't wired. Strip the band-mcp lane wiring and leave the `letta` lane as a placeholder. - adapters.py: add AdapterSpec.e2e_pending; register Letta with e2e_pending=True. ci_lanes() still emits the `letta` lane (include_pending), but specs()/the matrix exclude it — Letta is no longer a matrix cell and runs no @with_agents tests. A clear TODO documents flipping the flag to re-add it. - smoke/adapters/test_letta.py: replace the live smokes with one passing placeholder (asserts Letta is e2e_pending and still owns the lane) + a TODO. - Revert the static-agent path (provisioning.static_agent, fixtures wiring), the band-mcp setup-letta.sh / letta-lane-local.sh scripts, the e2e.yml letta step and BAND_API_KEY wiring, and the band-mcp docs in README/settings. Auto-relay stays removed (07f5950). The letta lane runs only adapter-agnostic baseline tests + the placeholder, needing no backend setup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * revert(letta): restore adapter + tests to the base (no PR diff) The only remaining differences in src/band/adapters/letta.py and its unit tests were cosmetic churn left over from the (now reverted) auto-relay work — comment wording and two renamed test methods, no behavioral change. With Letta E2E out of scope (INT-919), restore both files to the PR base so this PR doesn't touch the Letta adapter at all. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(e2e): drop stray Letta leftovers from the band-mcp work - examples/letta/docker-compose.yml: revert the out-of-scope image-org change (band -> band-ai) back to the base; band-mcp publishing/org belongs to the Letta E2E follow-up, not this PR. - e2e.yml: fix the stale letta-lane header comment (auto-relay -> e2e_pending). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): order required params first in ToolSpec.as_callable; fix e2e_pending doc Review findings on PR #402: #1 (latent crash): ToolSpec._build_signature emitted params in the model's field order, so a model declaring an optional field before a required one synthesized `def tool(opt=_default, req):` — a defaulted param before a non-defaulted one, which is a SyntaxError at exec() time (crashing as_callable() on the pydantic-ai/ agno path while the anthropic path worked). Sort required params first; stable sort preserves declaration order within each group. Adds a regression guard. #3 (doc): the e2e_pending comments claimed the flag excludes an adapter from `@with_agents`. It doesn't — with_agents resolves any registered adapter; e2e_pending only drops it from specs()/adapter_params() (the matrix). Reworded the AdapterSpec docstring and the Letta registration comment to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): guard ToolSpec ctx-field collision; fold build_tool_agent into build_adapter Review findings on PR #402: #2: ToolSpec.as_callable's pydantic-ai path prepends a `ctx` param; a model field also named `ctx` produced a duplicate-argument SyntaxError and conflated the RunContext with the field. Raise a clear ValueError at build time instead (agno path, which injects no ctx, still accepts a `ctx` field). Adds a regression guard. #4: build_tool_agent (sample_tools.py) hand-reimplemented build_adapter( Adapter.ANTHROPIC, ...) — same model/provider_key/prompt/additional_tools/features wiring, a second path that would silently drift from the registry builder. Remove it; test_isolation's bespoke two-different-tools-per-agent build now calls build_adapter(..., tools=[...], **EXECUTION_REPORTING) directly. EXECUTION_REPORTING (widely used) stays. README/docstrings updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): restore multi-turn budget multipliers in agno scenarios Review finding #7 on PR #402: the inner asyncio.wait_for budgets had been narrowed from min(e2e_timeout * 3, 100) / * 2 to a single-turn min(e2e_timeout, N). At the default E2E_TIMEOUT=120 the cap dominates so it's a no-op, but at a low local E2E_TIMEOUT these multi-turn flows (invite + run + relay + remove) would get one turn's budget and time out spuriously — and the comments still describe the multi-turn intent. Restore the *3 / *2 multipliers (back to base). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): hang-free in-process Parlant server + baseline smoke Parlant's Server is a "configure in the body, serve forever on exit" context manager: __aexit__ boots uvicorn (serve_app -> server.serve()) and blocks until a signal that never comes in-process, so `async with p.Server()` hangs on teardown and leaks ports. This kept Parlant out of the baseline matrix. Add running_parlant_server: enters the server for setup only, yields it for the run, and at teardown drives __aexit__ as a cancellable task — letting the serve loop come up (waiting on the public `ready` event) and then cancelling it. serve_app catches the CancelledError, so __aexit__'s finally runs the real cleanup (_exit_stack.aclose() shuts the plugin server + DB/evaluator) and the call returns instead of hanging. The helper also defaults to the OpenAI NLP service (authenticates with OPENAI_API_KEY) on freshly reserved ephemeral ports so reruns / concurrent sessions don't collide. Wire the new baseline smoke onto it, and migrate the dedicated adapter test's fixture onto the same helper (was Emcie default on hardcoded 8800/8818). Both verified live: clean exit, graceful shutdown, no port leak. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: harden ACP allow-option id coalescing and pydantic-ai benign-swallow Review findings on PR #402: #5: select_allow_option_id coalesced the camelCase/snake_case id spellings with `a or b`, so a present-but-empty `optionId` ("") fell through to the (absent) alias and the option was dropped — the caller then cancels instead of approving. Coalesce on absence (`is None`), not falsiness. Adds unit coverage for the allow-once preference, the no-allow case, and the empty-id edge. #6: the benign empty-output swallow matched the bare substring "output validation" inline. Extracted to a documented _is_output_validation_exhausted() helper (case-insensitive) that names the deliberate, fail-safe coupling to pydantic-ai's UnexpectedModelBehavior wording and why it must stay distinct from "Received empty model response". Behavior unchanged (swallow/propagate tests pass). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: exclude read-only/failed tools from benign empty-answer suppression The crewai and pydantic-ai adapters swallow a benign "empty final answer" when the agent already did productive work this turn. Both counted the wrong things as productive: - crewai counted any successful tool, including read-only lookups (band_get_participants, band_lookup_peers, ...), so an empty answer after a lookup-only turn was silently swallowed despite no reply going out. - pydantic-ai counted every FunctionToolResultEvent — read-only tools AND failed band tools (whose wrappers return "Error ..." strings), so a failed or non-replying turn could disappear with no Band message and no error. Add READ_ONLY_TOOL_NAMES to the tool registry (validated against ALL_TOOL_NAMES like the existing memory/contact sets) and gate both adapters: read-only tools never count as terminal work, and pydantic-ai also excludes known band tools that returned an error string. Custom/unknown tools are still treated as productive when run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: ruff-format ToolSpec ctx-annotation call in e2e guard test Pre-existing format drift (long as_callable(ctx_annotation=object) call) that fails the CI ruff-format check on this branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: ruff-format select_allow_option_id assertion in acp client test Pre-existing format drift (long select_allow_option_id assertion line) that fails the CI ruff-format check on this branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): matrix scenarios for recall, isolation, noisy room, tool + memory round-trip Add cross-adapter baseline matrix scenarios plus the toolkit affordances they need, port three legacy scenario tests onto the baseline registry, and remove the duplication they replace. Toolkit: - Factor running_agent out of running_provisioned_agent so a test can stop and restart one identity (rejoin / platform-history rehydration). - Add a decoupled runs_tool_loop registry flag + specs/across_adapters filter to select the custom-tool-capable subgroup (guard-pinned partition). - Add assert_contains_none (the tolerant dual of assert_contains_any). Scenarios (smoke/matrix/): - context recall: in-session + after-rejoin (proves /context rehydration). - room isolation: per-room context, no cross-leak. - noisy room: needle recall + ignore cross-talk, via the delivery barrier. - tool round-trip: tool fires AND opaque result in reply across the tool-loop subgroup (supersedes the hardcoded-include and standalone-crewai tests). - memory recall: store -> list -> get read-back across memory adapters. Retries scoped to rerun_except=["AssertionError"] so a failed behavioural assertion never masks a bug. Removed legacy tests/e2e/scenarios/ {context_persistence,room_isolation,noisy_busy_room}; kept the langgraph restart smoke (unique replay / exactly-once coverage). Documented "search before you build" in the baseline README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(e2e): sync baseline docs with the matrix-scenario work Fix drift introduced by the matrix scenarios + runs_tool_loop flag: - AGENTS.md: scenarios/ no longer holds context/isolation/noisy tests (moved to baseline/smoke/matrix/); describe what's actually left. - baseline/README.md: list the new smoke/matrix scenario files; document running_agent (rejoin), the runs_tool_loop subset filter, and assert_contains_none. - ADDING_AN_ADAPTER.md: document the runs_tool_loop flag + add it to the Step 2 checklist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): unify baseline agent API into @per_adapter / @with_adapters Replace the three inconsistent ways a baseline E2E test acquired agents (@across_adapters, @with_agents, and bare matrix fixtures) with two explicitly-named topology decorators: - @per_adapter(...): fan across the adapter matrix (bare = full matrix); inject `agent` (managed, running) or `cell` (an AdapterCell you drive yourself). - @with_adapters(...): a fixed set of adapters together in one room; inject `agent` / `agents`. Fixtures collapse to agent / agents / cell (+ internal adapter_id); matrix_agent and the implicit bare-fix…
…#406) Migrate all examples off the deprecated boolean reporting flags in favor of the features=AdapterFeatures(emit={...}) API. - anthropic/claude_sdk README snippets: enable_execution_reporting=True -> features=AdapterFeatures(emit={Emit.EXECUTION}) - crewai README: drop the disabling enable_execution_reporting=False (no features needed when reporting is off) - codex examples + run_agent.py: enable_task_events=True (and stale False-valued reporting booleans) -> features=AdapterFeatures(emit={Emit.TASK_EVENTS}) All changes are behavior-preserving; the adapters already mapped these booleans to the same Emit values internally. ✌️
* feat(adapters): Emit.USAGE seam — per-turn token usage capture
Add an additive SDK seam for capturing per-turn LLM token usage and reading
it back in the E2E baseline toolkit, mirroring the execution-reporting pattern.
SDK:
- TurnUsage (input/output/cache-read/cache-write, summed across a tool loop via
+) and Emit.USAGE in band.core.types.
- SimpleAdapter.emit_usage(): shared, best-effort, gated on Emit.USAGE. Usage
rides an accepted `task` event's metadata (USAGE_METADATA_KEY) — the backend
rejects an unknown `usage` message_type today; USAGE_EVENT_TYPE keeps the flip
to a first-class type a one-liner.
- Wire anthropic, langgraph, pydantic_ai, claude_sdk, agno, google_adk to map
their framework's usage onto TurnUsage and emit once per turn.
Toolkit:
- Usage/UsageRecord observation reading the usage metadata discriminator (so
capture.usage() doesn't collide with capture.tasks()), capture.usage(),
COST_AGENT shape, assert_nonzero_input_and_output() (the L4 gate).
Tests: usage-mapping unit tests per adapter + the emit gating; test_usage smoke
fans across the 6 wired adapters (@per_adapter).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(usage): extract TurnUsage.from_object/from_mapping constructors
De-duplicate the per-adapter usage mappers. Add TurnUsage.from_object (attribute
source) and TurnUsage.from_mapping (dict source) plus a shared _as_int coercion
in band.core.types; both return empty usage for a None/non-mapping source, so
each adapter's mapper collapses to a single declarative call with no local int
closure or null-guard.
- anthropic, claude_sdk, agno, google_adk: one-liner from_object/from_mapping
(also unifies anthropic's `or 0` with the isinstance coercion).
- langgraph: flattens the nested input_token_details then one from_mapping.
- pydantic_ai keeps its multi-name version fallback (the one case the shared
constructors don't cover); toolkit UsageRecord keeps a local coercion rather
than importing a private SDK helper across the test boundary.
Adds direct constructor tests (None/non-mapping/missing/non-int edge cases).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(adapters): Emit.USAGE for gemini, crewai, letta, opencode
Complete the per-adapter usage rollout using the shared TurnUsage.from_object/
from_mapping constructors:
- gemini: GenerateContentResponse.usage_metadata, summed across the tool loop.
- crewai: LiteAgentOutput.usage_metrics (dict), read once on kickoff success.
- letta: LettaResponse.usage in per_room mode; shared-mode stream has no
aggregate usage object, so it stays honest N-A (documented inline).
- opencode: defensive read of the assistant info.tokens dict (unmodeled server
payload), tracked per assistant message and summed at turn end — no schema
change to the opencode integration types.
Each adds Emit.USAGE to SUPPORTED_EMIT and a _usage_from_* mapper. Tests:
gemini/letta/opencode mappers in test_usage_mapping.py; crewai's in
test_crewai_adapter.py (under its sys.modules mock); the test_usage smoke fans
across all 9 live-capable adapters (letta is E2E-pending → unit-covered only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(usage): codex into unified seam; fix pydantic_ai/crewai benign-path emit
Live E2E (core/google/crewai lanes) surfaced two real defects and a coverage
gap in the Emit.USAGE rollout; fix all three properly (no reruns/loosened
asserts) and harden the cross-adapter smoke.
- codex: wire into the unified Emit.USAGE seam (SUPPORTED_EMIT + a per-turn
_turn_usage mapper off its per-turn token deltas, emitted via emit_usage in
_emit_turn_outcome). Previously codex emitted usage only via its legacy
codex_*_tokens task metadata, which capture.usage() (keyed on band_usage)
never read.
- pydantic_ai: emit usage on the benign empty-final-response path too (it
returned early, skipping the emit). Sum from the captured ModelResponses when
no result event fired; factor out _usage_from_usage_obj.
- crewai: capture usage from LLMCallCompletedEvent on the event bus and emit in
finally, so usage is recorded on the benign empty-answer path (kickoff raises,
no result). Fixes a wrong handler signature (the bus calls handler(source,
event)) that silently swallowed the accumulator.
E2E smoke (test_usage): derive the fan from exclude={CREWAI_FLOW} (letta
auto-excluded as e2e_pending); assert exactly one record per turn and a
cache-aware plausibility floor (input+cache_read+cache_write >= 20, since
caching adapters like claude_sdk report ~7 fresh input tokens + ~87k cached)
plus a realistic ceiling. Add deterministic loop-summing unit tests
(anthropic/crewai/pydantic_ai) proving per-call usage sums into the turn total.
Live-verified: anthropic, langgraph, pydantic_ai, claude_sdk, agno (core);
gemini, google_adk (google); crewai. codex/opencode run in CI's backends lane.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(usage): address code-review findings (concurrency, cache, task pollution)
Fix the review findings at the root rather than patching:
- crewai concurrency (#1/#3/#5): replace the per-turn handler on CrewAI's
process-global event bus with ONE persistent handler routing usage into a
per-turn contextvar accumulator — no cross-room contamination (CrewAI
copy_context()s at emit, so the contextvar is carried into the handler thread;
validated by a two-room prototype). Prefer the authoritative per-kickoff
result.usage_metrics on success (race-free, includes cache); fall back to the
contextvar accumulator only on the benign empty-answer path. Restores cache
capture (mapped incl. nested prompt_tokens_details).
- task-event pollution (#2): add a shared is_usage_event() discriminator and use
it so the ACP event_converter skips usage events (no bogus plan entry) and
Tasks.read excludes them (no collision with lifecycle tasks). Durable
first-class `usage` type tracked as INT-933.
- cache semantics (#4): define one cross-provider convention on TurnUsage —
input_tokens = total prompt incl. cache; cache_read/write = subset breakdown.
Add from_object/from_mapping `cache_in_input` flag; anthropic + claude_sdk fold
cache into input (their native input excludes it). total_tokens is now correct
for every adapter; E2E floor simplified to input_tokens>=20.
- cleanup: unify from_object/from_mapping via a shared _build.
Tests: cache-fold + is_usage_event unit tests; updated anthropic/claude_sdk
mapping expectations. Live-validated core (5/5), google (2/2), crewai (1/1).
Refs INT-933.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(usage): remove crewai usage, raw cache convention, review cleanup
Address the round-2 code review at the root:
- Remove CrewAI usage capture: its result.usage_metrics is a cumulative
lifetime counter (not per-turn, shared across concurrent rooms) and its
per-call events come in two incompatible provider shapes. Drop Emit.USAGE
from SUPPORTED_EMIT and all the contextvar/handler/mapper machinery; exclude
crewai from the usage smoke. Proper crewai usage tracked as a follow-up.
- Cache semantics: drop the per-adapter `cache_in_input` fold — cache-in-input
is provider-dependent, so a static per-adapter flag is wrong for multi-provider
adapters (agno+OpenAI includes cache, agno+Anthropic excludes it; opencode is
additive). TurnUsage fields are now the provider's RAW values; consumers use
input+cache_read+cache_write for prompt size. Fixes agno/opencode under-count
at the root and simplifies from_object/from_mapping.
- Remove Linear issue IDs from code comments (CLAUDE.md rule); fix stale test
docstrings; E2E floor back to the cache-inclusive sum.
Live-validated: core 5/5, google 2/2 (usage smoke now 9 cells, crewai excluded).
Unit 3468 pass; ruff + pyrefly clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(usage): fix stale docstrings + drop gratuitous crewai import reformat
- crewai.py: revert the band.core.types import to one line (it was only
multi-line to add TurnUsage, since removed) — keeps the net PR diff to the
explanatory NOTE.
- TurnUsage.to_dict / UsageRecord.total_tokens: correct docstrings for the raw
cache convention (to_dict feeds metadata, not content JSON; total_tokens is
raw input+output, not cache-normalized).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(opencode): remove dead timeout handler; emit usage on timeout; gate capture
Careful review of the opencode adapter surfaced four issues:
- on_message had an unreachable `except asyncio.TimeoutError` that duplicated
_watch_turn_completion's timeout handling (the watcher owns the turn timeout
via asyncio.wait_for and swallows it; nothing on_message awaits re-raises it).
Remove it — one timeout owner.
- _watch_turn_completion emitted usage only on the success branch; emit on the
timeout branch too (tokens spent before a timeout are still spent).
- The per-message usage capture ran on every message.updated even when
Emit.USAGE was off; gate it (keep assistant_message_ids.add unconditional).
- Clarify the _emit_turn_usage docstring.
Live-validated: opencode usage cell passes end-to-end (session→prompt→SSE→usage).
opencode unit tests 24 pass; ruff + pyrefly clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pydantic_ai): benign-path usage must sum only this run, not history
capture_run_messages() records the passed message_history + the new turn (it's
contextvar-scoped, so concurrency-safe). Summing usage across the whole captured
list double-counted every prior turn's ModelResponse on the benign
empty-final-response path (turn 2 reported turn1+turn2, etc.) — the same
cumulative over-count that got crewai removed. Snapshot the history length and
sum only captured[history_len:]. Verified with a TestModel two-turn prototype;
the happy path (result.usage()) was already per-run. Reviewed all other adapters
(agno RunOutput.metrics is per-run; others sum within one turn or use per-turn
deltas) — no other instance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(crewai): trim the deferred-usage note to its essential why
The note explaining why CrewAI omits Emit.USAGE had grown to seven lines of
provider-shape detail. Keep the load-bearing fact — usage_metrics is a
cumulative-lifetime counter, not per-turn, so per-turn capture is deferred —
and the actionable warning not to add emit_usage here.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): matrix smoke that per-turn usage is not cumulative across turns
Adds test_usage_not_cumulative_across_turns across every usage-emitting
adapter (same @per_adapter fan/exclusions as the single-turn usage smoke).
It's the live guard for the per-turn convention: review found adapters that
reported a cumulative count (a run/session total that grows every turn —
pydantic_ai's benign fallback summed the replayed history; crewai's
usage_metrics is a lifetime counter).
The two turns are deliberately asymmetric — a long first turn, a one-word
second turn — so the check is immune to LLM reply-length variance: a correct
per-turn second record sits far below the long turn, while a cumulative one
(long + tiny) rises to ~= the long turn. That's a scale-immune 'small vs
large' split, unlike a fragile ' 1x vs 2x' ratio of two equal turns. Adds a
COST_MULTI_TURN_AGENT shape whose prompt defers reply length to the user so
the test can drive that asymmetry.
Live-validated green on anthropic, claude_sdk, agno, gemini, google_adk, and
opencode; langgraph/pydantic_ai/codex were blocked only by an OpenAI account
quota (429), not the test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(usage): fold disjoint reasoning tokens into output across all such adapters
Reasoning/thinking tokens were dropped for providers that report them
DISJOINTLY from output (their own totals are input + output + reasoning):
codex, opencode, gemini, google_adk. Those turns undercounted cost on
reasoning models.
Root fix at the shared builder: TurnUsage._build/from_object/from_mapping
grow an optional reasoning= kwarg that folds reasoning into output_tokens,
so the four-field schema stays consistent with providers that already count
reasoning inside output (Anthropic thinking, OpenAI/LangChain completion
tokens — anthropic, claude_sdk, langgraph, letta need no change).
agno is deliberately left unfolded: as a multi-provider aggregator its
output_tokens already includes reasoning for some backends (OpenAI) and
excludes it for others (Gemini), so no static fold is correct for every
backend — folding would double-count OpenAI-backed runs. This mirrors the
RAW cache convention (don't apply a normalization the aggregator can't
guarantee).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pydantic_ai): scope benign-path usage by message identity, not position
The empty-final-response fallback summed captured[history_len:] to get this
run's usage, where history_len was the length of the RAW prior history. But
pydantic-ai runs _clean_message_history on the passed history — merging
adjacent same-type messages (e.g. the injected participants + contacts
requests) — so capture_run_messages holds a SHORTER, merged history and the
positional boundary slips, dropping the turn's leading response usage.
Root fix: snapshot the prior messages' identities before the run and keep
only captured messages whose id() wasn't in it. Real API ModelResponses are
never merged by _clean_message_history, so they keep identity; a newly-merged
prior request rides along but carries no usage, so the ModelResponse-only sum
ignores it. Immune to merge/reorder/count changes. Extracted _new_run_messages
with a regression test that reproduces the merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(usage): emit accumulated usage on every turn exit, not only clean runs
- anthropic/gemini/langgraph/google_adk/letta: emit_usage now runs from a
finally, so a turn that fails after a successful model call still reports
the tokens it spent (a first-call failure accumulates nothing and emits
nothing)
- pydantic_ai: two emit sites collapse into one finally-based emit; the
captured-messages fallback covers any raise and is gated on Emit.USAGE so
the fallback history scan never runs when usage emission is off
- opencode: the per-turn usage dict is turn-owned (fresh dict per
_begin_turn, the watch task drains the instance it captured), closing the
race where a new turn cleared usage before the previous turn's emit
- letta: response observation extracted to _process_response_messages so the
finally covers reporting/relay failures too
- emit_usage docstring now owns the finally-safety contract
- tests: shared usage-payload extractors built on is_usage_event; regression
tests for mid-loop failure, hard-raise fallback, and the opencode race
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(usage): harden turn-exit emission against cancellation and turn races
Follow-up to the finally-based usage emission, addressing code-review findings:
- SimpleAdapter.emit_usage: skip the emit when the calling task is being
cancelled (shutdown, a turn timeout), so teardown never blocks on a REST
send and a CancelledError can't fire mid-send and mask the original
exception. Covers all six adapters that emit from a finally.
- opencode: snapshot the turn's future and usage dict before the prompt
await, and register the watch task only while the turn is still current.
prompt_async can span the whole turn (session.idle may arrive mid-POST),
so reading room_state afterwards could wire this turn's watcher to the
next turn's state.
- google_adk: nest emit_usage inside runner.close() so a cancellation during
the emit can't skip close() and leak the per-message runner.
- langgraph/google_adk: gate per-event usage accumulation on Emit.USAGE so
the per-token stream loop does no TurnUsage churn when usage is off.
- pydantic_ai: build prior_message_ids only when usage is enabled (its sole
consumer is the now-gated fallback).
- tests: cancellation-skip and turn-race regressions; shared usage-payload
helpers take the tools double; one ModelResponse factory per test file.
Verified live: anthropic usage smokes green (per-turn, non-cumulative);
cancellation guard driven through the real adapter surface.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…411) * feat: L0–L4 baseline live-E2E coverage as matrix scenarios [INT-930] Close the live end-to-end coverage gaps derived from the L0–L4 conformance specs (INT-523/524/525/527/528), as scenario-named @per_adapter matrix tests that run across the whole adapter registry and reuse the existing baseline toolkit. No "levels" concept leaks into code or file names. Toolkit additions (the only new plumbing): - Replies.assert_at_most — the one sanctioned upper-bound runaway guard, localized off the floors-only ContentAssertions base (loop suppression only). - AdapterCell.run_many — K co-resident same-adapter instances in one room, started concurrently. - running_members — concurrent-start helper factored out of the agents fixture so the fixture and run_many share one implementation instead of drifting. - ReplyCapture.turn_boundary — server-timestamp `since=` accessor, DRY-promoted from test_isolation's private _turn_boundary (both now use it). Scenario tests (matrix-driven, floors-only assertions): - L0: identity/roster probe, participant management (invite+directed msg+remove), loop suppression (peer bounce, no self-dispatch runaway). - L1: custom prompt takes effect + coexists with platform tools. - L2: burst no-drop via delivery status + spanning multi-fact recall. - L3: same-adapter concurrency gate (incl. codex/opencode), peer-initiated delegation + self-recall. - L4: cold-restart dedup asserted model-independently at the delivery layer, plus the restart token split (replay vs new inference). Retirements (subsumed, removed in the same change): - test_peer_actor (folded into loop suppression). - test_agent_recalls_earlier_facts (folded into burst recall). Shared driving glue added to sample_agents/sample_tools; PR-run unit tests for the new primitives added to framework_conformance/test_toolkit_helpers. Validated live on anthropic (core lane): all 9 new test functions pass. Two SDK follow-ups filed from the live run: INT-942 (cold-boot rehydration replays history and a weak model re-executes it) and INT-943 (surface participant descriptions in the passive roster). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(e2e): harden topology-marker reads + guard against stacked decorators Follow-up to the L0–L4 baseline coverage work, hardening how the agent fixtures read a topology decorator's payload off a test node. - `MarkerPayload.from_node` — one validated read for the `@per_adapter` / `@with_adapters` payloads: fail-loud `UsageError` on a missing or malformed marker instead of a cryptic `IndexError` / `AttributeError` deep in a fixture, and the `isinstance` check makes the payload type real (the annotation was an unchecked cast on `Mark.args[0]`). `WithAdapters` / `PerAdapter` inherit it, and the four fixture read sites (`cell`, `peer`, `agent`, `agents`) now use it. - `agent_wiring` — reject the same topology decorator applied more than once (`get_closest_marker` would silently drop the extra), with unit tests. - Rename the wiring-rules test double `_FakeItem` -> `FakeItem`. - Remove `docs/baseline-e2e-coverage-plan.md` (its content now lives in the PR description; no longer needed as a tracked file). Behavior-preserving for valid tests; validated via the PR-run wiring/helper suite, ruff, and pyrefly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(e2e): apply code-review findings - test_burst_recall: size the no-drop barrier to the BURST_SIZE-turn backlog (`deadline_s = e2e_timeout * BURST_SIZE`) — the per-turn default would false-fail on a slow backend, and the wall-clock `timeout(extra=)` doesn't cover the barrier; widened the wall-clock backstop to match. - AdapterCell.run_many: fold to concurrent `self.running` members so provisioning is concurrent too, matching the @with_adapters group path (`_running_group_member` → `cell.running`); drops the serial provision loop. Behavior-preserving — same yielded list, distinct identities, validation guards stay ahead of it. - test_peer_delegation: reuse `since().from_sender()` in the cascade-barrier predicate instead of a hand-rolled index-slice generator (matches the assertion on the next line). - test_rehydration_idempotency: document why the delivery-layer dedup negative is sound — the capture opens before boot and the offline barrier is the positive control, so it is not vacuous. Confirmed live: test_peer_delegation[anthropic] green (exercises run_many + the predicate). ruff / pyrefly / run_many unit tests / E2E collection all pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(e2e): agent-driven band_create_chatroom matrix coverage Add a tool-loop-matrix scenario proving an agent creates a chat room via band_create_chatroom, plus the small toolkit plumbing to observe and reap an agent-created room (which is agent-owned with no human participant, so the user/observer client never sees it and it has no title to discover by): - ResourceManager.agent_room_ids(agent) — agent-scoped list_agent_chats read; a before/after snapshot around the turn surfaces the created room's id. - ResourceManager.adopt_room(room_id) — register a room this run didn't create via provision_room into the same teardown-reap path; idempotent. Reaped via the user-scoped delete, which the platform authorizes because the test user owns the agent that owns the room (agent-owner delete authz). - test_create_chatroom — the tool fired (tool_call events) AND exactly one new room landed in the agent's chat list; adopts before asserting so a failing assertion never leaks the room. Mirrors test_tool_round_trip's two-half shape. - CREATE_CHATROOM driving instruction + a PR-run adopt_room unit test. band_create_chatroom is outside the L0–L4 spec scope; this closes a separate platform-tool coverage gap. Validated live on anthropic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Squash the Copilot SDK adapter stack onto dev. Includes the adapter and converter, per-turn usage capture, core-lane E2E wiring, examples, ask_user handling, lifecycle/session hardening, single-instance runtime guard, and conformance coverage.
Squash the Letta lane stack onto dev. Includes the self-hosted Letta MCP bridge, Letta CI lane setup, pydantic-settings cleanup, Docker/example fixes, adapter lifecycle hardening, and baseline conformance updates. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: Windows support for band-sdk-python [INT-947]
Validate and fix the SDK so it installs, runs, and passes its unit +
live-E2E suites on Windows (Python 3.11 and 3.12).
Async cancellation hangs (Windows Proactor loop, exposed by CPython
bpo-43389 and the 3.11-only trace-function cancellation bug gh-106749):
- execution.py: Phase 2 loop uses asyncio.timeout() instead of
wait_for(queue.get(), ...); wait_for wraps the awaitable in a child
task whose cancellation can be lost on 3.11, wedging shutdown.
- working_state.py / mcp_server.py: replace suppress(CancelledError)
around `await task` with gather(task, return_exceptions=True) so an
external cancel aimed at the caller propagates instead of being
silently swallowed.
Cross-platform file/path handling:
- repo_init.py: replace Unix-only fcntl.flock with the filelock package
(fcntl on POSIX, msvcrt on Windows); accept a repo.path that is
absolute under either POSIX or Windows convention.
- codex.py: measure turn duration with perf_counter() so an instant
turn is non-zero on Windows (monotonic() ticks too coarsely).
Signals:
- shutdown.py: add_signal_handler()/SIGHUP degrade to a signal.signal()
fallback on the Proactor loop instead of raising NotImplementedError.
CI:
- ci.yml: run the test job on windows-latest as well as ubuntu-latest
(Python 3.11 + 3.12), using bash on every OS for step portability.
Tests:
- test_client_adapter.py: assert against os.path.abspath (cwd is
absolutized) rather than a hardcoded POSIX path.
- test_shutdown.py: cover the Windows signal.signal() fallback path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(e2e): run the full baseline matrix on Windows (all lanes) [INT-947]
Add an OS dimension to the E2E workflow so every lane runs on both
ubuntu-latest and windows-latest, not just Linux.
- e2e.yml: new `os` workflow_dispatch input (all|ubuntu|windows); the
`lanes` job emits the lane x OS cross-product ({lane, extra, os,
runner}); the `e2e` job runs-on ${{ matrix.runner }} with bash as the
default shell on every OS (Git Bash ships on windows-latest) so the
setup scripts and uv commands run identically. Registry/lane guards
are unchanged and still pass (test_e2e_lane_drift).
- setup-codex.sh: hand `codex` a native CODEX_CWD path (cygpath -w) so
the Windows .exe understands it; no-op on Linux (cygpath absent).
- setup-opencode.sh: also write a cwd-local opencode.json (native
opencode on Windows reads config from %APPDATA%, not ~/.config, but
honours a project-local config on every platform).
Validated live on Windows 11: core (anthropic/langgraph/agno) and
backends (codex, via the native codex CLI subprocess) baseline smokes
pass against the real platform.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(e2e): invoke backend setup scripts via `bash` for Windows [INT-947]
On the windows-latest runner, running a .sh directly relies on the exec
bit surviving checkout and Git Bash resolving the shebang. Invoke the
codex/opencode setup scripts as `bash <script>` so neither matters;
no-op on Linux.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(e2e): UTF-8 stdout on Windows + per-lane concurrency [INT-947]
- PYTHONUTF8=1 / PYTHONIOENCODING=utf-8 in the e2e job env: the
windows-latest runner defaults Python stdout to cp1252, so a framework
that logs non-ASCII (CrewAI's event bus emits 🤖/🔧/✅) raises
UnicodeEncodeError mid-turn. UTF-8 mode is the canonical fix; no-op on
Linux.
- Scope concurrency per (lane, OS) on the e2e job instead of one
workflow-wide group, so re-running a single lane to iterate on a fix
cancels only that lane+OS, not the whole matrix. Cross-lane races on
the shared platform stay safe: runs provision uniquely-named agents and
the orphan sweep only reaps agents older than the max-age threshold.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(codex): parse current app-server token-usage schema [INT-947]
The codex app-server sends thread/tokenUsage/updated as
{"tokenUsage": {"total": {...cumulative...}, "last": {...}}} with keys
inputTokens/outputTokens/reasoningOutputTokens/totalTokens, but
CodexTokenUsage.update() read a stale flat {"usage": {...}} shape with
reasoningTokens — so every counter parsed to 0 and Emit.USAGE no-oped
(test_usage[codex] observed no records). Read the nested cumulative
`total` block, accept reasoningOutputTokens, and keep the old flat shape
as a fallback so a protocol rollback doesn't silently zero counts.
Adds a concise DEBUG log of the parsed counts (no content/PII) and a
unit test pinning the current schema.
Also: add the E2E workflow status badge to the README.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* revert: drop speculative mcp_server cancellation change [INT-947]
The earlier suppress(CancelledError)->gather change in the MCP server's
lifespan teardown wasn't backed by any observed failure (the MCP server
isn't exercised by the failing Windows E2E lanes) and subtly regressed
error visibility: if http_task had already finished with a non-cancel
exception, suppress(CancelledError) re-raised it, but
gather(return_exceptions=True) swallows it. Keep the branch scoped to
proven Windows fixes; revert to the original teardown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(prompts): require the final answer to go through band_send_message [INT-947]
Weak models (documented for Gemini 2.5 Flash; also seen on the OpenAI-backed
langgraph/crewai/pydantic_ai cells) end a turn with the answer as plain
assistant text after using tools. Band only delivers band_send_message tool
calls, so that answer is silently dropped and the agent looks like it never
replied — the root cause behind identity_and_roster and the other
"marker not in reply" E2E failures, on Linux and Windows alike.
The base prompt only said "Plain text output is not delivered", which weak
models slip past on the final turn. Strengthen the delivery contract to call
out the preamble-vs-answer trap explicitly. Scoped to HOW to deliver an answer,
not WHETHER to speak, so the silence/mention design is unchanged.
Verified live: gemini identity_and_roster now emits identity + roster via
band_send_message and passes first try (was preamble-only before).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pydantic_ai): raise validation retries from the default 1 to 3 [INT-947]
pydantic-ai defaults to retries=1 for tool-arg and final-output validation.
With a tool-only agent on a small model, one retry is too tight: the model
occasionally needs another attempt to emit a valid tool call (e.g.
band_create_chatroom) or a non-empty final answer, and pydantic-ai otherwise
raises UnexpectedModelBehavior and marks the turn permanently failed
(create_chatroom[pydantic_ai], "Exceeded maximum retries (1) for output
validation"). A modest budget makes the turn resilient without masking a
genuinely stuck run (it still raises after the budget).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): let crewai_flow accept peer-initiated turns in the baseline [INT-947]
crewai_flow discards agent-authored messages unless configured as a downstream
router (accept_agent_initiated, default False to avoid A<->B echo loops). In
the baseline rooms it is a live participant that must react to peers — e.g. the
loop_suppression positive, where a peer's directed probe has to drive a turn —
so it was failing "no text message contained the marker" on every attempt.
Opt the baseline builder into router behavior. Safe: the runtime already drops
an agent's OWN messages before dispatch (execution.py self-filter), so
crewai_flow reacts to peers without ever looping on its own output — which is
exactly what the loop_suppression ceiling then verifies.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(prompts): state the delivery rule generally, not test-fitted [INT-947]
The first pass at strengthening the band_send_message rule was overfitted: it
enumerated the exact tools from the identity_and_roster test
(band_get_participants/band_lookup_peers/...) and echoed gemini's literal
"let me check" failure phrase. Neither belongs in a base prompt shipped to
every agent.
Keep only the general, timeless principle — a band_send_message call is the
only way words reach the room, and that includes the final answer composed
after tool use (the one point the original rule left implicit and weak models
slipped past). Re-validated live: gemini identity_and_roster still delivers
identity + roster via band_send_message and passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(e2e): fit provisioned names within the 24-char mention-handle cap [INT-947]
Root cause of the identity_and_roster failures on crewai/google_adk/langgraph:
the platform caps a mention handle at 24 chars, so a peer name longer than that
truncates in the rendered @mention (`…-invitable` -> `…-invita`). When the model
reports the roster by @mentioning peers (rather than stating plain names — a
model-nondeterministic choice, hence the cross-OS flakiness), the invitable
peer's full name never appears and the self-sourced assertion fails.
`e2e-band-{run_id}-{label}` with an 8-char run_id put every asserted peer name
over the cap (27 chars). Shrink the run_id to 5 hex so the asserted labels
(member/invitable/nonmember, ≤9) land at ≤24 — handle then equals name, so a
mentioned peer surfaces its full name and the assertion holds whether the model
states or @mentions it. NAME_PREFIX (the orphan-sweep guard) is unchanged, so the
sweep stays correctly scoped; 5 hex is ample entropy given its run-id + age guards.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pydantic_ai): pass rendered prompt as instructions, not system_prompt [INT-947]
pydantic-ai materializes `system_prompt` as a single SystemPromptPart only on
the first request; thereafter it ages into buried history. By the time the model
composes the post-tool reply, the prompt sits beneath the user prompt, tool call,
and tool return, so a custom rule that must ride *every* reply (e.g. a required
marker word) gets dropped — while the anthropic adapter, which re-sends `system=`
on every call, keeps it in force.
Use pydantic-ai's `instructions=` instead: it is re-attached to each ModelRequest,
including the post-tool-return request that generates the reply, mirroring the
anthropic behaviour. Fixes custom_prompt[pydantic_ai] dropping the every-reply
marker.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(agno): frame the Band contract via description so it precedes dev instructions [INT-947]
Agno assembles its system message as description -> <instructions> -> ... ->
additional_context. The adapter appended the Band operating contract to
additional_context, i.e. *after* the developer's own instructions — the reverse
of every other adapter, where render_system_prompt is the authoritative frame and
developer steering trails it. That put the anti-injection Security rule last (most
recent), so even a capable model over-applied it and refused benign peer requests:
loop_suppression[agno] failed on both OSes while the same model under the anthropic
adapter complied.
Write the Band block to `description` (prepended, preserving any developer
description) so it frames the agent ahead of the developer's instructions,
reproducing the cross-adapter ordering.
Also clarify the memory subject-scope guidance: a memory the sender frames in the
first person ("me"/"my"/"I") has that sender as its subject — resolve the sender's
id, not the agent's own. Fixes memory_subject_scope_inferred storing the agent's
own id as subject_id.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(opencode): seed a freshly-created session with in-session history [INT-947]
OpenCode delegates conversation memory to its server session (reused session_id)
and only replayed converted room history on the 404-recovery path. When turn 2
created a fresh session because no prior id was recoverable, the available history
was dropped, so the model saw only the latest message and answered "I don't have
access to my memory from previous conversations" — context_recall[opencode].
Replay history whenever a new server session is `created` (a superset of the old
404 case): a new session holds no context, so seed it; a reused session already
holds it server-side, so don't double it. The now-redundant restored_missing_session
flag is removed from _ensure_session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(claude_sdk): isolate the bridged agent from ambient Claude Code config [INT-947]
ClaudeAgentOptions.setting_sources was left unset, so the SDK defaulted to loading
the host's user + project settings (~/.claude and ./.claude). Filesystem skills and
subagents then surfaced as `Skill`/`Agent` tools, and the enlarged toolset tripped
ToolSearch — which withholds tool definitions (including our `mcp__band__*` tools)
behind a search step. On a Windows CI runner (with a global Claude Code install) the
model wandered through ToolSearch/Bash/Agent/Skill instead of calling the Band tool,
failing tool_round_trips[claude_sdk]; Linux's leaner session called it directly.
Pin setting_sources=[] so a bridged Band agent's capabilities are defined by the
adapter, not by whatever config sits on the host — deterministic across machines and
CI runners. Built-in tools stay available, so coding-agent use is unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(claude_sdk): disable ToolSearch deferral so Band tools stay in context [INT-947]
Completes the claude_sdk lockdown started by setting_sources=[]. The CLI enables
tool search once the tool set is large enough, withholding tool definitions behind
a `ToolSearch(select:...)` step. A model can then spend its turn searching instead
of calling the Band tool — the failure seen for tool_round_trips[claude_sdk] on the
windows runner. Set ENABLE_TOOL_SEARCH=false via the (merged, non-replacing) env so
the reply path (band_send_message) and any custom tool are always directly visible,
making tool availability deterministic across hosts. Built-in tools remain available.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(e2e): derive run-id length from the handle cap + enforce it [INT-947]
The 8->5 run-id shrink relied on a hand-computed "9+5+1+9=24" left in a docstring
with nothing enforcing it — add a longer @mentioned peer label and names silently
truncate again. Replace the magic number with a derived length
(CAP - len(prefix) - 1 - MAX_MENTIONED_LABEL_LEN) and a PR-run guard,
test_mentioned_peer_names_fit_handle_cap, that fails loudly if the prefix widens or
a mentioned label outgrows the budget. Behaviour is unchanged (still 5 hex).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(claude_sdk): drop ENABLE_TOOL_SEARCH flag; make setting_sources configurable [INT-947]
The ENABLE_TOOL_SEARCH=false env flag was the wrong lever: it coupled to a CLI
internal and regressed create_chatroom[claude_sdk] on BOTH OSes (the tool fired but
no room landed) while passing on the commit before it. The real fix is
setting_sources=[]: loading no host config keeps the tool set lean, so ToolSearch
never engages and the Band tools stay directly in context — no env override needed.
Also make setting_sources a constructor param (default [] = isolated). A caller
building a coding-style agent that wants its host skills/subagents can pass
["user", "project"] to opt back in, rather than the behaviour being hardcoded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(agno): surface swallowed run errors so failed turns retry [INT-947]
Agno catches exceptions inside Agent.arun (model/API failures included) and
reports them as an error-status result instead of raising: non-streaming runs
return a RunOutput with status=error, and streaming runs emit a RunErrorEvent
and never yield a final output. The adapter treated both as a successful turn
that produced no reply, so the runtime marked the message processed and the
turn died silently — observed in CI as empty-capture failures on flaky-network
runners while the same errors under other adapters were retried and recovered.
Raise AgnoRunError at the run chokepoint for both shapes, routing through the
existing failure path (generic error event to the room, detail to the agent
log, re-raise) so the message is marked failed and the platform retries —
restoring the cross-adapter contract.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(runtime): translate @[[uuid]] mentions in the current message [INT-947]
The platform normalizes mentions in stored content to @[[uuid]] form. History
formatting already translates these tokens to @handle, but the live message
reached adapters raw — so the only UUID the LLM ever saw in a turn was the
mention of itself. Probing the real model showed it then grabs that uuid for
id-shaped tool arguments (e.g. band_store_memory subject_id resolved to the
agent's own id instead of the sender's).
Run replace_uuid_mentions over the current message at both PlatformMessage
construction sites (DefaultPreprocessor and the one-shot runtime), matching
the history behavior, and state in the memory guidance that a handle or name
is never a valid subject_id.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pydantic_ai): run CustomToolDef handlers via shared execution semantics [INT-947]
The portable CustomToolDef wrapper called the handler directly as a sync
passthrough: an async handler returned an unawaited coroutine (failing
serialization) and a zero-argument handler was called with one positional arg
and raised TypeError — both work in every other adapter, which routes through
the shared custom-tool executor.
Extract the post-validation half of execute_custom_tool into
invoke_validated_custom_tool and route the pydantic-ai wrapper through it,
passing the InputModel instance pydantic-ai already validated. Passing the
instance (not a model_dump round-trip) keeps models with field aliases
working, since re-validating dumped field names would reject alias-only
models.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(e2e): align baseline URL defaults with the SDK defaults [INT-947]
BaselineSettings defaulted BAND_REST_URL/BAND_WS_URL to band.ai hosts while
the SDK (and docs) default to app.band.ai, so a live run configured with only
BAND_API_KEY_USER connected to the wrong host. The fallbacks exist only so
settings construction never raises when E2E is disabled — point them at the
same hosts the SDK itself uses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: restore required test check names
* Revert "ci: restore required test check names"
This reverts commit d21dc5d.
* fix(tests): allow one Windows timer tick in the idle-timeout wait assertion [INT-947]
Windows event-loop timers tick at ~15.6 ms granularity, so a 0.1 s
asyncio timeout can complete a few milliseconds early and the strict
elapsed >= 0.1 assertion flakes on windows runners (observed:
0.0939... >= 0.1). Allow one tick of slack.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: run crewai tests on Windows
* fix(e2e): add wait_for_reply barrier to close reply-capture race [INT-947]
The reply's message_created frame and the delivery-status message_updated
PROCESSED frame are independent, unordered platform events (a reply-row INSERT
vs a trigger-row UPDATE, on separate backend paths). So PROCESSED can reach the
observer before the reply frame, and reading capture.messages the instant
wait_for_processed returns races that gap — the cross-adapter "no agent messages
were captured" flake, seen on agno/parlant and latent everywhere.
- Add ReplyCapture.wait_for_reply: waits for PROCESSED AND the reply frame,
optionally scoped by sender_id/since, and returns the reply window to assert on.
- Migrate every reply-text-asserting site onto it; durable-state reads
(tool_calls/usage/events/memory) and ceiling/setup barriers keep
wait_for_processed.
- Add a deterministic fast-lane guard (tests/toolkit) that forces the racey
frame order — proves the bug and guards the fix without a live platform.
- Tighten flaky(reruns=2) masks whose only stated reason was a transient timeout
to rerun_except=["AssertionError"], so a real assertion regression fails loud.
- Correct the baseline README barrier guidance (drop the "processed implies the
reply is buffered" claim; document the two-barrier choice).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(e2e): default wait_for_reply sender to the trigger's recipient [INT-947]
The agent that processed the trigger is almost always the one whose reply you
want, so scope the reply barrier to recipient_id by default instead of "any
agent". This is the safe zero-config choice for multi-agent rooms (a peer's own
captured message no longer satisfies the wait) and lets call sites drop the
sender_id argument entirely; the kwarg stays as an escape hatch for the rare
cross-author case. Behaviour-preserving for the migrated single-replier sites.
Add a guard test that the default scopes to the recipient.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(e2e): drop now-redundant sender_id from wait_for_reply calls [INT-947]
wait_for_reply now defaults sender to the trigger's recipient, so the explicit
sender_id=<recipient> args are redundant. Remove them so the call sites read
clean and consistent; the kwarg remains for the rare cross-author case.
Behaviour-preserving.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(e2e): drop wait_for_reply's sender_id param [INT-947]
Every call scoped the reply to the same agent it barriered on, and that isn't a
coincidence: the wait keys on recipient_id's PROCESSED signal, so it can only
sensibly wait for recipient_id's own reply — a different author's reply isn't
gated by that signal at all (those cases use wait_until). So sender_id was
speculative generality: remove it and always scope to recipient_id. Update the
guard, README, and skill guidance to match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): strengthen reply-capture guard + fix stale barrier docs [INT-947]
- Guard: rewrite the weak "empty buffer" test to assert the real race state
(delivery reaches PROCESSED while the reply is not yet buffered, both at once)
and pin the reply-optional contract; add a since-cursor test (a prior turn's
reply must not satisfy this turn's wait) — the load-bearing reused-capture
behaviour that was untested.
- Docs: drop the "processed only after the reply is emitted" phrasing from the
test_tool_calls / test_usage docstrings (processed is stamped when the handler
completes; a reply is optional), and note test_tool_calls uses wait_for_reply
for its reply assertion. Fix the test_noisy_room comment, which still claimed
"no WS-ordering race" and described wait_for_processed though the code now
calls wait_for_reply.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(parlant): honest response-wait — retry to a budget, never mask a stall [INT-947]
The wait loop dropped a turn after a single empty poll window and, on some
paths, on a transient empty read after a positive signal. Poll in short windows
up to a configurable total budget (constructor: response_timeout/response_poll)
so a slow cold-start response is delivered; on a genuine stall, give up with no
reply. A Parlant preamble is an acknowledgment, not an answer, so it is never
forwarded as the reply — a stalled turn fails truthfully rather than looking
answered. Floor parlant at >=3.3.2 (older versions stall the in-process server's
post-preamble generation via event-loop starvation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(e2e): keep the pytest-timeout backstop above the reply barrier [INT-947]
A test with no timeout marker got a hard pytest-timeout equal to the soft reply
barrier, so a slow turn could SIGALRM-kill the process (skipping async unwind,
orphaning agents/servers) before the barrier raised a clean TimeoutError. Give
the backstop a fixed margin over the turn budget so the barrier always fires
first. Guarded by tests for _effective_timeout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(agents): comments describe code as-is, not a changelog [INT-947]
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): robust usage boundary, natural greeting rubric, roster clarity [INT-947]
- usage not-cumulative: compare the one-word turn against turn1 itself (the exact
cumulative boundary: a lifetime counter reports turn2 = turn1+turn2 >= turn1),
not a 0.5 fraction that reasoning-token folding (google_adk) makes brittle; and
drive turn1 to a reliably long reply.
- mutual-greeting judge: accept natural greetings ('nice to meet you', 'hey', …),
not only 'hello'/'hi', while still requiring both agents to greet.
- roster probe: ask for peers 'by name' so the invitable-peer lookup is unambiguous.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: intent-oriented lock timeout + drop redundant reply assertion [INT-947]
- repo_init: pass the timeout to lock.acquire() at the call site instead of the
FileLock constructor, so "acquire, but give up after timeout_s" is obvious
where it acquires (behavior identical — acquire() fell back to the same value).
- concurrent-instances: wait_for_reply already blocks until each instance's reply
is captured or raises, so the per-instance assert_present was a tautology.
Collapse to the sequential barrier loop whose completion is the proof.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(readme): remove the E2E badge [INT-947]
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix Parlant response wait unit test hang
* Fix E2E codex ACP setup and recall prompt
* fix(parlant): use perf_counter for the response-wait deadline [INT-947]
perf_counter is the highest-resolution monotonic clock; its resolution holds up
on Windows, where time.monotonic() is coarse. Matches the codex adapter's clock.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(acp): resolve the launcher path so codex/npx spawns on Windows [INT-947]
ACPClientAdapter passed a bare launcher name (e.g. 'npx') to spawn_agent_process
-> create_subprocess_exec, which on Windows can't find 'npx' because it's the
'npx.cmd' shim (PATHEXT isn't applied to a bare name) -> FileNotFoundError. Resolve
command[0] via shutil.which when handing the command to the runtime, so the spawned
executable is the full path; an unresolvable name is left as-is to fail loudly at
spawn. self._command stays the caller's input.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(runtime): support callable-object custom tool handlers [INT-947]
Two bugs surfaced for handlers that are callable objects (not plain functions):
- invoke: asyncio.iscoroutinefunction is False for an instance whose __call__ is
async, so the handler's coroutine was returned unawaited (leaking through every
adapter using this helper). Call the handler, then await if the result is
awaitable — covers sync/async functions and async __call__ objects alike.
- accepts-input check: @lru_cache required the handler to be hashable, so a callable
defining __eq__ without __hash__ (a valid handler) raised 'unhashable type'. Drop
the cache (an inspect.signature per tool call is negligible).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): named flakiness taxonomy + collection guard [INT-947]
Replace ad-hoc @pytest.mark.flaky markers with two named decorators so every
baseline test's rerun policy and reason are explicit:
- flaky_model(reason): a capable model occasionally misses a recall/tool/judge it
usually passes -> AssertionError IS retried.
- flaky_infra(reason): a live-turn timeout/cold start is transient, but an
AssertionError is a real bug -> rerun_except=[AssertionError] (fails loud).
Both stamp a flaky_reason marker (kind + reason). A collection guard
(assert_flaky_is_classified) rejects a raw @pytest.mark.flaky, so an unclassified
rerun policy can't slip in. Migrated all 32 existing sites faithfully.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add GitHub Copilot SDK adapter [INT-921] (#410)
Squash the Copilot SDK adapter stack onto dev. Includes the adapter and converter, per-turn usage capture, core-lane E2E wiring, examples, ask_user handling, lifecycle/session hardening, single-instance runtime guard, and conformance coverage.
* test(e2e): turn-level retry for flaky model turns (tenacity) [INT-947]
Add model_turn_retrying() — a tenacity AsyncRetrying preconfigured as the
turn-level counterpart to flaky_model: retry a turn that raises AssertionError (a
model bad moment — a missed recall/tool/answer it usually gets) up to N attempts,
reraising the final; a non-assertion error (a stuck-turn TimeoutError) is NOT
retried. Used via tenacity's native idiom:
async for attempt in model_turn_retrying():
with attempt:
mid = await user_ops.send_message(...)
replies = await capture.wait_for_reply(mid, agent.id, since=mark)
replies.assert_contains_any([marker])
This re-drives only the flaky turn, far cheaper than flaky_model (which re-runs
the whole test). Migrated test_recalls_after_rejoin to it (flaky_model -> the
per-turn retry + flaky_infra for transient timeouts); live-verified on anthropic.
tenacity is now an explicit test dep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(e2e): wire the Letta lane live [INT-944] (#413)
Squash the Letta lane stack onto dev. Includes the self-hosted Letta MCP bridge, Letta CI lane setup, pydantic-settings cleanup, Docker/example fixes, adapter lifecycle hardening, and baseline conformance updates.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: GitHub Copilot CLI over ACP + TCP transport [INT-945] * Harden copilot injected history e2e * Support single instance locks on Windows * fix(lint): suppress pyrefly missing-attribute for Windows-only msvcrt pyrefly runs on Linux CI where msvcrt's locking/LK_NBLCK/LK_UNLCK attributes are absent, failing the lint gate. Suppress inline, matching the existing pyrefly-ignore convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: baseline scorecard — N/A reasons on exclude + CI artifact … (#420) * feat(e2e): baseline scorecard — N/A reasons on exclude + CI artifact [INT-820] Excluded adapters vanished from baseline results with no row and the reason buried in a comment. Make the full adapter×test grid observable: - exclude carries a reason: @per_adapter(exclude=...) takes ExcludedAdapter( adapter, reason) records; a non-empty reason is required at decoration time (ExcludedAdapter.__post_init__), mirroring AdapterSpec.e2e_pending. - the PerAdapter marker carries the exclusions so collection can read them back; specs()/adapter_params() stay id-based (clean layering). - scorecard.py: pure functions (na_rows / outcome_row / merge / to_markdown) build the grid from the run reports (exact nodeid keys, no junit scraping) plus the marker exclusions; a settings-driven conftest hook writes per-run JSON, and a merge CLI folds the per-lane files into one artifact. - CI: each e2e lane emits artifacts/scorecard-<lane>-<os>.json; a final scorecard job merges them into artifacts/scorecard.json (+ markdown). It reports the matrix, it does not gate on it. - migrate all 13 exclude call sites to ExcludedAdapter with reasons. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): scorecard counts only registered-adapter cells [INT-820] outcome_row treated any parametrized nodeid as a matrix cell, so non-adapter parametrizations (e.g. test_send_event[thought]) leaked "error"/"task"/ "thought" into the grid as phantom adapters. Filter cell params to the registered Adapter ids; add a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(e2e): scorecard collector as a registered plugin, no globals [INT-820] The conftest routed session-wide hooks (pytest_runtest_logreport / pytest_sessionfinish) through module globals because logreport gets no config/session. Make ScorecardCollector a plugin object that owns its state and output path; the conftest registers one only when a path is set. Removes the globals and the `global` statement — the idiomatic pytest pattern for stateful hook collection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): exclude letta from partial-reboot scenario [INT-985] Two Letta agents in one org can't keep separate identities: Letta stores MCP tools by name at org scope and re-points the shared band_send_message row to the most-recently-registered adapter's server, so the rebooted agent's reply routes through the peer's MCP server and posts under the peer's identity. Confirmed with a live probe. The cascade-delete fix (no deregister on release) removed the 404s that previously masked this, exposing the identity cross-wire. Excluded with a plain-language reason until per-instance tool-name isolation lands (INT-985). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: add ci-required aggregate gate job [INT-975] Branch protection can't reliably require the unit/conformance CI today: the `test`/`test-crewai` jobs are matrices, so GitHub emits per-cell contexts (`test (ubuntu-latest, 3.11)`, …) and the bare `test (3.11)`/`test (3.12)` names in the ruleset never report — they sit "Expected" forever. dev requires no test context at all. Add one aggregate `ci-required` job that `needs` every gating job and fails unless all resolved to success. A matrix dependency is green only when all its cells passed, so a single required `ci-required` context covers the whole OS/Python matrix and survives future matrix changes without another ruleset edit. `if: always()` so a failed dependency still runs the gate (a skipped job reports nothing and would leave the required check stuck). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add isolated baseline conformance harness * test: share baseline adapter identity * docs: clarify shared baseline adapter registry * test: simplify baseline harness coverage
* fix: make Copilot Docker examples runnable * fix: recover ACP sessions after agent restart * Apply suggestion from @AlexanderZ-Band Co-authored-by: AlexanderZ-Band <alexander.zaikman@band.ai> * Apply suggestion from @AlexanderZ-Band Co-authored-by: AlexanderZ-Band <alexander.zaikman@band.ai> --------- Co-authored-by: amit-gazal-thenvoi <amit.gazal@band.ai>
Manual, resumable smoke proving a headless band-sdk agent inside a real Docker Sandbox receives a WebSocket message, replies over REST, and reconnects after an operator-driven Wi-Fi cycle. The host-side driver (probe.py) reuses tests/e2e/baseline's pytest-free toolkit (ResourceManager, UserOps, reply_capture) for dynamic agent/room provisioning instead of a static pre-provisioned credential. Includes a SKILL.md workflow for AI-agent orchestration (preflight/record-phase/render-report) alongside the plain operator path (setup.sh/run.sh/probe.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
203
to
219
Comment on lines
+80
to
+155
| name: resolve lanes | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| lanes: ${{ steps.gen.outputs.lanes }} | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v6 | ||
|
|
||
| - name: Set up E2E environment | ||
| uses: ./.github/actions/setup-e2e | ||
|
|
||
| # The registry import is framework-free (band.adapters is lazy) and pytest-free, | ||
| # so the default extra is enough to resolve the lane partition. | ||
| - name: Install dependencies | ||
| run: uv sync --extra dev | ||
|
|
||
| - name: Emit lane matrix from the registry | ||
| id: gen | ||
| env: | ||
| # The dispatch `lane` input; defaults to 'all' (the full matrix). | ||
| SELECTED_LANE: ${{ github.event.inputs.lane || 'all' }} | ||
| # The dispatch `os` input; defaults to 'all' (Linux + Windows). | ||
| SELECTED_OS: ${{ github.event.inputs.os || 'all' }} | ||
| run: | | ||
| uv run python - <<'PY' >> "$GITHUB_OUTPUT" | ||
| import json | ||
| import os | ||
| from tests.e2e.baseline.toolkit.adapters import ( | ||
| assert_registry_covers_discovered, | ||
| ) | ||
| from tests.e2e.baseline.toolkit.ci_lanes import ( | ||
| assert_every_adapter_has_a_ci_home, | ||
| assert_workflow_lane_gates_known, | ||
| ci_lanes, | ||
| ) | ||
| # Fail before any test runs if the registry or lane partition has drifted. | ||
| assert_registry_covers_discovered() | ||
| assert_every_adapter_has_a_ci_home() | ||
| # Fail if a `matrix.lane ==` gate below names a lane the registry no longer | ||
| # emits (its step would silently never run). | ||
| assert_workflow_lane_gates_known() | ||
| lanes = list(ci_lanes()) | ||
| # The registry stays authoritative: a chosen lane must be one it emits, so | ||
| # a stale dropdown option fails loudly here instead of running nothing. | ||
| selected = os.environ.get("SELECTED_LANE") or "all" | ||
| known = {lane.id for lane in lanes} | ||
| if selected != "all" and selected not in known: | ||
| raise SystemExit( | ||
| f"Requested lane {selected!r} is not one the registry emits " | ||
| f"(known: {sorted(known)}). Pick 'all' or a known lane." | ||
| ) | ||
| # OS is a workflow concern (runner image), not a registry fact: every lane | ||
| # runs on each selected OS. `runner` is the runs-on image; `os` is the short | ||
| # id surfaced in the job name. | ||
| runners = {"ubuntu": "ubuntu-latest", "windows": "windows-latest"} | ||
| selected_os = os.environ.get("SELECTED_OS") or "all" | ||
| if selected_os != "all" and selected_os not in runners: | ||
| raise SystemExit( | ||
| f"Requested os {selected_os!r} is not known " | ||
| f"(known: {sorted(runners)}). Pick 'all', 'ubuntu', or 'windows'." | ||
| ) | ||
| oses = list(runners) if selected_os == "all" else [selected_os] | ||
| # A lane that can only run on Linux (its backend has no Windows story — | ||
| # ``LINUX_ONLY_LANES``, surfaced as ``lane.linux_only``) contributes no | ||
| # windows cell, so a full-matrix dispatch never emits a cell that can't run. | ||
| include = [ | ||
| {"lane": lane.id, "extra": lane.extra, "os": osid, "runner": runners[osid]} | ||
| for lane in lanes | ||
| if selected == "all" or lane.id == selected | ||
| for osid in oses | ||
| if not (osid == "windows" and lane.linux_only) | ||
| ] | ||
| print("lanes=" + json.dumps({"include": include})) | ||
| PY | ||
|
|
||
| e2e: |
Comment on lines
+156
to
+291
| name: e2e (${{ matrix.lane }} / ${{ matrix.os }}) | ||
| needs: lanes | ||
| runs-on: ${{ matrix.runner }} | ||
| # Per-(lane, OS) concurrency: a new dispatch cancels only the matching | ||
| # lane+OS still running, never the whole matrix — so one lane can be | ||
| # re-run in isolation. (See the note above the jobs block.) | ||
| concurrency: | ||
| group: e2e-baseline-${{ matrix.lane }}-${{ matrix.os }} | ||
| cancel-in-progress: true | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: ${{ fromJSON(needs.lanes.outputs.lanes) }} | ||
| # Use bash on every OS so the setup scripts and uv commands run identically | ||
| # (Git Bash ships on the windows-latest runner). | ||
| defaults: | ||
| run: | ||
| shell: bash | ||
| env: | ||
| E2E_TESTS_ENABLED: "true" | ||
| # Force UTF-8 for stdout/stderr. On the windows-latest runner Python defaults | ||
| # to the cp1252 console codec, so a framework that logs emoji (e.g. CrewAI's | ||
| # event bus: 🤖/🔧/✅) raises UnicodeEncodeError mid-turn and fails the test. | ||
| # No-op on Linux (already UTF-8). | ||
| PYTHONUTF8: "1" | ||
| PYTHONIOENCODING: "utf-8" | ||
| # Turn budget (E2E_TIMEOUT) is not set here: it defaults to 120s in the e2e | ||
| # settings, and slow frameworks add headroom per-test via | ||
| # @pytest.mark.timeout(extra=...). See tests/e2e/baseline/conftest.py. | ||
| # The registry scopes which adapters run in this lane. | ||
| BAND_E2E_LANE: ${{ matrix.lane }} | ||
| # The dev-crewai venv lacks the other frameworks; tolerate missing framework | ||
| # configs there instead of failing fast (mirrors ci.yml's crewai job). | ||
| BAND_ALLOW_MISSING_FRAMEWORKS: ${{ matrix.extra == 'dev-crewai' && '1' || '0' }} | ||
| # Secrets carry an E2E_ prefix in GitHub (so they're clearly scoped to this | ||
| # workflow), but the code reads the unprefixed names — the mapping happens | ||
| # here: the env var keeps its code-expected name, the secret is prefixed. | ||
| # Band platform (the live target + the driver/observer user key). | ||
| BAND_REST_URL: ${{ secrets.E2E_BAND_REST_URL }} | ||
| BAND_WS_URL: ${{ secrets.E2E_BAND_WS_URL }} | ||
| BAND_API_KEY_USER: ${{ secrets.E2E_BAND_API_KEY_USER }} | ||
| # Optional second user key, for baseline smokes exercising two-user interaction. | ||
| BAND_API_KEY_USER_2: ${{ secrets.E2E_BAND_API_KEY_USER_2 }} | ||
| # Model providers (a lane simply ignores a key it has no adapter for). | ||
| ANTHROPIC_API_KEY: ${{ secrets.E2E_ANTHROPIC_API_KEY }} | ||
| OPENAI_API_KEY: ${{ secrets.E2E_OPENAI_API_KEY }} | ||
| GOOGLE_API_KEY: ${{ secrets.E2E_GOOGLE_API_KEY }} | ||
| # A PAT from a GitHub account with Copilot entitlement — NOT the ambient | ||
| # secrets.GITHUB_TOKEN (a repo-scoped installation token with no such | ||
| # entitlement). The copilot_sdk adapter (in the `core` lane) needs it; | ||
| # exposed to every lane's job env so `core` picks it up. | ||
| GITHUB_TOKEN: ${{ secrets.E2E_GITHUB_TOKEN }} | ||
| # Per-lane scorecard: each lane writes its own adapter×test slice (pass/fail for | ||
| # its cells, skip for out-of-lane cells, N/A + reason for exclusions). The final | ||
| # `scorecard` job merges them. See tests/e2e/baseline/scorecard.py. | ||
| BAND_E2E_SCORECARD_JSON: artifacts/scorecard-${{ matrix.lane }}-${{ matrix.os }}.json | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v6 | ||
|
|
||
| - name: Set up E2E environment | ||
| uses: ./.github/actions/setup-e2e | ||
|
|
||
| # --- Backend setup: each step is gated to the lane whose server/CLI it stands | ||
| # up (the `backends` lane co-runs codex + opencode; letta is its own lane). --- | ||
|
|
||
| - name: Set up Node (codex, opencode) | ||
| if: matrix.lane == 'backends' | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '20' | ||
|
|
||
| # Each setup script installs its backend and exports the env it discovered | ||
| # (CODEX_CWD / OPENCODE_BASE_URL / LETTA_BASE_URL) via $GITHUB_ENV. See | ||
| # .github/scripts/ for the inline detail. | ||
| - name: Install + authenticate the Codex CLI (codex) | ||
| if: matrix.lane == 'backends' | ||
| # Invoke via `bash` explicitly: on the windows-latest runner a directly | ||
| # executed .sh depends on the exec bit surviving checkout and Git Bash | ||
| # resolving the shebang; `bash <script>` sidesteps both. No-op on Linux. | ||
| run: bash .github/scripts/setup-codex.sh | ||
|
|
||
| # copilot_acp drives `copilot --acp`; auth is the job-env GITHUB_TOKEN (no login). | ||
| - name: Install the GitHub Copilot CLI (copilot_acp) | ||
| if: matrix.lane == 'backends' | ||
| run: .github/scripts/setup-copilot.sh | ||
|
|
||
| - name: Install + start the OpenCode server (opencode) | ||
| if: matrix.lane == 'backends' | ||
| env: | ||
| # The OpenCode Zen provider key the server reads via {env:...} substitution. | ||
| OPENCODE_ZEN_API_KEY: ${{ secrets.E2E_OPENCODE_ZEN_API_KEY }} | ||
| run: bash .github/scripts/setup-opencode.sh | ||
|
|
||
| # The Letta server runs in docker with a host-gateway alias so it can call | ||
| # back into the adapter's self-hosted MCP server (LocalMCPServer inside the | ||
| # pytest process) via host.docker.internal. OPENAI_API_KEY (mapped at job | ||
| # level) doubles as the Letta server's own LLM provider key. | ||
| - name: Start the Letta server (letta) | ||
| if: matrix.lane == 'letta' | ||
| run: .github/scripts/setup-letta.sh | ||
|
|
||
| # ----------------------------------------------------------------------------- | ||
|
|
||
| - name: Install dependencies (${{ matrix.extra }}) | ||
| run: uv sync --extra ${{ matrix.extra }} | ||
|
|
||
| # Flaky reply-latency tests opt into reruns per-test via @pytest.mark.flaky; | ||
| # the run itself stays plain so only those tests are retried, not the lane. | ||
| - name: Run baseline E2E (lane ${{ matrix.lane }}) | ||
| run: uv run pytest tests/e2e/baseline/ -v -s --no-cov | ||
|
|
||
| # The backends lane also exercises the editor-facing ACP path (codex-acp). | ||
| # It uses the same ~/.codex login and a local MCP server — no Band platform. | ||
| - name: Run codex-acp e2e (codex) | ||
| if: matrix.lane == 'backends' | ||
| run: uv run pytest tests/integrations/acp/test_e2e_codex_acp.py -v --no-cov | ||
|
|
||
| - name: Stop the Letta server (letta) | ||
| if: always() && matrix.lane == 'letta' | ||
| run: docker rm -f letta-server || true | ||
|
|
||
| # Emitted even when tests fail (session end still runs), so a red lane still | ||
| # contributes its pass/fail rows to the merged scorecard. | ||
| - name: Upload lane scorecard | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: scorecard-${{ matrix.lane }}-${{ matrix.os }} | ||
| path: artifacts/scorecard-*.json | ||
| if-no-files-found: ignore | ||
|
|
||
| # Fold the per-lane scorecards into one adapter×test grid (pass / fail / skip / N-A). | ||
| # `if: always()` so a failed lane still yields a scorecard; this job reports the | ||
| # matrix, it does not gate on it (gating policy is tracked separately). | ||
| scorecard: |
Comment on lines
+292
to
+335
| name: scorecard (merge lanes) | ||
| needs: e2e | ||
| if: always() | ||
| runs-on: ubuntu-latest | ||
| defaults: | ||
| run: | ||
| shell: bash | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v6 | ||
|
|
||
| - name: Set up E2E environment | ||
| uses: ./.github/actions/setup-e2e | ||
|
|
||
| - name: Install dependencies | ||
| run: uv sync --extra dev | ||
|
|
||
| - name: Download lane scorecards | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| pattern: scorecard-* | ||
| path: artifacts | ||
| merge-multiple: true | ||
|
|
||
| - name: Merge into one scorecard | ||
| run: | | ||
| shopt -s nullglob | ||
| files=(artifacts/scorecard-*.json) | ||
| if [ ${#files[@]} -eq 0 ]; then | ||
| echo "no lane scorecards to merge (all lanes failed before session end)" | ||
| exit 0 | ||
| fi | ||
| uv run python -m tests.e2e.baseline.scorecard merge "${files[@]}" \ | ||
| --out artifacts/scorecard.json --markdown artifacts/scorecard.md | ||
|
|
||
| - name: Upload scorecard | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: scorecard | ||
| path: | | ||
| artifacts/scorecard.json | ||
| artifacts/scorecard.md | ||
| if-no-files-found: ignore |
| logger.info("Provisioned agent %s and room %s", agent.id, room_id) | ||
| # The only stdout output: run.sh captures these two lines verbatim. | ||
| sys.stdout.write(f"BAND_AGENT_ID={agent.id}\n") | ||
| sys.stdout.write(f"BAND_API_KEY={agent.api_key}\n") |
| socket.socket(socket.AF_INET, socket.SOCK_STREAM) as a, | ||
| socket.socket(socket.AF_INET, socket.SOCK_STREAM) as b, | ||
| ): | ||
| a.bind(("", 0)) |
| socket.socket(socket.AF_INET, socket.SOCK_STREAM) as b, | ||
| ): | ||
| a.bind(("", 0)) | ||
| b.bind(("", 0)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
band-sdkLangGraph agent runs inside a real Docker Sandbox (sbx exec) against staging, proving WebSocket receipt, a REST reply, and reconnect after an operator-driven Wi-Fi cycle.probe.py(the host-side driver) reusestests/e2e/baseline's pytest-free toolkit (ResourceManager,UserOps,reply_capture) for dynamic agent/room provisioning + reap, instead of a static pre-provisioned credential or hand-rolled REST/WS calls.SKILL.mdworkflow (withpreflight/record-phase/render-reportscripts) lets an AI agent (Codex, Claude Code) drive and resume the workflow across the manual Wi-Fi-toggle checkpoint; a plain operator can run the same scenario by hand viasetup.sh/run.sh/probe.py— both paths share onestate.jsonand produce the sameevidence.md.examples/, nottests/e2e/.Went through an xhigh-effort code review (10 finder angles) after the initial scaffold; every confirmed finding was fixed at the root rather than patched, including:
uv run probe.py(and the skill scripts) fail withModuleNotFoundErrorin a clean environment — these scripts need this repo's own dev venv (uv sync --extra dev), not an isolated script venv, since they importtests/e2e/baseline(dev-only source, not part of the published package).setup.sh(was comparing against the wrong directory).record-phase.pycrash on the very first run (state.json didn't exist yet); introducedstate.load_or_create()so either entry path (record-phase.py startedorprobe.py --label provision) can start a run.render-report.pycomputing PASS/FAIL over all historical probe attempts instead of the latest per step (a retried-and-fixed probe used to still report FAIL forever).state.jsonwrites atomic (temp file + replace) given the workflow is designed around being interrupted.sbxinvocations with real, verified syntax (sbx create shell PATH,sbx exec -e/--workdir,sbx policy allow network --sandbox,sbx list --json) checked against the installedsbxv0.34.0.band.runtime.types.normalize_handleinstead of a hand-rolled reimplementation, one bootstrap helper instead of triplicated client construction, etc.)Test plan
ruff check/ruff format --checkcleanpyrefly checkclean (repo-wide;examples/**is excluded from its scope by design)py_compileclean; both shell scripts passbash -nsbxv0.34.0 (sbx list --json,sbx create --help,sbx exec --help,sbx policy allow network --help)🤖 Generated with Claude Code