Skip to content

fix: document run_stream() event contract and warn on thinking + streaming - #2

Merged
sjwan01 merged 1 commit into
mainfrom
fix/run-stream-event-contract
Aug 7, 2026
Merged

fix: document run_stream() event contract and warn on thinking + streaming#2
sjwan01 merged 1 commit into
mainfrom
fix/run-stream-event-contract

Conversation

@sjwan01

@sjwan01 sjwan01 commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Summary

The run_stream() event contract was undocumented and misleading, and the default thinking_enabled=True silently inherits a pydantic-ai streaming defect.

Three verified problems (real-API reproduction on deepseek-v4-flash, pydantic-ai 2.22.0 and 2.25.0):

  1. Upstream defect (not fixable here): thinking + streaming + tools nondeterministically drops post-tool-call text — the agent loop terminates after a tool returns (no second ModelResponse); run_end.output is not a reliable fallback. Non-streaming run() is always complete.
  2. Contract gap: run_stream() forwards only chunks yielded by extensions implementing on_agent_runner_event_stream; chain-mode events never reach the consumer. A bare consumer receives only run_end. The old docstring said "yields events one by one", which misled first-time users.
  3. Default value: thinking_enabled=True (default) inherits problem 1.

Changes (5 files, +164/−14)

  • run_stream() docstring — precise event contract: runtime events delivered to streaming extensions; chain-mode events never forwarded; bare consumer receives only run_end.
  • Extension.on_agent_runner_event_stream docstring — cross-references the contract; notes that without it no token/tool events reach callers.
  • AgentRunner.run_stream() warns once (first iteration) when thinking_enabled=True — the intent to stream is only known at call time, so the warning fires when the caller actually consumes the stream, not at construction. run() never warns.
  • README Known issues — records the pydantic-ai defect (verified mechanism + versions) and the non-streaming run() workaround; fixes the misleading Public API description; documents the streaming-extension hook.
  • Tests (TDD) — 4 new tests for the warning (fires once, dedup, run() silence, thinking_enabled=False silence) + 1 contract test (chain-only extension yields nothing to the consumer).

Verification

  • make check: ruff ✅ mypy strict ✅ 146 passed, 7 skipped (7 = Postgres, run in CI)
  • CI will run the full matrix (Python 3.11–3.14) including the Postgres tests.

Known limits

  • This is documentation + warning only — the upstream pydantic-ai defect (problem 1) is not fixed here (blocked upstream). The warning and README make the hazard visible before it is hit; run() or thinking_enabled=False are the workarounds.
  • A runtime path for bare consumers to observe token/tool events (run_with_events()) is a separate proposal, not part of this PR.

…aming

run_stream() yields almost nothing to a bare consumer (only run_end):
token and tool events are delivered to extensions implementing
on_agent_runner_event_stream, and only their chunks reach the caller.
The old docstring said "yields events one by one", misleading first-time
users into expecting token chunks.

- Document the precise run_stream() event contract in the docstring:
  runtime events go to streaming extensions; a bare consumer receives
  only run_end; chain-mode events are never forwarded.
- Cross-reference the contract from Extension.on_agent_runner_event_stream.
- AgentRunner warns once, on the first consumed run_stream() iteration,
  when thinking_enabled=True: pydantic-ai 2.22+ (verified on
  deepseek-v4-flash) drops post-tool-call text in thinking + streaming
  mode; run_end.output is not reliable. run() never warns — the intent
  to stream is only known when the caller actually consumes the stream
  (async generator body runs on first iteration).
- README: record the pydantic-ai streaming defect under Known issues with
  the non-streaming run() workaround; fix the misleading run_stream()
  description; document the streaming-extension hook in Extensions.

Tests: TestThinkingStreamingWarning covers warn-once dedup, zero warnings
for run() and thinking_enabled=False, and message content; a
chain-only-extension contract test pins that chain events never reach the
consumer.
@sjwan01 sjwan01 closed this Aug 7, 2026
@sjwan01 sjwan01 reopened this Aug 7, 2026
@sjwan01
sjwan01 merged commit 3958842 into main Aug 7, 2026
8 of 20 checks passed
@sjwan01
sjwan01 deleted the fix/run-stream-event-contract branch August 7, 2026 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant