Skip to content

bedrock incremental stream path is a buffered-replay (no true token streaming) #356

Description

@zmaril

What

The bedrock provider's incremental/streaming path does not stream token-by-token — it produces the full response and replays it as a single buffered result. A caller streaming from a Bedrock model gets the whole message at once instead of incremental updates.

Why

The bedrock Rust driver (crates/pidgin-ai/src/api/bedrock/driver.rs) is buffered-only: it has stream / stream_simple but no stream_streaming / stream_streaming_simple (unlike the anthropic and mistral drivers, which gained true incremental entry points in #343). As a result, BedrockBackend::stream_incremental (crates/pidgin-ai/src/providers/bedrock_backend.rs) delegates to the buffered stream_simple and wraps it via AssistantEventReader::from_buffered.

This is a pre-existing structural limitation, not a regression: before #349 the default Provider::stream_incremental seam impl already did a buffered replay of stream. #349 only made that replay honor reasoning (by replaying stream_simple instead of stream). Reasoning/thinking IS lowered correctly on the bedrock incremental path; the gap is purely that it isn't truly incremental.

pi's bedrock (vendor/pi/packages/ai/src/api/bedrock-converse-stream.ts) uses the real Bedrock ConverseStream API and streams incrementally.

Fix

Port a real incremental entry to the bedrock driver — a stream_streaming (and stream_streaming_simple mirroring the anthropic/mistral shape from #343) backed by the Bedrock ConverseStream API — and route BedrockBackend::stream_incremental through it instead of the from_buffered replay. Then the bedrock incremental tests can assert true event-by-event streaming rather than a buffered replay.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions