Skip to content

test: preserve multi-tool assistant turns in E2E showcase #367

Description

@gnanirahulnutakki

Context

CodeRabbit review of PR #365 found that the cloud-gated E2E showcase constructs invalid conversation history when Ollama returns more than one tool call in a single assistant turn. This defect is pre-existing on dev at 8af2340965a28894129a7300611f8dc210023776; PR #365 only formatted the block and did not change its AST or behavior.

Exact evidence

Pre-existing source:

CodeRabbit's exact finding:

Update the message construction around the multi-call loop to append the original assistant tool-call message only once, outside the loop. Keep one corresponding tool message per call inside the loop, using each call’s tool_name, while preserving the existing evaluation behavior.

When resp.message.tool_calls contains N calls, the current loop appends N synthetic assistant messages, each containing only one call. A valid transcript has one assistant turn containing the original ordered set of N tool calls, followed by N corresponding tool-result messages. The current malformed history can make a later model turn reject, reinterpret, or lose the relationship among parallel calls.

The affected class is guarded by @ollama_required, so ordinary CI and cloud-disabled local runs skip this path. That allowed the defect to remain invisible even though the showcase reports multi-turn coverage when credentials are present.

Expected sequence

For an assistant response containing calls [call_a, call_b], append exactly:

  1. one assistant message preserving the original assistant content and ordered tool_calls=[call_a, call_b];
  2. one tool message for call_a, using call_a.function.name and its corresponding result;
  3. one tool message for call_b, using call_b.function.name and its corresponding result.

Continue to send one /evaluate request per tool call and preserve the existing evaluation counter/decision behavior.

Acceptance criteria

  • The original assistant tool-call turn is appended once, outside the per-call result loop.
  • All tool calls from that turn remain attached to the single assistant message in original order.
  • Exactly one matching tool-result message is appended per call, in the same order, with the correct tool name and result.
  • /evaluate is still invoked exactly once per call with that call's name and parsed arguments.
  • Add a credential-free regression using a fake response with at least two tool calls; assert the exact assistant/tool message sequence presented to the next model turn.
  • The regression fails against dev at 8af2340965a28894129a7300611f8dc210023776.
  • The existing single-call behavior remains valid.
  • The cloud-disabled showcase retains its expected skips; an optional credentialed smoke may be recorded separately without making CI depend on an external model.
  • Ruff 0.13.0 and repository pre-commit hooks pass.

Related: #365, #361

Metadata

Metadata

Assignees

No one assigned

    Labels

    release-hygieneRelease process & versioning hygiene

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions