test: preserve multi-tool assistant turns

[gnanirahulnutakki]

Summary

• preserve the original assistant response once when a turn contains one or more tool calls
• append one ordered tool-result message per call using Ollama's current tool_name field
• keep one /evaluate request per call with its own parsed arguments and preserve the exact displayed evaluation count
• propagate rejected follow-up model requests instead of allowing the showcase to report a false success
• report simulated execution success only for explicit PERMIT; report explicit DENY as denied and missing or unusable evidence as unknown
• add credential-free single-call, two-call, decision-outcome, and rejected-follow-up regressions outside the 28-capability showcase accounting

Contract evidence

Ollama's official tool-calling documentation appends response.message once, processes its ordered tool_calls, then appends one role: tool message with tool_name per result:

https://docs.ollama.com/capabilities/tool-calling

Validation

• red proof on exact base 3a5015b00a78604f8db4fc8f65433e0e883668a2: the two-call regression failed because the old loop produced two synthetic assistant turns and legacy name fields
• PYTHONPATH=. uv run --project . pytest tests/test_e2e_showcase.py tests/test_e2e_showcase_transcript.py -q --tb=short: 29 passed, 10 skipped, with five pre-existing pytest warnings tracked by test: migrate E2E showcase class fixtures before pytest 10 #371
• Ruff 0.13 check: passed
• Ruff 0.13 format check: passed
• repository pre-commit hooks: passed, including private-key and gitleaks checks
• ./scripts/check-local.sh --quick: passed
• git diff --check: passed
• independent review: clear before the CodeRabbit follow-up
• hosted CodeRabbit found one valid fail-closed result issue; signed/DCO commit 18954b63db3b239552cccbf7fa0b33766c218fcc addresses it and re-review is pending

The ten skips are the unchanged credential-disabled cloud showcase cases. No external model, Docker, or cluster access was required for this deterministic transcript fix.

README was reviewed; no update is needed because public behavior and the 28-capability inventory are unchanged.

Scope

Adjacent legacy transcript builders and the optional workflow dependency gap are intentionally excluded from this focused issue and tracked by #374 and #375.

AI assistance disclosure

This change was implemented with Codex assistance. The author reviewed the diff, contract source, tests, and validation evidence.

Closes #367

Summary by CodeRabbit

• Bug Fixes

  • Improved multi-turn conversation handling for tool-call transcripts, including correct ordering and preservation of full assistant messages.
  • Tool-call evaluation payloads are now constructed consistently, and missing/invalid evaluation evidence yields an unknown outcome instead of a misleading result.
  • Errors from follow-up model requests now surface correctly instead of being silently ignored.

• Tests

  • Added end-to-end regression tests covering multi-tool and tool-call transcript behavior, including evaluation calls and failure propagation.

[coderabbitai]

📝 Walkthrough

Walkthrough

The multi-turn Ollama showcase now preserves complete assistant tool-call messages, constructs corresponding tool results, and propagates follow-up chat errors. New credential-free tests validate transcript ordering, evaluation calls, reporting, and failure behavior.

Changes

Ollama transcript handling

Layer / File(s)	Summary
Preserve multi-turn transcript flow python/tests/test_e2e_showcase.py	Chat exceptions now propagate, assistant responses remain intact in conversation history, and tool-result messages are created per tool call using centralized evaluation results.
Add transcript regression coverage python/tests/test_e2e_showcase_transcript.py	New tests validate ordered multi-tool transcripts, per-call evaluation payloads, unknown evaluation results, reporting, and follow-up error propagation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Ollama
  participant Showcase
  participant Evaluate
  Ollama->>Showcase: Assistant message with tool calls
  Showcase->>Evaluate: Evaluate each tool call
  Evaluate-->>Showcase: Status and decision
  Showcase->>Ollama: Original assistant message and tool results
  Ollama-->>Showcase: Follow-up response or propagated error

Loading

Possibly related issues

• test: normalize Ollama tool-call transcripts across cloud harnesses #374 — Covers the same Ollama transcript normalization and error-propagation changes in python/tests/test_e2e_showcase.py.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes preserve one assistant tool-call turn, append one tool message per call, keep /evaluate per call, and add the required regressions.
Out of Scope Changes check	✅ Passed	No clear out-of-scope code changes are evident; the edits and new tests all support the multi-tool transcript fix.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: preserving assistant turns that contain tool calls.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch gnanirahulnutakki/issue-367-multi-tool-turn

---

_{Comment @coderabbitai help to get the list of available commands.}

[gnanirahulnutakki]

Follow-up decomposition from the exact-dev audit:

• test: normalize Ollama tool-call transcripts across cloud harnesses #374 owns the same malformed assistant/tool transcript patterns in the five adjacent cloud harnesses.
• ci: make the credentialed Ollama showcase fail closed #375 owns the credentialed showcase dependency and fail-closed skip gap.

Both are intentionally outside this focused #367 PR.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/tests/test_e2e_showcase.py`:
- Around line 879-886: Update the tool-result construction in the evaluation
response flow so it never reports success without valid evidence: emit “denied”
only when the evaluation decision is explicitly DENY, and emit “unknown” for
non-200 responses, unusable decisions, or other missing evidence. Preserve the
existing successful result only for an observed, permitted evaluation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 523593b3-b132-483f-9893-4fbf696b3b9a

📥 Commits

Reviewing files that changed from the base of the PR and between 3a5015b and d280fc6.

📒 Files selected for processing (2)

• python/tests/test_e2e_showcase.py
• python/tests/test_e2e_showcase_transcript.py