[codex] add agent harness v0 with replay eval - #11
Open
RCF-117 wants to merge 4 commits into
Open
Conversation
RCF-117
marked this pull request as ready for review
July 7, 2026 07:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
当前项目已有 LLM workflow、POI/交通/行程校验等后端能力,但这些能力主要散落在 workflow/service 层,还没有统一的 agent tool layer。为了后续支持 tool calling、memory、eval/replay 和更稳定的 agent 编排,本次新增一个 backend-only 的 Agent Harness v0。
这个改动不替换现有
/plan/stream行为,也不改前端协议,主要是为后续 agent 化提供可测试、可观测、可回放的基础设施。本次新增
1. Agent 通用协议
新增
backend/app/agent/schemas.py,定义统一的 harness 数据结构:ToolCallToolResultToolWarningToolContextToolSpecToolTraceEntryHarnessRequestHarnessRunResult所有工具统一通过
ToolCall(name, args)调用,并返回结构化ToolResult,包含ok / data / warnings / degraded / error。2. Tool Registry
新增
backend/app/agent/registry.py,提供工具注册与统一调用入口。目前注册的工具包括:
parse_user_intentestimate_visit_durationcompute_transitroute_sort_dayvalidate_itinerary未知工具和工具异常都会被包装成结构化错误,方便 agent runner 记录和降级处理。
3. Harness Tool Adapters
新增
backend/app/tools/harness_tools.py,把现有后端能力包装成统一 tool:parse_user_intent_tool:封装现有 intent parserestimate_visit_duration_tool:封装默认停留时间规则compute_transit_tool:封装现有交通估算/降级逻辑route_sort_day_tool:接入已有顺路排序工具validate_itinerary_tool:校验缺坐标、重复 POI、时间顺序、日程结构完整性等其中
validate_itinerary会优先使用 stop 的显式 slot 判断早餐/午餐/晚餐/景点/酒店,避免仅依赖category=eat导致三餐误判。4. Deterministic Runner v0
新增
backend/app/agent/runner.py,提供固定流程 harness:它不会调用 LLM,适合离线测试、回归测试和调试 tool trace。
5. LLM-planned Harness Runner
新增:
backend/app/agent/planner.pybackend/app/agent/live_runner.pybackend/app/agent/recorder.pyLLMToolPlanner会调用现有llm.stream_chat(...)生成 tool plan,但执行阶段仍由本地 runner 注入可信参数,避免模型幻觉参数或传入空 stops。同时 v0 会强制补齐最小工具链:
也就是说,LLM 可以参与规划,但不会完全控制执行。
6. Replay Recording & Eval
新增
evals/travel_plan_eval.py和 replay fixtures:evals/fixtures/travel_plan_replay.jsonlevals/fixtures/agent_harness_live.jsonl每次 live harness 可以记录:
评估脚本支持旧 workflow
days格式,也支持新的 agent harness record 格式。当前指标包括:
测试
新增测试覆盖:
默认测试不会调用 API。只有显式设置:
才会运行 live LLM 测试并记录 replay 数据。
本地验证结果:
设计边界
本次没有:
/plan/stream目标是先建立轻量、可测试、可回放的 agent harness v0,为后续 tool calling、memory、POI grounding 和 eval 体系打基础。