feat(harbor): add Prime Agent adapter with continual-harness support - #49
feat(harbor): add Prime Agent adapter with continual-harness support#49Octane0411 wants to merge 3 commits into
Conversation
Prime Agent descends from badlogic/pi-mono and kept its CLI contract, so the invocation and usage accounting mirror Harbor's pi adapter. What is specific to Prime is the continual harness -- prompts, memories, skills and subagent specs it refines itself during a run -- so the adapter can pin a trial to a harness checkpoint and export whatever the run left behind, which is what makes a chain of harness generations measurable. Two upstream behaviours are handled explicitly because both fail silently: - Prime gates auto-refine on a session-local harness directory, so --no-session disables refinement structurally, before autoRefine is even consulted. A sessionless run still looks healthy -- reward, trajectory and exit status are all normal -- while never refining. Sessions are kept when refinement is expected; frozen and probe runs stay sessionless so they cannot persist. - The shipped trigger thresholds (turnInterval=25, 20min cooldown) never fire on a benchmark-length episode, so refine_turn_interval / refine_cooldown_ms make that an explicit experiment parameter rather than an assumption. Also supports a pre-baked runtime mounted read-only (runtime_prefix), for environments where Prime's postinstall cannot reach the release CDNs it needs. install() then degrades to a version check and the trial needs no network for the agent itself; pinning the bundle digest fixes Node, Prime Agent and the Python runtime together. The guide documents the merge requirement for multi-episode chains: refinement lands in the session-local harness, so an export holds both the injected parent and the episode's own store, and carrying a chain forward means merging them rather than picking one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 8 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds a Prime Agent Harbor adapter with installation, runtime configuration, state persistence, execution, usage accounting, tests, architecture registration, and user documentation. ChangesPrime Agent Harbor integration
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant HarborEnvironment
participant PrimeAgent
participant PrimeAgentCLI
participant AgentContext
HarborEnvironment->>PrimeAgent: Provide execution environment
PrimeAgent->>HarborEnvironment: Hydrate agent and harness state
PrimeAgent->>PrimeAgentCLI: Run instruction with session and credential settings
PrimeAgentCLI-->>PrimeAgent: Emit JSONL execution events
PrimeAgent->>AgentContext: Populate token and cost usage
PrimeAgent->>HarborEnvironment: Export updated agent directory
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/guides/prime-agent.md`:
- Around line 21-24: Update the Prime agent export flow around
_hydrate_agent_dir and export_agent_dir so the uploaded auth.json is excluded,
redacted, or deleted before agent_dir is exported. Preserve auth.json
availability during the trial while ensuring credentials cannot appear in
exported logs or artifacts when export_agent_dir is enabled.
In `@src/evolve/integrations/harbor/prime_agent.py`:
- Around line 217-220: Prevent auth.json from being persisted in exported run
artifacts: update the export flow around _export_agent_dir and
environment.download_dir to remove _agent_dir/auth.json after command completion
before downloading, or restrict the export to harness and session files. Add a
regression test confirming exported artifacts never contain auth.json.
- Around line 182-184: Update the model-name validation in the Prime Agent
initialization flow before assigning provider and model: split the value once,
then reject it when either segment is empty while preserving the existing
provider/model format error. Add parameterized tests covering "/gpt" and
"openai/" and verify both raise the expected ValueError.
- Around line 206-213: Update the command construction in the Prime Agent
invocation to enable shell pipefail before running the pipeline, ensuring
failures from prime-agent propagate through tee and produce a nonzero status.
Preserve the existing output redirection and tee behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5f81dcba-a803-4e41-aa6c-3e84b6767f2a
📒 Files selected for processing (6)
ARCHITECTURE.mddocs/guides/prime-agent.mdmkdocs.ymlsrc/evolve/integrations/harbor/prime_agent.pytests/test_prime_agent.pytests/test_recipe_composition.py
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a68d017e04
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if self._export_agent_dir: | ||
| # Captures harness_state.json plus anything the run persisted, so a | ||
| # checkpoint chain can be rebuilt on the host afterwards. | ||
| await environment.download_dir(self._agent_dir, self.logs_dir / EXPORT_DIRNAME) |
There was a problem hiding this comment.
When auth_json_path is used as documented and export_agent_dir retains its default value, _hydrate_agent_dir() uploads the credential file to <agent_dir>/auth.json, after which this call downloads that entire directory into the trial logs. The authentication tokens therefore become persisted Harbor artifacts and may be copied into experiment results; export only the harness/session state or remove auth.json before downloading the directory.
Useful? React with 👍 / 👎.
| f"--provider {shlex.quote(provider)} --model {shlex.quote(model)} " | ||
| f"{cli_flags + ' ' if cli_flags else ''}" | ||
| f"{shlex.quote(instruction)} " | ||
| f"2>&1 </dev/null | stdbuf -oL tee /logs/agent/{OUTPUT_FILENAME}" |
There was a problem hiding this comment.
When prime-agent exits nonzero because of invalid credentials, an API outage, or a runtime crash, this pipeline returns tee's successful status because it does not enable pipefail. Harbor can consequently treat a failed agent invocation as successful and continue to verification/export with incomplete output; enable pipeline failure propagation or explicitly preserve the agent process's exit status.
Useful? React with 👍 / 👎.
…dation Three findings from the automated reviews, all real: - **auth.json reached the exported artifacts.** _hydrate_agent_dir uploads the credential into the agent directory and the whole directory is then exported into the trial logs, where retention and result sharing carry it along. This was not hypothetical: the run that produced this adapter left 229 copies of a live OAuth token in its artifacts. The adapter now deletes auth.json from the agent directory before the export leaves the container. - **tee masked prime-agent failures.** tee closed the pipeline, so a crashed or unauthenticated agent still exited 0 and Harbor scored the trial as a completed run. Enabled pipefail. - **Empty provider or model was accepted.** "/gpt" and "openai/" passed the substring check and produced an invalid command; both segments are now validated. Tests cover each fix, including that the credential removal happens before the download rather than merely somewhere in the run. Two existing tests selected the run command by list position, which the new removal step invalidated; they now select by content, which is what they meant. ARCHITECTURE.md budget raised to match the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
tests/test_prime_agent.py (2)
257-270: 🩺 Stability & Availability | 🔵 Trivial | 🏗️ Heavy liftTest failure propagation at runtime, not only command construction.
This test verifies the placement of
set -o pipefailin the command string. It does not prove that a failingprime-agentcommand produces a failure whenteesucceeds. Execute the generated pipeline with a failing Prime Agent stub and assert that the command executor reports failure. Keep the current string assertion as a construction check.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_prime_agent.py` around lines 257 - 270, Extend test_agent_failure_is_not_masked_by_tee to execute the generated run_command using a failing Prime Agent stub and a successful tee, then assert the command executor reports failure. Preserve the existing set -o pipefail placement assertions and use the test’s established command-execution helpers.
246-247: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winReject blank-after-trimming provider and model components.
If whitespace-only components are invalid, add cases such as
" /gpt"and"openai/ ".src/evolve/integrations/harbor/prime_agent.pycurrently checks truthiness without trimming, so these values pass validation and reach the CLI. Trim or reject each component.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_prime_agent.py` around lines 246 - 247, Extend test_model_name_must_carry_both_provider_and_model with whitespace-padded cases such as " /gpt" and "openai/ ", and update the model-name validation in the relevant Prime agent logic to trim each provider and model component before checking for emptiness. Ensure whitespace-only components are rejected before reaching the CLI.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_prime_agent.py`:
- Around line 288-293: Update the test around agent.agent_commands and
environment.downloads to record both command execution and directory export in a
single ordered event stream. Assert that the auth.json removal event occurs
before the download_dir/export event, and verify the exported directory does not
contain auth.json; replace the current assertion that only checks removal is the
last agent command.
---
Nitpick comments:
In `@tests/test_prime_agent.py`:
- Around line 257-270: Extend test_agent_failure_is_not_masked_by_tee to execute
the generated run_command using a failing Prime Agent stub and a successful tee,
then assert the command executor reports failure. Preserve the existing set -o
pipefail placement assertions and use the test’s established command-execution
helpers.
- Around line 246-247: Extend test_model_name_must_carry_both_provider_and_model
with whitespace-padded cases such as " /gpt" and "openai/ ", and update the
model-name validation in the relevant Prime agent logic to trim each provider
and model component before checking for emptiness. Ensure whitespace-only
components are rejected before reaching the CLI.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c7ccfdf8-24a9-420b-9d72-71d09742f3a6
📒 Files selected for processing (4)
ARCHITECTURE.mddocs/guides/prime-agent.mdsrc/evolve/integrations/harbor/prime_agent.pytests/test_prime_agent.py
🚧 Files skipped from review as they are similar to previous changes (3)
- docs/guides/prime-agent.md
- ARCHITECTURE.md
- src/evolve/integrations/harbor/prime_agent.py
…happens The previous assertion checked that `rm -f auth.json` was the last recorded agent command, but downloads were tracked in a separate list, so the test still passed if the export ran first — it asserted the wrong thing about the thing that matters most. Commands and transfers now share one ordered event stream and the test asserts the removal index precedes the download index. Verified by deleting the removal step and confirming the test fails. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 participant
Adds
evolve.integrations.harbor.prime_agent:PrimeAgentso EvolveX can evaluatePrime Agent — and, more
interestingly, measure its continual harness across a chain of generations.
Why it is small
Prime Agent descends from
badlogic/pi-monoand kept that CLI contract, so theinvocation and the
message_endusage accounting mirror Harbor's existingpiadapter. The Prime-specific part is the harness: the adapter can pin a trial to
a checkpoint and export whatever the run persisted, which is what makes a
generation chain observable at all.
Two upstream behaviours that fail silently
Both were found the hard way while running Terminal-Bench 2.0, and both are
handled explicitly rather than left to the caller:
--no-sessiondisables refinement structurally. Prime gates auto-refineon a session-local harness directory, so a sessionless run never refines no
matter how
autoRefineis configured. Nothing about it looks wrong — reward,trajectory and exit status are all normal — it simply never learns. Sessions
are kept when refinement is expected; frozen and probe runs stay sessionless
so they cannot persist anything.
turnInterval=25and a 20 minute cooldown, while these episodes finish in ahandful of turns.
refine_turn_interval/refine_cooldown_msmake that anexplicit experiment parameter, because "measure shipped behaviour" and
"measure refinement" are different questions with different answers.
Restricted networks
runtime_prefixaccepts a pre-baked runtime mounted read-only, for environmentswhere Prime's postinstall cannot reach the release CDNs it provisions Python
from.
install()then degrades to a version check and the trial needs nonetwork for the agent itself. Pinning the bundle digest also fixes Node, Prime
Agent and the Python runtime together, which is a stronger guarantee than a
version string when arms of an experiment run on different days.
Contents
src/evolve/integrations/harbor/prime_agent.pytests/test_prime_agent.pytest_codex_candidate.pydocs/guides/prime-agent.mdARCHITECTURE.md,mkdocs.yml,tests/test_recipe_composition.pyThe guide also documents a consequence worth knowing before interpreting a
multi-episode result: refinement lands in the session-local harness, so an
export contains both the injected parent copy and the episode's own store.
Carrying a chain forward means merging them — preferring the global file
replays the parent unchanged, and preferring the session-local file discards
everything learned earlier. Either mistake yields a checkpoint chain that reads
as a healthy learning curve while nothing accumulates.
Checks
ruff check·ruff format --check·ty check·uv lock --check·pytest -q→ 964 passed, 3 skipped.Exercised end-to-end against Terminal-Bench 2.0 (89 tasks) on Docker, including
a 16-generation harness chain and a held-out sealed comparison.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Bug Fixes