From e3a6ac3536abe96c23d83d0bbd4ad81b0ac2f362 Mon Sep 17 00:00:00 2001 From: Qiuyang Mang Date: Wed, 29 Apr 2026 20:15:01 -0700 Subject: [PATCH 1/7] 1 --- _posts/2026-04-28-harbor.md | 280 ++++++++++++++++++ .../2026-04-28-harbor/comparison-chart.html | 109 +++++++ 2 files changed, 389 insertions(+) create mode 100644 _posts/2026-04-28-harbor.md create mode 100644 assets/html/2026-04-28-harbor/comparison-chart.html diff --git a/_posts/2026-04-28-harbor.md b/_posts/2026-04-28-harbor.md new file mode 100644 index 0000000..f91b0f6 --- /dev/null +++ b/_posts/2026-04-28-harbor.md @@ -0,0 +1,280 @@ +--- +layout: distill +title: "Your Next Long-Context Recipe: Open-Ended Problems" +description: "Coding data lifts reasoning. Agentic coding is dominant. We add the next source — 172 open-ended problems with continuous scoring, no ground truth, now one harbor run away. Kimi K2.6 and Claude Opus 4-7 go head-to-head: 50+ turns, 56 tool calls, 191K tokens per problem." + +date: 2026-04-29 +date_display: "Apr 29, 2026" +htmlwidgets: true + +authors: + - name: Kaiyuan Liu + url: "mailto:lky04@cs.washington.edu" + affiliations: + name: University of Washington + - name: Zhifei Li + url: "mailto:andylizf@outlook.com" + affiliations: + name: Princeton University + - name: Wenhao Chai + url: "https://wenhaochai.com/" + affiliations: + name: Princeton University + - name: Qiuyang Mang + url: "https://joyemang33.github.io/" + affiliations: + name: University of California, Berkeley + - name: Lin Shi + affiliations: + name: Harbor team + - name: Zhixuan Zhu + affiliations: + name: Harbor team + - name: Crystal Zhou + affiliations: + name: Harbor team + - name: Frontier-CS team + url: "https://frontier-cs.org" + - name: Harbor team + url: "https://harborframework.com" + +toc: + - name: A New Source of Long-Context Data + - name: "What Makes Open-Ended Problems Different" + - name: "Example: Polyomino Packing" + - name: "172 Problems, One Command" + - name: "Empirical Check: Kimi K2.6 vs Claude Opus 4-7" + - name: Try It + +_styles: > + d-article img { + max-width: 70%; + height: auto; + display: block; + margin: 1.5rem auto; + border-radius: 6px; + } + d-article img.full { + max-width: 100%; + } + d-article pre { + font-size: 0.85em; + } + d-article .stat-cards { + display: grid; + grid-template-columns: repeat(3, 1fr); + gap: 16px; + margin: 1.5rem 0; + } + d-article .stat-card { + background: #f9fafb; + border-radius: 10px; + padding: 20px; + text-align: center; + border: 1px solid #e5e7eb; + } + d-article .stat-card .number { + font-size: 2em; + font-weight: 800; + line-height: 1.1; + } + d-article .stat-card .label { + font-size: 0.85em; + color: #6b7280; + margin-top: 6px; + } + d-article .opus-color { color: #6366f1; } + d-article .kimi-color { color: #10b981; } +--- + +
+ Frontier CS +
+ +
+Coding has long been a strong long-context data source. By 2023, mixing coding data into pretraining was already known to lift reasoning. Today, agentic coding is the dominant choice. We're adding another source: 172 open-ended problems with continuous scoring and no ground truth — now one harbor run away. Two frontier agents sustain sessions of up to 450+ turns and up to 400+ tool calls per problem, producing some of the longest agentic trajectories in any public benchmark. +
+ +## A New Source of Long-Context Data + +Coding has long been a strong long-context data source. By 2023, mixing coding data into pretraining was already known to lift reasoning. Today, agentic coding is the dominant choice: downstream-aligned, information-dense, globally consistent, dependency-rich. + +We're adding another coding source: **open-ended problems**. Each ships a domain-specific grader that returns a continuous score, not pass/fail. + +What makes them a good long-context source? Three properties: + +1. **No ceiling.** Every problem has room to improve. An agent that scores 0.6 today can try for 0.7 tomorrow. The trajectory never converges to a single "correct" answer. +2. **Long-horizon reasoning.** Agents sustain sessions of up to 450+ turns and 400+ tool calls, generating up to 530K output tokens on a single problem. This is not a one-shot generation — it is sustained, iterative problem-solving over hundreds of steps. +3. **Dense feedback signal.** The continuous score creates a natural reward landscape. Every intermediate solution attempt gets a meaningful gradient, not just "wrong." + +## What Makes Open-Ended Problems Different + +Here's where open-ended problems split from SWE. SWE tasks are well-defined: a spec, a reference behavior, a verifiable right answer. Open-ended problems lack that layer. The reason is not that ground truth is hard to label at scale; it is that mathematically, **none exists**. + +Consider building the largest red-blue edge-colored complete graph with no monochromatic 5-clique. This is the lower-bound problem for the Ramsey number $R(5,5)$: its true value lies somewhere in $[43, 48]$, and nobody knows the exact number. Any construction that pushes the known lower bound up by even one is a genuine contribution. + +How hard are these problems for humans? In [our Calico experiment]({{ '/blog/calico' | relative_url }}), we placed one open-ended problem in UC Berkeley's official programming contest with **2,000+ participants**. Out of 285 submissions, **only one surpassed the strongest AI agent**. + +## Example: Polyomino Packing + +One concrete example from the [Frontier-CS benchmark](https://frontier-cs.org): given a set of polyomino pieces, pack as many as possible into a fixed grid. The grader scores each submission by packing density — a continuous signal, not pass/fail. + +![Frontier-CS Teaser](https://raw.githubusercontent.com/FrontierCS/Frontier-CS/main/assets/teaser.png) + +The search space is combinatorially vast: piece ordering, rotation, placement strategy, and local compaction all interact. There is no known polynomial-time optimal solution. An agent must iteratively generate code, compile, test, analyze results, and refine its approach across many rounds. This single problem can produce trajectories spanning hundreds of agent turns. + +The animation below shows an adaptive evolution agent progressively discovering denser packing strategies — better ordering, symmetry handling, and local compaction — over successive iterations: + +
+
+ Online Evolution on Polyomino Packing +
Adaptive evolution progressively discovers denser packing strategies on a polyomino task.
+
+
+ +
+This task is far from saturated. A human expert achieves a packing score of 92, while the best current AI agent reaches only the low 80s — leaving substantial room for improvement. Frontier-CS tasks are designed with deep search spaces and high expert-level ceilings. +
+ +## 172 Problems, One Command + +All 172 open-ended problems ship as a [Harbor](https://harborframework.com) adapter, joining the ~70 benchmarks already on Harbor (SWE-Bench, Terminal-Bench, USACO, GPQA, BFCL, and more). Harbor gives you a standard task format, a standard agent interface, a trial orchestrator, and a provider layer — local Docker, Daytona, Modal, E2B, or GKE. + +```bash +# Run with any standard agent CLI +uv run harbor run -d frontier-cs-algorithm \ + -a claude-code -m "anthropic/claude-opus-4-6" +``` + +Plug in Claude Code, OpenHands, Codex CLI, Aider, Terminus — any agent behind Harbor's `BaseAgent`. Each task spins up two isolated services: `main` (the agent's workspace) and `judge` (the upstream go-judge with the real grader). The agent never sees test data or the grader — it only sees the problem statement and its own code. + +
+Design principle: Let the agent use our grader, without letting the agent touch our grader. The score comes from the upstream judge — authentic, byte-equivalent to the native Frontier-CS eval. The agent is fully isolated. +
+ +## Empirical Check: Kimi K2.6 vs Claude Opus 4-7 + +We ran two frontier coding agents on **105 non-interactive algorithmic tasks** from Frontier-CS. The results tell two stories: how hard the problems are, and how differently two top agents attack them. + + +
+
+
105
+
Problems tested
+
+
+
35.1
+
Opus 4-7 score
+
+
+
34.0
+
Kimi K2.6 score
+
+
+ +**Essentially tied on the scoreboard — but wildly different under the hood.** + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricClaude Opus 4-7Kimi K2.6
Partial Score35.134.0Tied
Avg. Steps83.0 up to 45956.5 up to 405+47%
Avg. Turns80.8 up to 45655.7+45%
Avg. Output Tokens200K up to 531K149K+34%
Avg. Thinking Calls—56.5 up to 404Every step
Avg. Tool Calls45.0 up to 29155.7 up to 405+24%
Zero-Score Tasks52 / 10550 / 105~Half
Token Limit Hit1782×
+ + + +
+Two strategies, one scoreboard. Kimi K2.6 thinks on every single step (up to 404 thinking calls per problem) but uses 25% fewer output tokens (149K vs 200K). Only 8 of its tasks hit the 96K output cap, compared to 17 for Opus. Denser thinking, tighter budget, same result.

+Opus compensates with sheer volume: up to 456 turns and up to 531K output tokens on a single problem. Both agents sustain sessions of hundreds of tool calls — this is genuinely long-horizon agentic behavior. Half the problems defeat both agents entirely. These are genuine frontiers. +
+ +What does this mean for long-context data? Every one of these 105 runs produces a **rich, multi-turn trajectory** — the kind of data that is hard to synthesize and expensive to collect from humans. The continuous scoring means each trajectory comes with a natural quality signal. Two different agents attacking the same problem produce structurally different trajectories at the same quality level, giving natural diversity for free. + +## Try It + +The adapter is in the Harbor monorepo and the dataset is published on the Harbor registry: + +```bash +# Sanity check with the oracle agent +uv run harbor run -d frontier-cs-algorithm + +# A real run with Claude Code +uv run harbor run -d frontier-cs-algorithm \ + -a claude-code -m "anthropic/claude-opus-4-6" + +# Or generate the tasks yourself from any Frontier-CS checkout +cd adapters/frontier-cs-algorithm +uv run frontier-cs-algorithm \ + --source https://github.com/FrontierCS/Frontier-CS.git \ + --output-dir ../../datasets/frontier-cs-algorithm \ + --use-published-judge +``` + +Pointers: + +- [Harbor task registry — `yanagiorigami/frontier-cs`](https://registry.harborframework.com/datasets/yanagiorigami/frontier-cs) — the published dataset +- [Adapter source](https://github.com/harbor-framework/harbor/tree/main/adapters/frontier-cs-algorithm) — Harbor monorepo +- [Upstream Frontier-CS](https://github.com/FrontierCS/Frontier-CS) — problems, graders, paper +- [Parity artifacts](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/229) — Hugging Face + +Algorithmic is live today, and the rest of the Frontier-CS tracks are next on the list. Ping us on [Discord](https://discord.com/invite/k4hd2nU4UE) if you hit anything weird, want to contribute a new agent comparison, or just want to chat about open-ended evals. diff --git a/assets/html/2026-04-28-harbor/comparison-chart.html b/assets/html/2026-04-28-harbor/comparison-chart.html new file mode 100644 index 0000000..88c0336 --- /dev/null +++ b/assets/html/2026-04-28-harbor/comparison-chart.html @@ -0,0 +1,109 @@ + + + + + + + + + +
+
+
+
+ + + From 20603636f584164f88b19dc14a3d4b8a9ff5e5a0 Mon Sep 17 00:00:00 2001 From: wenhao Date: Thu, 30 Apr 2026 16:00:03 -0400 Subject: [PATCH 2/7] Polish writing style: kill em-dashes, sync numbers, drop intro repetition - Replace em-dashes with colons or sentence breaks throughout body and front-matter - Remove prose parentheses (Plug in, Two strategies callout, Harbor benchmarks list) - Sync opening callout to table: up to 456 turns / 405 tool calls / 531K tokens - Update front-matter description with current data - Drop the body opener that repeated the lead callout's first two sentences - Normalize Pointers list bullets to colon-separated descriptions Co-Authored-By: Claude Opus 4.7 (1M context) --- _posts/2026-04-28-harbor.md | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/_posts/2026-04-28-harbor.md b/_posts/2026-04-28-harbor.md index f91b0f6..9cdac0c 100644 --- a/_posts/2026-04-28-harbor.md +++ b/_posts/2026-04-28-harbor.md @@ -1,7 +1,7 @@ --- layout: distill title: "Your Next Long-Context Recipe: Open-Ended Problems" -description: "Coding data lifts reasoning. Agentic coding is dominant. We add the next source — 172 open-ended problems with continuous scoring, no ground truth, now one harbor run away. Kimi K2.6 and Claude Opus 4-7 go head-to-head: 50+ turns, 56 tool calls, 191K tokens per problem." +description: "Coding data lifts reasoning. Agentic coding is dominant. We add the next source: 172 open-ended problems with continuous scoring, no ground truth, all one harbor run away. Kimi K2.6 and Claude Opus 4-7 go head-to-head, sustaining up to 456 turns, 405 tool calls, and 531K output tokens per problem." date: 2026-04-29 date_display: "Apr 29, 2026" @@ -92,19 +92,19 @@ _styles: >
-Coding has long been a strong long-context data source. By 2023, mixing coding data into pretraining was already known to lift reasoning. Today, agentic coding is the dominant choice. We're adding another source: 172 open-ended problems with continuous scoring and no ground truth — now one harbor run away. Two frontier agents sustain sessions of up to 450+ turns and up to 400+ tool calls per problem, producing some of the longest agentic trajectories in any public benchmark. +Coding has long been a strong long-context data source. By 2023, mixing coding data into pretraining was already known to lift reasoning. Today, agentic coding is the dominant choice. We're adding another source: 172 open-ended problems with continuous scoring and no ground truth, all one harbor run away. Two frontier agents sustain sessions of up to 456 turns and up to 405 tool calls per problem, producing some of the longest agentic trajectories in any public benchmark.
## A New Source of Long-Context Data -Coding has long been a strong long-context data source. By 2023, mixing coding data into pretraining was already known to lift reasoning. Today, agentic coding is the dominant choice: downstream-aligned, information-dense, globally consistent, dependency-rich. +Agentic coding has settled into the dominant long-context recipe: downstream-aligned, information-dense, globally consistent, dependency-rich. We're adding another coding source: **open-ended problems**. Each ships a domain-specific grader that returns a continuous score, not pass/fail. What makes them a good long-context source? Three properties: 1. **No ceiling.** Every problem has room to improve. An agent that scores 0.6 today can try for 0.7 tomorrow. The trajectory never converges to a single "correct" answer. -2. **Long-horizon reasoning.** Agents sustain sessions of up to 450+ turns and 400+ tool calls, generating up to 530K output tokens on a single problem. This is not a one-shot generation — it is sustained, iterative problem-solving over hundreds of steps. +2. **Long-horizon reasoning.** Agents sustain sessions of up to 456 turns and 405 tool calls, generating up to 531K output tokens on a single problem. This is not one-shot generation. It is sustained, iterative problem-solving over hundreds of steps. 3. **Dense feedback signal.** The continuous score creates a natural reward landscape. Every intermediate solution attempt gets a meaningful gradient, not just "wrong." ## What Makes Open-Ended Problems Different @@ -117,13 +117,13 @@ How hard are these problems for humans? In [our Calico experiment]({{ '/blog/cal ## Example: Polyomino Packing -One concrete example from the [Frontier-CS benchmark](https://frontier-cs.org): given a set of polyomino pieces, pack as many as possible into a fixed grid. The grader scores each submission by packing density — a continuous signal, not pass/fail. +One concrete example from the [Frontier-CS benchmark](https://frontier-cs.org): given a set of polyomino pieces, pack as many as possible into a fixed grid. The grader scores each submission by packing density: a continuous signal, not pass/fail. ![Frontier-CS Teaser](https://raw.githubusercontent.com/FrontierCS/Frontier-CS/main/assets/teaser.png) The search space is combinatorially vast: piece ordering, rotation, placement strategy, and local compaction all interact. There is no known polynomial-time optimal solution. An agent must iteratively generate code, compile, test, analyze results, and refine its approach across many rounds. This single problem can produce trajectories spanning hundreds of agent turns. -The animation below shows an adaptive evolution agent progressively discovering denser packing strategies — better ordering, symmetry handling, and local compaction — over successive iterations: +The animation below shows an adaptive evolution agent progressively discovering denser packing strategies through better ordering, symmetry handling, and local compaction across successive iterations:
@@ -133,12 +133,12 @@ The animation below shows an adaptive evolution agent progressively discovering
-This task is far from saturated. A human expert achieves a packing score of 92, while the best current AI agent reaches only the low 80s — leaving substantial room for improvement. Frontier-CS tasks are designed with deep search spaces and high expert-level ceilings. +This task is far from saturated. A human expert achieves a packing score of 92, while the best current AI agent reaches only the low 80s, leaving substantial room for improvement. Frontier-CS tasks are designed with deep search spaces and high expert-level ceilings.
## 172 Problems, One Command -All 172 open-ended problems ship as a [Harbor](https://harborframework.com) adapter, joining the ~70 benchmarks already on Harbor (SWE-Bench, Terminal-Bench, USACO, GPQA, BFCL, and more). Harbor gives you a standard task format, a standard agent interface, a trial orchestrator, and a provider layer — local Docker, Daytona, Modal, E2B, or GKE. +All 172 open-ended problems ship as a [Harbor](https://harborframework.com) adapter, joining the ~70 benchmarks already on Harbor: SWE-Bench, Terminal-Bench, USACO, GPQA, BFCL, and more. Harbor gives you a standard task format, a standard agent interface, a trial orchestrator, and a provider layer: local Docker, Daytona, Modal, E2B, or GKE. ```bash # Run with any standard agent CLI @@ -146,10 +146,10 @@ uv run harbor run -d frontier-cs-algorithm \ -a claude-code -m "anthropic/claude-opus-4-6" ``` -Plug in Claude Code, OpenHands, Codex CLI, Aider, Terminus — any agent behind Harbor's `BaseAgent`. Each task spins up two isolated services: `main` (the agent's workspace) and `judge` (the upstream go-judge with the real grader). The agent never sees test data or the grader — it only sees the problem statement and its own code. +Plug in Claude Code, OpenHands, Codex CLI, Aider, Terminus, or any agent behind Harbor's `BaseAgent`. Each task spins up two isolated services: `main` for the agent's workspace, and `judge` for the upstream go-judge with the real grader. The agent never sees test data or the grader. It only sees the problem statement and its own code.
-Design principle: Let the agent use our grader, without letting the agent touch our grader. The score comes from the upstream judge — authentic, byte-equivalent to the native Frontier-CS eval. The agent is fully isolated. +Design principle: Let the agent use our grader, without letting the agent touch our grader. The score comes from the upstream judge: authentic, byte-equivalent to the native Frontier-CS eval. The agent is fully isolated.
## Empirical Check: Kimi K2.6 vs Claude Opus 4-7 @@ -172,7 +172,7 @@ We ran two frontier coding agents on **105 non-interactive algorithmic tasks** f -**Essentially tied on the scoreboard — but wildly different under the hood.** +**Essentially tied on the scoreboard. Wildly different under the hood.** @@ -244,11 +244,11 @@ We ran two frontier coding agents on **105 non-interactive algorithmic tasks** f
-Two strategies, one scoreboard. Kimi K2.6 thinks on every single step (up to 404 thinking calls per problem) but uses 25% fewer output tokens (149K vs 200K). Only 8 of its tasks hit the 96K output cap, compared to 17 for Opus. Denser thinking, tighter budget, same result.

-Opus compensates with sheer volume: up to 456 turns and up to 531K output tokens on a single problem. Both agents sustain sessions of hundreds of tool calls — this is genuinely long-horizon agentic behavior. Half the problems defeat both agents entirely. These are genuine frontiers. +Two strategies, one scoreboard. Kimi K2.6 thinks on every single step, with up to 404 thinking calls per problem, while using 25% fewer output tokens: 149K vs 200K. Only 8 of its tasks hit the 96K output cap, compared to 17 for Opus. Denser thinking, tighter budget, same result.

+Opus compensates with sheer volume: up to 456 turns and up to 531K output tokens on a single problem. Both agents sustain sessions of hundreds of tool calls. This is genuinely long-horizon agentic behavior. Half the problems defeat both agents entirely. These are genuine frontiers.
-What does this mean for long-context data? Every one of these 105 runs produces a **rich, multi-turn trajectory** — the kind of data that is hard to synthesize and expensive to collect from humans. The continuous scoring means each trajectory comes with a natural quality signal. Two different agents attacking the same problem produce structurally different trajectories at the same quality level, giving natural diversity for free. +What does this mean for long-context data? Every one of these 105 runs produces a **rich, multi-turn trajectory**, the kind of data that is hard to synthesize and expensive to collect from humans. The continuous scoring means each trajectory comes with a natural quality signal. Two different agents attacking the same problem produce structurally different trajectories at the same quality level, giving natural diversity for free. ## Try It @@ -272,9 +272,9 @@ uv run frontier-cs-algorithm \ Pointers: -- [Harbor task registry — `yanagiorigami/frontier-cs`](https://registry.harborframework.com/datasets/yanagiorigami/frontier-cs) — the published dataset -- [Adapter source](https://github.com/harbor-framework/harbor/tree/main/adapters/frontier-cs-algorithm) — Harbor monorepo -- [Upstream Frontier-CS](https://github.com/FrontierCS/Frontier-CS) — problems, graders, paper -- [Parity artifacts](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/229) — Hugging Face +- [Harbor task registry: `yanagiorigami/frontier-cs`](https://registry.harborframework.com/datasets/yanagiorigami/frontier-cs): the published dataset +- [Adapter source](https://github.com/harbor-framework/harbor/tree/main/adapters/frontier-cs-algorithm): Harbor monorepo +- [Upstream Frontier-CS](https://github.com/FrontierCS/Frontier-CS): problems, graders, paper +- [Parity artifacts](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/229): Hugging Face Algorithmic is live today, and the rest of the Frontier-CS tracks are next on the list. Ping us on [Discord](https://discord.com/invite/k4hd2nU4UE) if you hit anything weird, want to contribute a new agent comparison, or just want to chat about open-ended evals. From 328852dfaf3d9cede7006544821ca2d8ddd13dcc Mon Sep 17 00:00:00 2001 From: wenhao Date: Thu, 30 Apr 2026 16:11:00 -0400 Subject: [PATCH 3/7] Expand Harbor adapter section: isolation, trial flow, parity numbers - After the design-principle callout, add three short paragraphs covering: the structural isolation (read-only mounts, single-problem judge mount, HTTP-only channel), the five-step trial flow, and the three judge image modes (per-trial build, locally cached, published frozen). - Add new section "Parity With the Native Eval" with the 10-problem, 3-run-per-side comparison: 68.92% native vs 53.37% Harbor on claude-code@2.1.112 / claude-opus-4-6, plus the oracle-sweep 70.23% baseline on every problem with a shipped reference. Notes which pieces are intentionally aligned (prompt, CLI, chk.cc) and what changes (transport: in-process vs HTTP). - Update TOC to include the new Parity section. Co-Authored-By: Claude Opus 4.7 (1M context) --- _posts/2026-04-28-harbor.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/_posts/2026-04-28-harbor.md b/_posts/2026-04-28-harbor.md index 9cdac0c..c82c4cd 100644 --- a/_posts/2026-04-28-harbor.md +++ b/_posts/2026-04-28-harbor.md @@ -43,6 +43,7 @@ toc: - name: "What Makes Open-Ended Problems Different" - name: "Example: Polyomino Packing" - name: "172 Problems, One Command" + - name: "Parity With the Native Eval" - name: "Empirical Check: Kimi K2.6 vs Claude Opus 4-7" - name: Try It @@ -152,6 +153,25 @@ Plug in Claude Code, OpenHands, Codex CLI, Aider, Terminus, or any agent behind Design principle: Let the agent use our grader, without letting the agent touch our grader. The score comes from the upstream judge: authentic, byte-equivalent to the native Frontier-CS eval. The agent is fully isolated. +The isolation is structural, not policy. The agent's mounts are read-only and hold only the problem statement, the workspace guide, and a config file. The judge container holds the test data and the `chk.cc` grader, but only ever sees one problem directory at a time. The only channel between them is an HTTP submission endpoint that takes a single C++ file and returns a number. There is no shared filesystem to scrape, no environment variable to read the grader out of, no way to swap the verifier mid-trial. + +Each trial follows the same five-step path: bring up `main` and `judge`, hand the agent the problem statement and let it write, compile, and stress-test inside `main`, run the verifier, which submits `solution.cpp` to `judge:8081/submit`, poll for the result, and write the normalized partial score to `/logs/verifier/reward.txt`. The agent never speaks to the judge. The verifier is the only thing that does. + +For reproducibility, three judge image modes are supported: per-trial build by default, a locally cached `frontier-cs-judge:latest`, or the published frozen image `yanagiorigami/frontier-cs-harbor-judge:latest`. The published image is what we pin for the parity numbers below. + +## Parity With the Native Eval + +Does Harbor preserve native scores? We took 10 problems, the subset of the first 15 with a published reference solution, and ran `claude-code@2.1.112` with `claude-opus-4-6`, three runs each, on both sides: + +| Side | Partial Score (0-100) | +|------|----------------------:| +| Native Frontier-CS eval | 68.92% ± 11.54% | +| Harbor adapter | 53.37% ± 9.88% | + +Three things are aligned on purpose: the agent prompt, the agent CLI invocation, and the `chk.cc` grader inside the go-judge. The adapter's `agent_constants.py` mirrors the upstream version; the only intentional difference is that `CLAUDE.md` is renamed to `AGENT.md` so the same task runs under non-Claude agents too. The only thing that changes between the two evals is transport: in-process on the native side, HTTP on the Harbor side. + +The variance gap mostly comes from runs where the agent ran out of tokens before submitting; per Frontier-CS convention these count as 0. On the full set, the oracle agent, which copies the reference solution, scores **70.23%** across every problem with a shipped reference, with 0 harness errors. Per-problem breakdowns and raw run arrays live in the adapter's [`parity_experiment.json`](https://github.com/harbor-framework/harbor/tree/main/adapters/frontier-cs-algorithm). + ## Empirical Check: Kimi K2.6 vs Claude Opus 4-7 We ran two frontier coding agents on **105 non-interactive algorithmic tasks** from Frontier-CS. The results tell two stories: how hard the problems are, and how differently two top agents attack them. From 249d01baca15b1c363501e1a02c46db8ba420399 Mon Sep 17 00:00:00 2001 From: wenhao Date: Thu, 30 Apr 2026 16:13:37 -0400 Subject: [PATCH 4/7] Reorder sections: K2.6 vs Opus before the Harbor adapter The empirical evidence (105 runs, sustained hundreds of turns) now lands right after the Polyomino example, while "172 Problems, One Command" + "Parity With the Native Eval" move to the end and read as one continuous "here's the adapter, here's parity, here's the command" block leading into "Try It". TOC reordered to match. Co-Authored-By: Claude Opus 4.7 (1M context) --- _posts/2026-04-28-harbor.md | 72 ++++++++++++++++++------------------- 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/_posts/2026-04-28-harbor.md b/_posts/2026-04-28-harbor.md index c82c4cd..5dfa3c4 100644 --- a/_posts/2026-04-28-harbor.md +++ b/_posts/2026-04-28-harbor.md @@ -42,9 +42,9 @@ toc: - name: A New Source of Long-Context Data - name: "What Makes Open-Ended Problems Different" - name: "Example: Polyomino Packing" + - name: "Empirical Check: Kimi K2.6 vs Claude Opus 4-7" - name: "172 Problems, One Command" - name: "Parity With the Native Eval" - - name: "Empirical Check: Kimi K2.6 vs Claude Opus 4-7" - name: Try It _styles: > @@ -137,41 +137,6 @@ The animation below shows an adaptive evolution agent progressively discovering This task is far from saturated. A human expert achieves a packing score of 92, while the best current AI agent reaches only the low 80s, leaving substantial room for improvement. Frontier-CS tasks are designed with deep search spaces and high expert-level ceilings. -## 172 Problems, One Command - -All 172 open-ended problems ship as a [Harbor](https://harborframework.com) adapter, joining the ~70 benchmarks already on Harbor: SWE-Bench, Terminal-Bench, USACO, GPQA, BFCL, and more. Harbor gives you a standard task format, a standard agent interface, a trial orchestrator, and a provider layer: local Docker, Daytona, Modal, E2B, or GKE. - -```bash -# Run with any standard agent CLI -uv run harbor run -d frontier-cs-algorithm \ - -a claude-code -m "anthropic/claude-opus-4-6" -``` - -Plug in Claude Code, OpenHands, Codex CLI, Aider, Terminus, or any agent behind Harbor's `BaseAgent`. Each task spins up two isolated services: `main` for the agent's workspace, and `judge` for the upstream go-judge with the real grader. The agent never sees test data or the grader. It only sees the problem statement and its own code. - -
-Design principle: Let the agent use our grader, without letting the agent touch our grader. The score comes from the upstream judge: authentic, byte-equivalent to the native Frontier-CS eval. The agent is fully isolated. -
- -The isolation is structural, not policy. The agent's mounts are read-only and hold only the problem statement, the workspace guide, and a config file. The judge container holds the test data and the `chk.cc` grader, but only ever sees one problem directory at a time. The only channel between them is an HTTP submission endpoint that takes a single C++ file and returns a number. There is no shared filesystem to scrape, no environment variable to read the grader out of, no way to swap the verifier mid-trial. - -Each trial follows the same five-step path: bring up `main` and `judge`, hand the agent the problem statement and let it write, compile, and stress-test inside `main`, run the verifier, which submits `solution.cpp` to `judge:8081/submit`, poll for the result, and write the normalized partial score to `/logs/verifier/reward.txt`. The agent never speaks to the judge. The verifier is the only thing that does. - -For reproducibility, three judge image modes are supported: per-trial build by default, a locally cached `frontier-cs-judge:latest`, or the published frozen image `yanagiorigami/frontier-cs-harbor-judge:latest`. The published image is what we pin for the parity numbers below. - -## Parity With the Native Eval - -Does Harbor preserve native scores? We took 10 problems, the subset of the first 15 with a published reference solution, and ran `claude-code@2.1.112` with `claude-opus-4-6`, three runs each, on both sides: - -| Side | Partial Score (0-100) | -|------|----------------------:| -| Native Frontier-CS eval | 68.92% ± 11.54% | -| Harbor adapter | 53.37% ± 9.88% | - -Three things are aligned on purpose: the agent prompt, the agent CLI invocation, and the `chk.cc` grader inside the go-judge. The adapter's `agent_constants.py` mirrors the upstream version; the only intentional difference is that `CLAUDE.md` is renamed to `AGENT.md` so the same task runs under non-Claude agents too. The only thing that changes between the two evals is transport: in-process on the native side, HTTP on the Harbor side. - -The variance gap mostly comes from runs where the agent ran out of tokens before submitting; per Frontier-CS convention these count as 0. On the full set, the oracle agent, which copies the reference solution, scores **70.23%** across every problem with a shipped reference, with 0 harness errors. Per-problem breakdowns and raw run arrays live in the adapter's [`parity_experiment.json`](https://github.com/harbor-framework/harbor/tree/main/adapters/frontier-cs-algorithm). - ## Empirical Check: Kimi K2.6 vs Claude Opus 4-7 We ran two frontier coding agents on **105 non-interactive algorithmic tasks** from Frontier-CS. The results tell two stories: how hard the problems are, and how differently two top agents attack them. @@ -270,6 +235,41 @@ Opus compensates with sheer volume: up to 456 turns and +Design principle: Let the agent use our grader, without letting the agent touch our grader. The score comes from the upstream judge: authentic, byte-equivalent to the native Frontier-CS eval. The agent is fully isolated. + + +The isolation is structural, not policy. The agent's mounts are read-only and hold only the problem statement, the workspace guide, and a config file. The judge container holds the test data and the `chk.cc` grader, but only ever sees one problem directory at a time. The only channel between them is an HTTP submission endpoint that takes a single C++ file and returns a number. There is no shared filesystem to scrape, no environment variable to read the grader out of, no way to swap the verifier mid-trial. + +Each trial follows the same five-step path: bring up `main` and `judge`, hand the agent the problem statement and let it write, compile, and stress-test inside `main`, run the verifier, which submits `solution.cpp` to `judge:8081/submit`, poll for the result, and write the normalized partial score to `/logs/verifier/reward.txt`. The agent never speaks to the judge. The verifier is the only thing that does. + +For reproducibility, three judge image modes are supported: per-trial build by default, a locally cached `frontier-cs-judge:latest`, or the published frozen image `yanagiorigami/frontier-cs-harbor-judge:latest`. The published image is what we pin for the parity numbers below. + +## Parity With the Native Eval + +Does Harbor preserve native scores? We took 10 problems, the subset of the first 15 with a published reference solution, and ran `claude-code@2.1.112` with `claude-opus-4-6`, three runs each, on both sides: + +| Side | Partial Score (0-100) | +|------|----------------------:| +| Native Frontier-CS eval | 68.92% ± 11.54% | +| Harbor adapter | 53.37% ± 9.88% | + +Three things are aligned on purpose: the agent prompt, the agent CLI invocation, and the `chk.cc` grader inside the go-judge. The adapter's `agent_constants.py` mirrors the upstream version; the only intentional difference is that `CLAUDE.md` is renamed to `AGENT.md` so the same task runs under non-Claude agents too. The only thing that changes between the two evals is transport: in-process on the native side, HTTP on the Harbor side. + +The variance gap mostly comes from runs where the agent ran out of tokens before submitting; per Frontier-CS convention these count as 0. On the full set, the oracle agent, which copies the reference solution, scores **70.23%** across every problem with a shipped reference, with 0 harness errors. Per-problem breakdowns and raw run arrays live in the adapter's [`parity_experiment.json`](https://github.com/harbor-framework/harbor/tree/main/adapters/frontier-cs-algorithm). + ## Try It The adapter is in the Harbor monorepo and the dataset is published on the Harbor registry: From 561c7f1072d1f722c6bbee83f0ee6c668ea8f858 Mon Sep 17 00:00:00 2001 From: Hanchenli Date: Thu, 30 Apr 2026 17:27:49 -0700 Subject: [PATCH 5/7] update --- _posts/2026-04-28-harbor.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/_posts/2026-04-28-harbor.md b/_posts/2026-04-28-harbor.md index 5dfa3c4..5d3ba87 100644 --- a/_posts/2026-04-28-harbor.md +++ b/_posts/2026-04-28-harbor.md @@ -24,6 +24,10 @@ authors: url: "https://joyemang33.github.io/" affiliations: name: University of California, Berkeley + - name: Hanchen Li + url: "https://hanchenli.github.io/" + affiliations: + name: University of California, Berkeley - name: Lin Shi affiliations: name: Harbor team @@ -93,14 +97,14 @@ _styles: >
-Coding has long been a strong long-context data source. By 2023, mixing coding data into pretraining was already known to lift reasoning. Today, agentic coding is the dominant choice. We're adding another source: 172 open-ended problems with continuous scoring and no ground truth, all one harbor run away. Two frontier agents sustain sessions of up to 456 turns and up to 405 tool calls per problem, producing some of the longest agentic trajectories in any public benchmark. +Coding has long been a strong long-context data source. By 2023, mixing coding data into pretraining was already known to lift reasoning. Today, agentic coding is the dominant choice. Frontier-CS is aiming to add another source: 172 open-ended problems with continuous scoring and no ground truth, all one harbor run away. Two frontier agents sustain sessions of up to 456 turns and up to 405 tool calls per problem, producing some of the longest agentic trajectories in any public benchmark.
## A New Source of Long-Context Data -Agentic coding has settled into the dominant long-context recipe: downstream-aligned, information-dense, globally consistent, dependency-rich. +Coding has been the de facto standard for measuring model performance. The reason is simple: it is a natural domain for long-context, agentic behavior. Coding problems are often complex, with large search spaces and no single "right" answer. They require sustained reasoning, iterative refinement, and tool use. The code itself serves as a persistent memory, allowing agents to build on previous attempts. However, most public coding benchmarks today fail to fully stress test the long horizon capabilities of models or agents similar to where they are truly deployed in real life. -We're adding another coding source: **open-ended problems**. Each ships a domain-specific grader that returns a continuous score, not pass/fail. +Frontier-CS aims to alleviate this problem by adding another coding source: **open-ended problems**. Each ships a domain-specific grader that returns a continuous score, not pass/fail. What makes them a good long-context source? Three properties: @@ -272,7 +276,7 @@ The variance gap mostly comes from runs where the agent ran out of tokens before ## Try It -The adapter is in the Harbor monorepo and the dataset is published on the Harbor registry: +The adapter is already available in the Harbor monorepo and the dataset is published on the Harbor registry: ```bash # Sanity check with the oracle agent From 44e4e89198351cf260810f66ec3dbe0f51bb6079 Mon Sep 17 00:00:00 2001 From: Hanchenli Date: Thu, 30 Apr 2026 17:36:51 -0700 Subject: [PATCH 6/7] update --- _posts/2026-04-28-harbor.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/_posts/2026-04-28-harbor.md b/_posts/2026-04-28-harbor.md index 5d3ba87..2b402b3 100644 --- a/_posts/2026-04-28-harbor.md +++ b/_posts/2026-04-28-harbor.md @@ -239,6 +239,9 @@ Opus compensates with sheer volume: up to 456 turns and Date: Thu, 30 Apr 2026 21:52:20 -0400 Subject: [PATCH 7/7] Same Score, Different Failure Modes: trace-level Kimi/Opus comparison Replace the placeholder Kimi section with a trace-grounded analysis: - All 17 Opus crashes share the same 96K-output-token-per-call cap (verified across all trajectory.json files); none of Kimi's 2 crashes do. - Walk frontier-cs-220 (Opus 0 / Kimi 1.000): Opus's trajectory shows 4 sequential thinking-only assistant chunks, 0 tool calls, empty artifacts dir; Kimi runs 207 turns with explicit context compaction. - Add frontier-cs-0 as a second mode: Opus self-validates its solver on random cases, judge TLEs 70/70 (returncode=-9); Kimi ships conservative algo for 0.74. - Drop the 17 zeros from Opus's average and its mean rises 35.1 -> 41.8. Header renamed from "In-depth Study of How Kimi Achieves Similar Scores" because the trace says Opus hits a self-imposed ceiling, not that Kimi catches up. Co-Authored-By: Claude Opus 4.7 (1M context) --- _posts/2026-04-28-harbor.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/_posts/2026-04-28-harbor.md b/_posts/2026-04-28-harbor.md index 2b402b3..b524ed9 100644 --- a/_posts/2026-04-28-harbor.md +++ b/_posts/2026-04-28-harbor.md @@ -47,6 +47,7 @@ toc: - name: "What Makes Open-Ended Problems Different" - name: "Example: Polyomino Packing" - name: "Empirical Check: Kimi K2.6 vs Claude Opus 4-7" + - name: "Same Score, Different Failure Modes" - name: "172 Problems, One Command" - name: "Parity With the Native Eval" - name: Try It @@ -239,8 +240,17 @@ Opus compensates with sheer volume: up to 456 turns and Previous context has been compacted.` Kimi proactively summarized its working state and kept going. Opus burns its budget inside one extended-thinking block per call; Kimi spreads it across many shorter calls and compacts between them. The first hits a hard ceiling, the second does not. + +A second mode looks softer but lands the same. On `frontier-cs-0`, Opus ran 99 steps, validated its solver against random benchmarks, and concluded `17/18 hit theoretical maximum`. The judge then ran the actual test set: 70 of 70 cases timed out (`returncode=-9`), partial score 0. Kimi shipped a slower, more conservative algorithm and pulled 0.74. Self-confidence on synthetic checks does not predict observed wall time on the real grader. + +Across the 17 problems where Opus hit the cap, Kimi got positive scores on 8, including 1.000 on cs-220 and 0.994 on cs-87. Drop those 17 zeros from Opus's average and its mean rises from 35.1 to 41.8, well ahead of Kimi's 34.0. The headline parity is not Kimi catching up. It is Opus losing points on the hardest problems by hitting a ceiling Kimi's harness does not have. ## 172 Problems, One Command