diff --git a/_posts/2026-04-28-harbor.md b/_posts/2026-04-28-harbor.md new file mode 100644 index 0000000..b524ed9 --- /dev/null +++ b/_posts/2026-04-28-harbor.md @@ -0,0 +1,317 @@ +--- +layout: distill +title: "Your Next Long-Context Recipe: Open-Ended Problems" +description: "Coding data lifts reasoning. Agentic coding is dominant. We add the next source: 172 open-ended problems with continuous scoring, no ground truth, all one harbor run away. Kimi K2.6 and Claude Opus 4-7 go head-to-head, sustaining up to 456 turns, 405 tool calls, and 531K output tokens per problem." + +date: 2026-04-29 +date_display: "Apr 29, 2026" +htmlwidgets: true + +authors: + - name: Kaiyuan Liu + url: "mailto:lky04@cs.washington.edu" + affiliations: + name: University of Washington + - name: Zhifei Li + url: "mailto:andylizf@outlook.com" + affiliations: + name: Princeton University + - name: Wenhao Chai + url: "https://wenhaochai.com/" + affiliations: + name: Princeton University + - name: Qiuyang Mang + url: "https://joyemang33.github.io/" + affiliations: + name: University of California, Berkeley + - name: Hanchen Li + url: "https://hanchenli.github.io/" + affiliations: + name: University of California, Berkeley + - name: Lin Shi + affiliations: + name: Harbor team + - name: Zhixuan Zhu + affiliations: + name: Harbor team + - name: Crystal Zhou + affiliations: + name: Harbor team + - name: Frontier-CS team + url: "https://frontier-cs.org" + - name: Harbor team + url: "https://harborframework.com" + +toc: + - name: A New Source of Long-Context Data + - name: "What Makes Open-Ended Problems Different" + - name: "Example: Polyomino Packing" + - name: "Empirical Check: Kimi K2.6 vs Claude Opus 4-7" + - name: "Same Score, Different Failure Modes" + - name: "172 Problems, One Command" + - name: "Parity With the Native Eval" + - name: Try It + +_styles: > + d-article img { + max-width: 70%; + height: auto; + display: block; + margin: 1.5rem auto; + border-radius: 6px; + } + d-article img.full { + max-width: 100%; + } + d-article pre { + font-size: 0.85em; + } + d-article .stat-cards { + display: grid; + grid-template-columns: repeat(3, 1fr); + gap: 16px; + margin: 1.5rem 0; + } + d-article .stat-card { + background: #f9fafb; + border-radius: 10px; + padding: 20px; + text-align: center; + border: 1px solid #e5e7eb; + } + d-article .stat-card .number { + font-size: 2em; + font-weight: 800; + line-height: 1.1; + } + d-article .stat-card .label { + font-size: 0.85em; + color: #6b7280; + margin-top: 6px; + } + d-article .opus-color { color: #6366f1; } + d-article .kimi-color { color: #10b981; } +--- + +
+ Frontier CS +
+ +
+Coding has long been a strong long-context data source. By 2023, mixing coding data into pretraining was already known to lift reasoning. Today, agentic coding is the dominant choice. Frontier-CS is aiming to add another source: 172 open-ended problems with continuous scoring and no ground truth, all one harbor run away. Two frontier agents sustain sessions of up to 456 turns and up to 405 tool calls per problem, producing some of the longest agentic trajectories in any public benchmark. +
+ +## A New Source of Long-Context Data + +Coding has been the de facto standard for measuring model performance. The reason is simple: it is a natural domain for long-context, agentic behavior. Coding problems are often complex, with large search spaces and no single "right" answer. They require sustained reasoning, iterative refinement, and tool use. The code itself serves as a persistent memory, allowing agents to build on previous attempts. However, most public coding benchmarks today fail to fully stress test the long horizon capabilities of models or agents similar to where they are truly deployed in real life. + +Frontier-CS aims to alleviate this problem by adding another coding source: **open-ended problems**. Each ships a domain-specific grader that returns a continuous score, not pass/fail. + +What makes them a good long-context source? Three properties: + +1. **No ceiling.** Every problem has room to improve. An agent that scores 0.6 today can try for 0.7 tomorrow. The trajectory never converges to a single "correct" answer. +2. **Long-horizon reasoning.** Agents sustain sessions of up to 456 turns and 405 tool calls, generating up to 531K output tokens on a single problem. This is not one-shot generation. It is sustained, iterative problem-solving over hundreds of steps. +3. **Dense feedback signal.** The continuous score creates a natural reward landscape. Every intermediate solution attempt gets a meaningful gradient, not just "wrong." + +## What Makes Open-Ended Problems Different + +Here's where open-ended problems split from SWE. SWE tasks are well-defined: a spec, a reference behavior, a verifiable right answer. Open-ended problems lack that layer. The reason is not that ground truth is hard to label at scale; it is that mathematically, **none exists**. + +Consider building the largest red-blue edge-colored complete graph with no monochromatic 5-clique. This is the lower-bound problem for the Ramsey number $R(5,5)$: its true value lies somewhere in $[43, 48]$, and nobody knows the exact number. Any construction that pushes the known lower bound up by even one is a genuine contribution. + +How hard are these problems for humans? In [our Calico experiment]({{ '/blog/calico' | relative_url }}), we placed one open-ended problem in UC Berkeley's official programming contest with **2,000+ participants**. Out of 285 submissions, **only one surpassed the strongest AI agent**. + +## Example: Polyomino Packing + +One concrete example from the [Frontier-CS benchmark](https://frontier-cs.org): given a set of polyomino pieces, pack as many as possible into a fixed grid. The grader scores each submission by packing density: a continuous signal, not pass/fail. + +![Frontier-CS Teaser](https://raw.githubusercontent.com/FrontierCS/Frontier-CS/main/assets/teaser.png) + +The search space is combinatorially vast: piece ordering, rotation, placement strategy, and local compaction all interact. There is no known polynomial-time optimal solution. An agent must iteratively generate code, compile, test, analyze results, and refine its approach across many rounds. This single problem can produce trajectories spanning hundreds of agent turns. + +The animation below shows an adaptive evolution agent progressively discovering denser packing strategies through better ordering, symmetry handling, and local compaction across successive iterations: + +
+
+ Online Evolution on Polyomino Packing +
Adaptive evolution progressively discovers denser packing strategies on a polyomino task.
+
+
+ +
+This task is far from saturated. A human expert achieves a packing score of 92, while the best current AI agent reaches only the low 80s, leaving substantial room for improvement. Frontier-CS tasks are designed with deep search spaces and high expert-level ceilings. +
+ +## Empirical Check: Kimi K2.6 vs Claude Opus 4-7 + +We ran two frontier coding agents on **105 non-interactive algorithmic tasks** from Frontier-CS. The results tell two stories: how hard the problems are, and how differently two top agents attack them. + + +
+
+
105
+
Problems tested
+
+
+
35.1
+
Opus 4-7 score
+
+
+
34.0
+
Kimi K2.6 score
+
+
+ +**Essentially tied on the scoreboard. Wildly different under the hood.** + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricClaude Opus 4-7Kimi K2.6
Partial Score35.134.0Tied
Avg. Steps83.0 up to 45956.5 up to 405+47%
Avg. Turns80.8 up to 45655.7+45%
Avg. Output Tokens200K up to 531K149K+34%
Avg. Thinking Calls56.5 up to 404Every step
Avg. Tool Calls45.0 up to 29155.7 up to 405+24%
Zero-Score Tasks52 / 10550 / 105~Half
Token Limit Hit178
+ + + +
+Two strategies, one scoreboard. Kimi K2.6 thinks on every single step, with up to 404 thinking calls per problem, while using 25% fewer output tokens: 149K vs 200K. Only 8 of its tasks hit the 96K output cap, compared to 17 for Opus. Denser thinking, tighter budget, same result.

+Opus compensates with sheer volume: up to 456 turns and up to 531K output tokens on a single problem. Both agents sustain sessions of hundreds of tool calls. This is genuinely long-horizon agentic behavior. Half the problems defeat both agents entirely. These are genuine frontiers. +
+ +What does this mean for long-context data? Every one of these 105 runs produces a **rich, multi-turn trajectory**, the kind of data that is hard to synthesize and expensive to collect from humans. The continuous scoring means each trajectory comes with a natural quality signal. Two different agents attacking the same problem produce structurally different trajectories at the same quality level, giving natural diversity for free. + +## Same Score, Different Failure Modes + +The 17-vs-8 row in the table above is not noise. Every one of Opus's 17 crashed trials exits with the same message, `Claude's response exceeded the 96000 output token maximum`. None of Kimi's two crashes do. + +To see what that ceiling looks like in practice, take **`frontier-cs-220`** ("Playing Around the Table"), where Opus scored 0 and Kimi scored 1.000. Opus's `trajectory.json` has six steps. Step 0 is the user prompt. Steps 1 through 4 each finish with `output_tokens: 8` of visible text and a multi-thousand-character encrypted thinking signature; total completion tokens accumulate to 384K. Step 5 is the API error and the trial exits with code 1. The artifacts directory is empty. Opus spent the entire trial inside extended thinking without emitting one tool call. + +Kimi on the same problem ran 207 context turns over 147 wall-clock minutes, made 41 tool calls, and converged to 1.000. One marker stands out near the top of the trace: `Previous context has been compacted.` Kimi proactively summarized its working state and kept going. Opus burns its budget inside one extended-thinking block per call; Kimi spreads it across many shorter calls and compacts between them. The first hits a hard ceiling, the second does not. + +A second mode looks softer but lands the same. On `frontier-cs-0`, Opus ran 99 steps, validated its solver against random benchmarks, and concluded `17/18 hit theoretical maximum`. The judge then ran the actual test set: 70 of 70 cases timed out (`returncode=-9`), partial score 0. Kimi shipped a slower, more conservative algorithm and pulled 0.74. Self-confidence on synthetic checks does not predict observed wall time on the real grader. + +Across the 17 problems where Opus hit the cap, Kimi got positive scores on 8, including 1.000 on cs-220 and 0.994 on cs-87. Drop those 17 zeros from Opus's average and its mean rises from 35.1 to 41.8, well ahead of Kimi's 34.0. The headline parity is not Kimi catching up. It is Opus losing points on the hardest problems by hitting a ceiling Kimi's harness does not have. + +## 172 Problems, One Command + +All 172 open-ended problems ship as a [Harbor](https://harborframework.com) adapter, joining the ~70 benchmarks already on Harbor: SWE-Bench, Terminal-Bench, USACO, GPQA, BFCL, and more. Harbor gives you a standard task format, a standard agent interface, a trial orchestrator, and a provider layer: local Docker, Daytona, Modal, E2B, or GKE. + +```bash +# Run with any standard agent CLI +uv run harbor run -d frontier-cs-algorithm \ + -a claude-code -m "anthropic/claude-opus-4-6" +``` + +Plug in Claude Code, OpenHands, Codex CLI, Aider, Terminus, or any agent behind Harbor's `BaseAgent`. Each task spins up two isolated services: `main` for the agent's workspace, and `judge` for the upstream go-judge with the real grader. The agent never sees test data or the grader. It only sees the problem statement and its own code. + +
+Design principle: Let the agent use our grader, without letting the agent touch our grader. The score comes from the upstream judge: authentic, byte-equivalent to the native Frontier-CS eval. The agent is fully isolated. +
+ +The isolation is structural, not policy. The agent's mounts are read-only and hold only the problem statement, the workspace guide, and a config file. The judge container holds the test data and the `chk.cc` grader, but only ever sees one problem directory at a time. The only channel between them is an HTTP submission endpoint that takes a single C++ file and returns a number. There is no shared filesystem to scrape, no environment variable to read the grader out of, no way to swap the verifier mid-trial. + +Each trial follows the same five-step path: bring up `main` and `judge`, hand the agent the problem statement and let it write, compile, and stress-test inside `main`, run the verifier, which submits `solution.cpp` to `judge:8081/submit`, poll for the result, and write the normalized partial score to `/logs/verifier/reward.txt`. The agent never speaks to the judge. The verifier is the only thing that does. + +For reproducibility, three judge image modes are supported: per-trial build by default, a locally cached `frontier-cs-judge:latest`, or the published frozen image `yanagiorigami/frontier-cs-harbor-judge:latest`. The published image is what we pin for the parity numbers below. + +## Parity With the Native Eval + +Does Harbor preserve native scores? We took 10 problems, the subset of the first 15 with a published reference solution, and ran `claude-code@2.1.112` with `claude-opus-4-6`, three runs each, on both sides: + +| Side | Partial Score (0-100) | +|------|----------------------:| +| Native Frontier-CS eval | 68.92% ± 11.54% | +| Harbor adapter | 53.37% ± 9.88% | + +Three things are aligned on purpose: the agent prompt, the agent CLI invocation, and the `chk.cc` grader inside the go-judge. The adapter's `agent_constants.py` mirrors the upstream version; the only intentional difference is that `CLAUDE.md` is renamed to `AGENT.md` so the same task runs under non-Claude agents too. The only thing that changes between the two evals is transport: in-process on the native side, HTTP on the Harbor side. + +The variance gap mostly comes from runs where the agent ran out of tokens before submitting; per Frontier-CS convention these count as 0. On the full set, the oracle agent, which copies the reference solution, scores **70.23%** across every problem with a shipped reference, with 0 harness errors. Per-problem breakdowns and raw run arrays live in the adapter's [`parity_experiment.json`](https://github.com/harbor-framework/harbor/tree/main/adapters/frontier-cs-algorithm). + +## Try It + +The adapter is already available in the Harbor monorepo and the dataset is published on the Harbor registry: + +```bash +# Sanity check with the oracle agent +uv run harbor run -d frontier-cs-algorithm + +# A real run with Claude Code +uv run harbor run -d frontier-cs-algorithm \ + -a claude-code -m "anthropic/claude-opus-4-6" + +# Or generate the tasks yourself from any Frontier-CS checkout +cd adapters/frontier-cs-algorithm +uv run frontier-cs-algorithm \ + --source https://github.com/FrontierCS/Frontier-CS.git \ + --output-dir ../../datasets/frontier-cs-algorithm \ + --use-published-judge +``` + +Pointers: + +- [Harbor task registry: `yanagiorigami/frontier-cs`](https://registry.harborframework.com/datasets/yanagiorigami/frontier-cs): the published dataset +- [Adapter source](https://github.com/harbor-framework/harbor/tree/main/adapters/frontier-cs-algorithm): Harbor monorepo +- [Upstream Frontier-CS](https://github.com/FrontierCS/Frontier-CS): problems, graders, paper +- [Parity artifacts](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/229): Hugging Face + +Algorithmic is live today, and the rest of the Frontier-CS tracks are next on the list. Ping us on [Discord](https://discord.com/invite/k4hd2nU4UE) if you hit anything weird, want to contribute a new agent comparison, or just want to chat about open-ended evals. diff --git a/assets/html/2026-04-28-harbor/comparison-chart.html b/assets/html/2026-04-28-harbor/comparison-chart.html new file mode 100644 index 0000000..88c0336 --- /dev/null +++ b/assets/html/2026-04-28-harbor/comparison-chart.html @@ -0,0 +1,109 @@ + + + + + + + + + +
+
+
+
+ + +