Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 6 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,13 @@
<p align="center">A shared optimizer platform for AI applications — starting with open-source GEPA on a language-agnostic task contract.</p>
<p align="center">
<a href="https://pypi.org/project/synth-optimizers/">PyPI</a> ·
<a href="https://github.com/synth-laboratories/containers">Containers</a> ·
<a href="src/synth_optimizers/docs/gepa/containers/20-contract.md">Task contract</a> ·
<a href="https://github.com/synth-laboratories/synth-cookbooks-public/tree/main/cookbooks/optimizers/gepa">Cookbooks</a> ·
<a href="docs/hosted-optimizers.md">Hosted jobs</a>
</p>

`synth-optimizers` provides a shared Rust optimizer core for running search
algorithms against any task exposed through the
[`synth-containers`](https://github.com/synth-laboratories/containers) HTTP
algorithms against any task exposed through the public optimizer HTTP task
contract.

- **Shared platform core** — reusable Rust machinery for container I/O, workspaces, cache profiles, budgets, telemetry, failure handling, and replayable evidence.
Expand All @@ -25,7 +24,8 @@ contract.

The shared [`synth_optimizer_platform`](rust/crates/synth_optimizer_platform/)
crate is the substrate for optimizer implementations; GEPA is the first public
local algorithm; GELO is hosted-only in the public package (execution in optimizers-beta).
local algorithm; GELO is hosted-only in the public package and runs on Synth
hosted optimizer infrastructure.
Hosted GEPA/GELO submission is covered in [`docs/hosted-optimizers.md`](docs/hosted-optimizers.md).

## Install
Expand All @@ -36,29 +36,15 @@ pip install synth-optimizers
uv add synth-optimizers
```

Latest daily dev build:

```bash
pip install --pre \
synth-containers==0.2.0.dev202605312141 \
synth-optimizers==0.2.0.dev202606010003

uv add --prerelease allow \
synth-containers==0.2.0.dev202605312141 \
synth-optimizers==0.2.0.dev202606010003
```

Install [`uv`](https://github.com/astral-sh/uv) for local development and editable installs.

## Local development

Pair this repo with [`synth-containers`](https://github.com/synth-laboratories/containers)
at matching dev versions, then sync and install editable:
Sync the repo and install the local Python/Rust extension in editable mode:

```bash
cd optimizers
uv sync --group dev
uv pip install -e ../containers
uv pip install -e .
uv run maturin develop --manifest-path rust/crates/synth_optimizers_py/Cargo.toml
```
Expand Down Expand Up @@ -179,7 +165,7 @@ Agent docs: [skills/gepa/SKILL.md](skills/gepa/SKILL.md).
- [GEPA docs (gepa-ai)](https://gepa-ai.github.io/gepa/) — algorithm overview, case studies, and adapter guides
- [GEPA paper](https://arxiv.org/abs/2507.19457) — *GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning*
- [Cookbooks](https://github.com/synth-laboratories/synth-cookbooks-public/tree/main/cookbooks/optimizers/gepa) — runnable GEPA examples
- [synth-containers](https://github.com/synth-laboratories/containers) — the task contract
- [GEPA task contract](src/synth_optimizers/docs/gepa/containers/20-contract.md) — the public HTTP task contract
- [uv](https://github.com/astral-sh/uv) — Python package and project manager
- [GEPA service OpenAPI](rust/crates/synth_gepa/openapi/gepa-service-v1.yaml)

Expand Down
16 changes: 8 additions & 8 deletions docs/hosted-optimizers.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Hosted optimizer jobs run through the Synth API with `SYNTH_API_KEY`. The public
client and CLI submit to `/api/v1/optimizers/runs`; the backend owns auth,
billing, durable run status, events, and artifacts. No public workflow requires
`OPTIMIZERS_BETA_SERVICE_TOKEN`.
a separate optimizer service token.

By default, `synth-optimizers` also sends a best-effort usage-registration event
when an optimizer run is kicked off. The event is intentionally content-free:
Expand Down Expand Up @@ -53,7 +53,7 @@ frontier_events = list(client.algorithm_events(response.run_id, limit=10))
## GELO

GELO is hosted-only in the public package. There is no public `gelo run`; local
GELO execution remains an internal `optimizers-beta` workflow.
GELO execution is not part of the public product surface.

```bash
export SYNTH_API_KEY="..."
Expand Down Expand Up @@ -107,12 +107,12 @@ Hosted GEPA and GELO share the generic optimizer observability routes:
GELO also exposes compatibility aliases `goex_events()` and `goex_event_stream()`.
Prefer the generic `algorithm_*` methods for new SDK and CLI integrations.

### Plugin lanes
### Extension fields

GELO exposes typed plugin-lane config for roadmap compatibility, but the public
launch accepts only the SFT beta lane. RLVR, OPSD, and unknown plugin kinds are
rejected fail-closed by the SDK materializer and backend submit validation until
their hosted backends are explicitly enabled.
The public hosted GELO path accepts the documented base Go-Explore prompt-space
configuration. Experimental extension fields are not a launch surface; clients
should omit them unless a Synth operator provides a dated compatibility note for
the target hosted API.

## Launch Evidence

Expand All @@ -125,7 +125,7 @@ target API and commits being promoted:
- Public lifecycle event evidence uses bounded `event_backfill()` reads; streaming
`events()` tails are for monitoring, not finite launch proof.
- GELO `gelo submit --preset crafter_smoke --tunnel-url ... --follow` reaches terminal success.
- `gelo watch RUN_ID --slice board` returns public board rows without private paths.
- `gelo watch RUN_ID --slice board` returns public board rows without internal paths.
- Artifacts needed by the public client, such as `checkpoint_frontier`, are readable.
- Billing evidence records optimizer spend for GEPA and GELO, or a launch waiver is recorded.

Expand Down
5 changes: 3 additions & 2 deletions skills/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@ Portable agent skills for working with the public optimizers in this repo.
- `gelo/` — configure and debug **hosted-only** GELO (Go-Explore in prompt space):
per-LLM-call checkpoints, async container contracts, achievement ladders, resume
semantics, three-layer storage (env container / optimizer evidence / hosted durable),
and `HostedOptimizerClient` submit/watch. Local execute stays in `optimizers-beta`.
and `HostedOptimizerClient` submit/watch. Local execute is not part of the
public product surface.

Each skill is a folder with a required `SKILL.md` (Agent Skills format). Runnable
cookbooks that exercise these optimizers live in
[`synth-cookbooks-public`](https://github.com/synth-laboratories/synth-cookbooks-public)
(GEPA) and private cookbooks / `optimizers-beta` examples (GELO containers).
for public GEPA examples and the hosted GELO launch quickstart.
39 changes: 19 additions & 20 deletions skills/gelo/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: gelo
description: Use when building, configuring, or debugging GELO (Go-Explore in prompt space) hosted jobs and GELO-compatible task containers — especially per-LLM-call checkpoint creation, async rollout contracts, achievement ladders, resume semantics, checkpoint retention/eviction, and the three-layer storage model (env container vs optimizer evidence vs hosted durable substrate). GELO is hosted-only in the public package; local execute stays in optimizers-beta.
description: Use when building, configuring, or debugging GELO (Go-Explore in prompt space) hosted jobs and GELO-compatible task containers — especially per-LLM-call checkpoint creation, async rollout contracts, achievement ladders, resume semantics, checkpoint retention/eviction, and the three-layer storage model (env container vs optimizer evidence vs hosted durable substrate). GELO is hosted-only in the public package.
---

# GELO Skill
Expand All @@ -11,8 +11,8 @@ HTTP.

**Product rule (locked):** GELO customer execution is **hosted-only**
(`HostedOptimizerClient.submit_gelo`, `synth-optimizers gelo submit`). Unlike GEPA,
there is no public `gelo run`. Engineers iterate locally via `optimizers-beta`
(`goex_local.sh`, `goex_serve.sh`) — internal, not the customer contract.
there is no public `gelo run`. Local GELO execution is not part of the public
product surface.

Normative hosted API/types: `GELO_HOSTED_SDK_CLI_SPEC.md`. Launch checklist:
`Jstack/.jstack/daily_notes/2026-06-09/gelo_blog_application_lanes_and_release_quality_plan.md`.
Expand All @@ -28,13 +28,13 @@ Load only what the question needs:

| Question | Read first |
|----------|------------|
| Checkpoint / async rollout contract | `optimizers-beta/evals/vending_bench/GO_EX_CONTAINER_REQUIREMENTS.md` §3–5 |
| Why miner needs per-call traces | `optimizers-beta/GO_EX_CHECKPOINT_MINING_DESIGN.md`, `GO_EX_CHECKPOINT_DATA_MINER_HANDOFF.md` |
| Optimizer knobs | `optimizers-beta/examples/*_goex_*_overlay.json`, `crates/synth_go_ex/src/config.rs` |
| Container route matrix | `optimizers-beta/goex_release.txt` §7 |
| Hosted storage boundaries | `optimizers-beta/HOSTED_OPTIMIZER_STORAGE_DESIGN.md` R4, R11 |
| Checkpoint / async rollout contract | `src/synth_optimizers/docs/gelo/containers/20-contract.md` |
| Why miner needs per-call traces | `src/synth_optimizers/docs/gelo/containers/00-overview.md` and the checkpoint sections below |
| Optimizer knobs | `src/synth_optimizers/docs/gelo/20-sdk.md` and `GELO_HOSTED_SDK_CLI_SPEC.md` |
| Container route matrix | `src/synth_optimizers/docs/gelo/containers/20-contract.md` |
| Hosted storage boundaries | `src/synth_optimizers/docs/gelo/30-hosted.md` |
| Public SDK / bundled docs | `src/synth_optimizers/gelo.py`, `hosted.py`, `docs/gelo/`, `GELO_HOSTED_SDK_CLI_SPEC.md` |
| Env-specific | `optimizers-beta/dungeongrid/go_ex.md`, `seven_wonders/go_ex.md` |
| Env-specific | Public GELO cookbook and bundled container docs |

## Mental Model

Expand Down Expand Up @@ -144,8 +144,8 @@ terminal snapshot (VendingBench fix pattern in `react_loop.py`).
Reference implementations:

- Crafter: `src/synth_optimizers/docs/gelo/containers/00-overview.md` (per-LLM checkpoint contract; no private repo dependency)
- VendingBench: `optimizers-beta/evals/vending_bench/agent/react_loop.py`
- Vending store: `optimizers-beta/evals/vending_bench/service/checkpoints.py`
- Public GELO quickstart: `https://github.com/synth-laboratories/synth-cookbooks-public/tree/main/cookbooks/optimizers/gelo`
- Bundled contract docs: `src/synth_optimizers/docs/gelo/containers/20-contract.md`

### Optimizer-side consumption

Expand Down Expand Up @@ -260,9 +260,8 @@ artifacts.
| `frontier_prune_soft_cap` / `retain_tail` | Bound checkpoint frontier projection |
| `rollout_evidence_compact_*` | Compact old evidence while retaining tail |

**Hosted note:** `platform_workspace.sqlite` dominates optimizers-beta disk (~95% of
terminal run dirs). Storage phase S2 snapshots this to S3; terminal local GC follows
publish (R11).
**Hosted note:** optimizer evidence can dominate terminal run storage. Hosted
storage should snapshot terminal evidence before local worker cleanup.

### Layer 3 — Hosted durable substrate (customer observability)

Expand Down Expand Up @@ -334,7 +333,7 @@ with client.open_synth_tunnel("http://127.0.0.1:8943") as tunnel:
synth-optimizers gelo submit --preset crafter_smoke --tunnel-url http://127.0.0.1:8943 --follow
```

Public walkthrough: `https://github.com/synth-laboratories/cookbooks/tree/main/code/training/prompt_learning/gelo`.
Public walkthrough: `https://github.com/synth-laboratories/synth-cookbooks-public/tree/main/cookbooks/optimizers/gelo`.

## Launch promo

Expand All @@ -350,9 +349,9 @@ Check slots before promising a customer run:
Full terms: the hosted docs page (`synth-optimizers gelo console` -> hosted) and the
public cookbook walkthrough above.

**Container URL reachability:** the optimizers-beta worker must reach the container
(from Railway: use SynthTunnel or hosted pool not `127.0.0.1` on the user's laptop
unless tunneled).
**Container URL reachability:** the hosted optimizer worker must reach the
container. Use SynthTunnel or a hosted pool; do not pass `127.0.0.1` unless it
is reachable from the hosted worker through a tunnel.

---

Expand Down Expand Up @@ -392,9 +391,9 @@ When `theme_count=0`, `imported_empty` miner, or resume 404:
Focused checks (do not run full GELO jobs unless the change requires it):

- Container Python: `python -m py_compile synth_service_app.py` (or env's service app).
- Optimizer Rust: `cargo check -p synth_go_ex` from `optimizers-beta`.
- Optimizer Rust: run the relevant Rust check in the operator implementation repo when changing hosted execution code.
- Checkpoint smoke: env-specific scripts (e.g. `dungeongrid_plus_checkpoint_smoke.py`).
- Hosted path: `optimizers-beta/HOSTED_LOCAL_E2E.md` smoke via `HostedOptimizerClient`.
- Hosted path: smoke via the public `HostedOptimizerClient` against the target hosted API.
- Parse contract: one manual `GET /rollouts/{id}` jq inspection of `scheduled_checkpoints`.

Do not add test files unless the user explicitly asks.
Expand Down
4 changes: 2 additions & 2 deletions src/synth_optimizers/docs/gelo/00-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Unlike GEPA (bundled docs via `gepa console`), GELO does **not** offer public lo
execute (`gelo run`).
Customers submit **hosted jobs** via `HostedOptimizerClient` / `synth-optimizers gelo submit`.
Config authoring (presets, materialize, typed `GeloHostedConfig`) lives in this package;
execution runs on Synth infrastructure (`api.usesynth.ai` → optimizers-beta).
execution runs on Synth infrastructure (`api.usesynth.ai` to hosted optimizer workers).

## Install

Expand Down Expand Up @@ -49,7 +49,7 @@ GeloPreset / materialized config_json
└─ seed_candidate react_system_prompt
gelo submit ──► api.usesynth.ai ──► optimizers-beta ──► your container
gelo submit ──► api.usesynth.ai ──► hosted optimizer worker ──► your container
get_state / goex-events / get_artifact(checkpoint_frontier)
Expand Down
6 changes: 3 additions & 3 deletions src/synth_optimizers/docs/gelo/10-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ synth-optimizers gelo materialize \

| Flag | Default | Purpose |
|------|---------|---------|
| `--preset` | source option | Built-in preset. Public presets: `sokoban_smoke` (GameBench, recommended), `crafter_smoke`, `crafter` (legacy synth-containers). |
| `--preset` | source option | Built-in preset. Public presets: `sokoban_smoke` (GameBench, recommended) and `crafter_smoke`. |
| `--toml` | source option | Structured public GELO TOML or JSON input. |
| `--overlay` | — | Structured overlay for `--toml`. |
| `--container-url` | — | Direct container URL. Mutually exclusive with `--container-pool`. |
Expand Down Expand Up @@ -159,5 +159,5 @@ There is no local GELO run board in the public CLI (hosted observability uses
| Not in public CLI | Where it lives |
|-------------------|----------------|
| `gelo run` | **Does not exist** — GELO is hosted-only. |
| `gelo service` | `optimizers-beta` internal serve. |
| Local theme board / TUI | `optimizers-beta` (`goex_tui`, internal scripts). |
| `gelo service` | Internal hosted-service operation. |
| Local theme board / TUI | Internal operator tooling. |
30 changes: 15 additions & 15 deletions src/synth_optimizers/docs/gelo/20-sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,24 +76,24 @@ Wire shape for `config_json` on submit. Sections:
- `go_ex` — engine: `max_rollouts`, `proposer_rounds`, checkpoint cadence/budget, concurrency
- `seed_candidate` — baseline `react_system_prompt`
- `proposers` — map of `GeloProposerRole` → proposer config
- `plugins` — optional plugin lanes; SFT beta is accepted, RLVR/OPSD fail closed
- `plugins` — reserved extension field; omit it for the public hosted GELO path
- `cache`, `disk_budget` — optional

`.to_config_json()` serializes for submit. `GeloHostedConfig.algorithm` is `go-ex`, so it can
be passed directly to `HostedOptimizerClient.submit(config)`.

## Plugin lanes
## Extension fields

GELO exposes typed plugin-lane config so public SDK code can name the extension
point without implying every backend exists. SFT is the only accepted beta lane
in this release; RLVR, OPSD, and unknown plugin kinds are rejected by the SDK
materializer and backend submit validation.
GELO exposes a reserved extension section for future compatibility, but the
public hosted path accepts the documented base Go-Explore prompt-space
configuration. Omit extension lanes unless a Synth operator provides a dated
compatibility note for the target hosted API.

```python
from synth_optimizers.gelo import GeloPluginKind, GeloPluginSection, GeloPluginsSection
from synth_optimizers.gelo import GeloPluginsSection

plugins = GeloPluginsSection(
lanes=(GeloPluginSection(kind=GeloPluginKind.SFT),),
lanes=(),
)
```

Expand All @@ -120,22 +120,22 @@ long GELO runs. Cloudflared and ngrok materialize as public URLs without
`container.auth_refresh`.

Use `.materialize(...)` when you need the raw config JSON dictionary for compatibility.
Materialize from TOML + overlay without hand-editing JSON copied from internal overlays.
Materialize from TOML + overlay without hand-editing hosted worker JSON.

GELO submit defaults to launch-promo billing. Pass `billing_mode="paid"` to use normal
paid hosted GELO without launch-promo model and concurrency gates.

## Config authoring vs execution

| In `optimizers` (public) | In `optimizers-beta` (internal) |
|--------------------------|-----------------------------------|
| Types, presets, materialize | `synth_go_ex` execute |
| `submit_gelo`, observability | `goex_local.sh`, serve, TUI |
| In `optimizers` (public) | Hosted implementation |
|--------------------------|-----------------------|
| Types, presets, materialize | Executes on Synth hosted optimizer workers |
| `submit_gelo`, observability | Exposed through the public Synth API |

Customers never need `OPTIMIZERS_BETA_SERVICE_TOKEN`.
Customers only need `SYNTH_API_KEY` for hosted GELO submission and reads.

## O11y

After submit, poll `get_state` for phase/tick/themes; use `get_state_slice("board")` for
human-readable theme rows; tail `goex_events` for incremental updates. Full normative artifact
names: `goex_release.txt` §8 in optimizers-beta.
names: use the public hosted API response and event schema for compatibility.
8 changes: 4 additions & 4 deletions src/synth_optimizers/docs/gelo/30-hosted.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ but with `algorithm: "go-ex"` and a checkpoint-heavy config shape.
Client (SYNTH_API_KEY)
→ POST /api/v1/optimizers/runs { algorithm: "go-ex", config_json: {...} }
→ backend (auth, billing, projection)
optimizers-beta /v1/runs
hosted optimizer worker
→ HTTP → your task container (rollouts, checkpoints, resume)
```

Expand All @@ -21,7 +21,7 @@ container** — hosted storage does not replace them.

The GELO launch promo gives the first 20 organizations a `$500` hosted Go-Ex proposer
spend grant, valid for 14 days after claim. Hosted `go-ex` submits auto-claim the grant
when slots remain, attach the Autumn product `gelo_launch_promo`, and preflight at least
when slots remain, attach the GELO launch promo grant, and preflight at least
`$1` of `optimizer_go_ex_llm_spend` headroom before the run is queued. In-container policy
LLM calls are still owned by the task container.

Expand Down Expand Up @@ -77,9 +77,9 @@ Container must be up and pass `GET /health` before submit.
| "Durable across restarts" | Storage S1+ (not S0 alone) |
| "Reliable uplift optimizer" | Algorithm Tier B — separate from hosting |

Local E2E proof runbook: `optimizers-beta/HOSTED_LOCAL_E2E.md` (uses public client only).
Local E2E proof should use the public client against the target hosted API.

## Billing

Terminal jobs should record `optimizer_algorithm=go-ex` and `optimizer_go_ex_llm_spend`
(Autumn feature). Policy LLM spend runs in-container; proposer spend on hosted worker.
Policy LLM spend runs in-container; proposer spend runs on the hosted worker.
Loading