Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 19 additions & 4 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The entire Korg backend is implemented in Rust, which provides compile-time guar

- **Ownership and Borrowing**: Rust's ownership model ensures that there is always one and only one "owner" of any given piece of memory. Data can be immutably borrowed by multiple parties or mutably borrowed by exactly one party. This statically prevents data races at compile time. In Korg, shared state like the `Blackboard` is wrapped in concurrency primitives like `Arc<Mutex<T>>`, which enforce these borrowing rules at runtime, ensuring that even in a highly concurrent multi-agent system, there are no data races.
- **Absence of Null and Undefined Behavior**: Rust's type system eliminates null pointer dereferences through the `Option<T>` and `Result<T, E>` enums. This forces developers to handle the absence of a value explicitly, preventing entire classes of bugs and security vulnerabilities common in other systems languages.
- **Fearless Concurrency**: The combination of ownership rules and type-system constraints allows Korg to manage the 4-Persona Swarm with "fearless concurrency." The compiler guarantees that any data shared between threads (i.e., between the Leader orchestrator and its various tasks) is done so safely.
- **Fearless Concurrency**: The combination of ownership rules and type-system constraints allows Korg to manage the 5-Persona Swarm with "fearless concurrency." The compiler guarantees that any data shared between threads (i.e., between the Leader orchestrator and its various tasks) is done so safely.

### 1.2. Isolated Execution via Process Sandboxing

Expand All @@ -34,17 +34,17 @@ At the heart of Korg's concurrency model is the **Blackboard**, a central, obser
- **CRDT-like Merging**: The `perform_semantic_merge` function in `src/leader.rs` exemplifies this principle. It takes the outputs from all competing personas and uses a "Synthesizer" persona (Lucas) to reconcile them into a single, cohesive set of mutations. This is analogous to a CRDT merge function that resolves concurrent updates into a deterministic, final state.
- **Observability**: The Blackboard is the "Gravitational Well" around which the swarm orbits. It is the single source of truth for system telemetry. Personas emit `SwarmTelemetryPulse` messages, which the Leader ingests into the Blackboard's `trace_buffer`. The `Evaluator` then reads this buffer to assess the swarm's health, creating a tight, observable feedback loop. The state is exposed via the `/api/state` endpoint for the web cockpit, providing real-time observability to the human operator.

## 3. 4-Persona Adversarial Swarm Topology
## 3. 5-Persona Adversarial Swarm Topology

Korg employs a fixed 4-persona swarm, where each agent has a specialized, adversarial role. This division of labor creates a system of checks and balances that promotes robust, high-quality output.
Korg employs a fixed 5-persona swarm, where each agent has a specialized, adversarial role. This division of labor creates a system of checks and balances that promotes robust, high-quality output.

1. **Orchestrator Captain (The Planner)**:
- **Role**: High-level strategist and planner.
- **Function**: Receives the initial root task from the operator. Decomposes the task into smaller, actionable work packages (`decompose_into_persona_packages` in `src/leader.rs`). Initiates the `ContractNegotiation` phase, proposing acceptance criteria for the task. The Captain sets the direction for the entire swarm.

2. **Auditor Harper (The Researcher/Critic)**:
- **Role**: Context-gatherer, researcher, and adversarial critic.
- **Function**: During `ContractNegotiation`, Harper (acting as part of the `Evaluator`) critiques the Captain's proposed criteria for ambiguity and relevance, forcing a more robust contract. During execution, Harper is assigned "Research" tasks, gathering information from the codebase or external sources to inform the Builder. Harper's primary function is to prevent the swarm from "hallucinating" or working with incorrect assumptions.
- **Function**: During `ContractNegotiation`, Harper critiques the Captain's proposed criteria for ambiguity and relevance, forcing a more robust contract. During execution, Harper is assigned "Research" tasks, gathering information from the codebase or external sources to inform the Builder. Harper's primary function is to prevent the swarm from "hallucinating" or working with incorrect assumptions.

3. **Builder Benjamin (The Coder)**:
- **Role**: Primary code generator and implementer.
Expand All @@ -54,6 +54,10 @@ Korg employs a fixed 4-persona swarm, where each agent has a specialized, advers
- **Role**: Merger and finalizer.
- **Function**: After all personas have submitted their work, the `LeaderOrchestrator` invokes Lucas. As shown in `perform_semantic_merge`, Lucas's job is to take the winning proposal from the **Arena** and intelligently merge it with complementary ideas from the other personas. This produces a final, cohesive patch that is more robust than any single persona's output.

5. **Evaluator (The Harsh Adversarial Critic)**:
- **Role**: Independent scoring authority and Generator/Evaluator loop.
- **Function**: Spawned separately by the Leader (`Spawning Evaluator to score persona ...`), the Evaluator receives a generator persona's output and scores it against the negotiated contract rubrics in the **Arena**, following the Anthropic-style Generator/Evaluator loop. It is a distinct persona (`Persona::Evaluator`), not a role played by any of the four workers above, and it is what turns the swarm's output into a measured, adversarially-checked verdict.

This adversarial collaboration ensures that a plan is critiqued before it's executed, code is generated based on a solid plan, and multiple proposed solutions are synthesized into a superior final product.

## 4. State Transition Sequence
Expand Down Expand Up @@ -143,6 +147,17 @@ Korg maintains a tamper-proof audit trail of every significant action using a cr
- Simultaneously, it uses the `state_merkle_root` to find the corresponding state blob and rehydrate the logical `Blackboard`.
- This powerful feature allows an operator to rewind the AI's "thought process" and codebase to any valid, signed point in history and "fork" its execution in a new direction, all with cryptographic guarantees of integrity.

### 5.1. Verifiable Flight-Recorder Capabilities

Beyond the in-process provenance chain above, Korg ships a set of independently-verifiable, "flight-recorder" capabilities that let third parties check an agent's work without trusting Korg itself:

- **`korg-ledger@v1`**: A frozen, hash-chained, HLC-ordered, tamper-evident event ledger with cross-language conformance (Rust/Python/JS). Each event carries an Ed25519 per-event signature, and the ledger supports git-tip structural anchoring for offline structural verification.
- **Gold Seal (`goldseal@v1`)**: A public, independently-verifiable certificate of agent work, produced by the `korg-seal` CLI (mint/anchor/resolve/verify) and checkable by three conformant verifiers (`korg-verify` in Rust, plus Python and JS) as well as zero-install in-browser verifiers hosted on GitHub Pages.
- **Honest `korg run-once` pipeline**: Runs a real patch through a real `cargo check` and attests a mutation count that *equals* the real `git diff` (file count + changed paths). When it cannot produce a real result, it reports an honest null rather than fabricating one.
- **Provider model**: The default is a hermetic `DeterministicProvider` (fixture-only); `--provider ollama` runs a live local model on arbitrary tasks.

**Planned (not yet shipped):** live *network* time-witness anchoring (the anchor structure and offline verification are shipped; only the trusted-time witness fetch remains an honest limit), `crates.io`/npm publishing of the CLIs and verifiers, and a `korg fork` / execution-checkpoint-restore CLI (the primitives exist; the shipped reversibility today is `korg rewind`).

## 6. OCR Pixel Redaction & Visual Firewall (`src/vision_policy.rs`)

Since Korg personas can interact with GUIs and web browsers, a critical security vector is the accidental exposure of sensitive data in screenshots. The Visual Policy Engine in `src/vision_policy.rs` acts as a fail-secure firewall for all visual data.
Expand Down
53 changes: 37 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,12 @@ korg campaign --web --prompt "Optimize the database connection pool"
# Pure autonomous goal mode (--goal is a top-level flag)
korg --goal "Write and validate a full test suite for src/parser.rs"

# Run the full multi-persona swarm on a REAL local model — every persona
# (Captain, Harper, Benjamin, Lucas, Evaluator) runs as a real worker
# subprocess doing real, measured, attested work. Defaults to a hermetic
# deterministic provider; `--provider ollama` makes it live.
korg --goal "Fix the failing test in src/lib.rs" --provider ollama --model qwen2.5:7b

# Preview without committing (dry-run; --preview is a top-level flag)
korg --preview "Refactor the main event loop"
```
Expand Down Expand Up @@ -216,6 +222,14 @@ korg-verify <path-to-ledger.jsonl>
> pipeline cannot attest a number the worktree does not actually show — that is
> the guarantee, independent of model quality.
> **Verify it in your browser — sends nothing.** Zero-install, client-side
> verifiers (Web Crypto) for any `korg-ledger@v1` journal or Gold Seal:
> [verify a session](https://new1direction.github.io/korg/web/index.html) ·
> [verify a Gold Seal](https://new1direction.github.io/korg/web/seal.html) ·
> [time-travel explorer](https://new1direction.github.io/korg/web/explore.html).
> They hash-chain, check the causal DAG, validate Ed25519 signatures, and
> re-derive the human summary from the events — all locally.
> Speculative branch/fork and named checkpoints (`korg fork`, `korg checkpoints
> list|restore`) are planned, not yet shipped. The reversibility surface today is
> `korg rewind`.
Expand Down Expand Up @@ -267,6 +281,8 @@ Korg treats AI cognition the same way a hypervisor treats compute and Git treats
| Speculative branches | 🚧 planned ||||
| Execution checkpoints | 🚧 planned ||||
| Cryptographic audit trail |||||
| Independently-verifiable Gold Seal |||||
| Honest attestation (real diff, never fabricated) |||||
| Micro-healing |||||
| Model-agnostic |||||

Expand All @@ -282,7 +298,7 @@ Korg treats AI cognition the same way a hypervisor treats compute and Git treats
| Ledger ordering | Hybrid Logical Clocks (HLC) |
| Workspace snapshots | Git Merkle tree (O(1) restore via `write-tree` / `read-tree`) |
| Cryptographic attestation | Ed25519 (ed25519-dalek) |
| Semantic governance | BERT cosine similarity (Candle / Hugging Face) |
| Semantic governance | BERT cosine similarity via the optional `candle` feature (Hugging Face); a deterministic embedding fallback runs when `candle` is not built |
| TUI dashboard | Ratatui + Crossterm |
| Web cockpit | Axum + SSE |
| Syntax highlighting | Syntect + tree-sitter |
Expand All @@ -291,7 +307,7 @@ Korg treats AI cognition the same way a hypervisor treats compute and Git treats

## Architecture Deep Dive

**[Read the full technical write-up](https://github.com/New1Direction/korg/blob/main/ARCHITECTURE.md)** *(coming soon)*
**[Read the full technical write-up](https://github.com/New1Direction/korg/blob/main/ARCHITECTURE.md)**

### Real-World Audit Ledger Example
You can inspect a real-world cognitive audit ledger produced by Korg. This NDJSON file records a live session where Claude Code was prompted to call Korg's MCP tools to refactor a function and rename all call sites, capturing the full HLC causal graph and `actor_id` recorder metadata:
Expand All @@ -309,23 +325,28 @@ The short version:

## Status

Korg is in active development. Current test coverage: **175 tests, 0 failures** (162 cargo across 8 crates + 13 pytest for the PyO3 bridge).
Korg is in active development, built on a **frozen `korg-ledger@v1` spec with cross-language conformance** (Rust + Python + JS). Test footprint: **300+ Rust tests across the workspace plus Python/JS conformance suites**, CI-gated (build · tests · cross-language oracle · differential fuzz) and green on `main`.

- [x] Append-only cognitive ledger with HLC ordering
**Shipped:**
- [x] Append-only, hash-chained cognitive ledger with HLC ordering
- [x] Deterministic replay and projection rebuilds
- [x] Reversible execution — rewind the ledger to any prior sequence point (tamper-evident `LedgerRewind`)
- [x] Per-event Ed25519 signatures + structural anchoring (`korg-ledger@v1` §8)
- [x] **Gold Seal (`goldseal@v1`)** — a public, independently-verifiable certificate of agent work, with zero-install in-browser verifiers
- [x] **Honest pipeline** (`korg run-once`) — real patch → real `cargo check` → an attested mutation count that equals the real `git diff`; never fabricates (reports an honest null instead)
- [x] **Live local model** (`--provider ollama`) — real per-persona work on arbitrary tasks
- [x] **Multi-agent swarm** (Captain, Harper, Benjamin, Lucas, Evaluator) — genuine worker subprocesses doing real, measured, attested work with DAG data-flow between personas
- [x] Zero-config Claude Code capture (PostToolUse/Stop hooks → verifiable per-session ledgers)
- [x] Micro-healing effect layer · TUI dashboard + Web cockpit
- [x] Cryptographic provenance attestation · single-authority CognitionMode governance
- [x] Preview / dry-run mode (`--preview`)
- [ ] Speculative warm-boot execution (in progress)
- [ ] Execution checkpoints / restore CLI (primitive exists; CLI planned)
- [x] Micro-healing effect layer
- [x] Multi-agent swarm orchestration (Captain, Harper, Benjamin, Lucas)
- [x] TUI dashboard + Web cockpit
- [x] Cryptographic provenance attestation
- [x] Single-authority CognitionMode governance
- [ ] `cargo install korg` on crates.io
- [ ] Remote swarm workers
- [ ] WASM backends
- [ ] IDE language server integration
- [ ] Distributed checkpoint synchronization

**Planned / not yet shipped:**
- [ ] Speculative branches / fork + execution-checkpoint restore CLI (primitives exist; CLI planned)
- [ ] `cargo install korg` on crates.io · npm-published verifier
- [ ] Live network anchoring resolver (trusted-time witness — the remaining honest limit)
- [ ] Remote swarm workers · WASM backends · IDE language-server integration · distributed checkpoint sync
- [ ] Fully passive capture without agent cooperation

---

Expand Down
1 change: 0 additions & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ without human intervention.
| Feature | Description |
|---|---|
| `korg autoheal` | Autonomous recovery loop: detect failure → identify root cause → rewind to last clean checkpoint → retry with corrected strategy |
| Micro-healing at the effect layer | Per-tool failure recovery without full session rewind |
| Incremental checkpoints | Background checkpoint compression — only delta since last checkpoint is stored |
| Recovery confidence scoring | Each autoheal attempt is scored; low-confidence recoveries escalate to human approval |
| Doom-loop circuit breaker | Hard limit on recursive recovery attempts with structured escalation |
Expand Down
36 changes: 24 additions & 12 deletions adapters/claude-code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,17 +55,24 @@ graph identical in shape to what korgex's own agent loop produces.

## Two ways to use it

### 1. Live tail mode (recommended)
> **Recommended live path: `korg-setup` + `korg-hook`.** The verifiable,
> zero-config capture model (PostToolUse/Stop hook → per-session verifiable
> ledger at `~/.korg/sessions/<id>.jsonl`) is described in
> [What it produces](#what-it-produces). The `korg-ingest-claude --tail/--once`
> CLI below still works, but it emits the **legacy un-chained flat ledger**;
> run `korg-backfill --migrate-flat <path>` to upgrade an existing flat ledger
> to the per-session format.

`korg follows you in real time` — the adapter watches
`~/.claude/projects/**/*.jsonl` and streams new events into a korg
ledger as Claude Code writes them. Install, run once, get a continuously-
growing ledger forever:
### Legacy flat-file tail (`korg-ingest-claude`)

The adapter watches `~/.claude/projects/**/*.jsonl` and streams new events into
a **legacy flat (un-chained)** korg ledger as Claude Code writes them. Install,
run once, get a continuously-growing flat ledger:

```bash
pip install -e ./adapters/claude-code

# Watch ~/.claude/projects, append events to ~/.korg/claude-events.jsonl
# Watch ~/.claude/projects, append events to ~/.korg/claude-events.jsonl (legacy flat ledger)
korg-ingest-claude --tail

# Just print events to stdout (no file write):
Expand All @@ -83,18 +90,23 @@ The byte-offset state is persistent — kill the process and restart any
time; it picks up exactly where it left off. The state file at
`~/.korg/claude-tail-state.json` survives reboots.

### 2. One-shot backfill
### One-shot backfill of history

For the existing pile of session files on your disk right now, use
`korg-backfill` — it re-derives the **verifiable per-session ledgers** under
`~/.korg/sessions/` for every historical session.

For the existing pile of session files on your disk right now:
The legacy `korg-ingest-claude --once` still exists, but it writes the **legacy
un-chained flat ledger** instead:

```bash
korg-ingest-claude --once --out ~/.korg/claude-events.jsonl
# [korg-ingest-claude] one-shot pass complete · files_active=42 events=128304 ...
korg-ingest-claude --once --out ~/.korg/claude-events.jsonl # legacy flat ledger
# [korg-ingest-claude] one-shot pass complete · files_active=N events=M ...
```

Runs once, ingests everything you haven't ingested yet, exits.

### 3. Library usage
### Library usage

For embedding into your own ledger plumbing:

Expand All @@ -109,7 +121,7 @@ def emit(body: dict) -> int | None:


# Single-session replay:
adapter = ClaudeCodeAdapter(emit, source_agent="agent:claude-code@2.1.150")
adapter = ClaudeCodeAdapter(emit, source_agent="agent:claude-code@2.1.0")
with Path("~/.claude/projects/-Users-you-Documents-foo/abc.jsonl").expanduser().open() as f:
stats = adapter.ingest(f)

Expand Down
13 changes: 3 additions & 10 deletions adapters/korg-setup/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,16 +109,9 @@ you can preview every action before committing.

## Tests

45 tests cover:

- 14 for the `~/.claude.json` editor (idempotent register/remove,
atomic backups, preservation of unrelated keys, env vars).
- 13 for the launchd integration (plist shape, write idempotency,
install/uninstall, platform gating, `is_loaded` parser).
- 11 for the setup orchestrator (binary detection, dry-run, idempotency,
ledger-dir creation, launchctl-failure reporting, no-daemon path).
- 7 for the status reporter (empty install, partial install, loaded
launchd detection, format rendering).
A test suite covers the `~/.claude.json` editor, launchd integration,
setup orchestrator, status reporter, discovery, bridge registration, and
Claude settings.

Run them with the Korg workspace venv:

Expand Down
10 changes: 6 additions & 4 deletions adapters/recall-mcp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,17 +181,19 @@ Result text format (one line per match, designed for direct LLM consumption):

## Tests

49 cover everything end-to-end:
Tests cover everything end-to-end:

- 9 tests for the text flattener (per-event-type extraction, trimming, fallbacks).
- 9 tests for the index (incremental load, malformed-line tolerance,
- 11 tests for the index (incremental load, malformed-line tolerance,
partial-line hold-back, multi-file, new-file pickup, triggered_by preservation).
- 17 tests for the recall engine (substring + semantic, top_n, min_score,
- 16 tests for the recall engine (substring + semantic, top_n, min_score,
tool_filter, automatic refresh, auto-mode fallback, explicit-semantic
raising without fastembed).
- 14 tests for the MCP server (full initialize/list/call roundtrip,
- 16 tests for the MCP server (full initialize/list/call roundtrip,
unknown-method errors, notification handling, ping, tool_filter through
the JSON-RPC surface, malformed-line tolerance, serve_stdio loop).
- 17 tests for the introspect document (korg:introspect@v1 schema,
callables, exit_codes, command_id stability).

Run them with the Korg workspace venv (which has fastembed):

Expand Down
Loading
Loading