diff --git a/README.md b/README.md index a942b39..bc6fc97 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,7 @@ claude cd /path/to/em-aisoftwarefactory # 3. Auto-routes to correct repository and injects knowledge -python3 -m orchestrator implement SEMI-1413 +python3 -m harness implement SEMI-1413 # 4. Follow the printed instructions in Claude Code ``` @@ -83,7 +83,7 @@ An **Engineering OS** that provides: | Guide | Description | |-------|-------------| | **[Quickstart](docs/guides/QUICKSTART.md)** | Get started | -| **[Orchestrator Usage](#orchestrator-usage)** | Single & multi-repo | +| **[Harness Usage](#harness-usage)** | Single & multi-repo | | **[Skills Reference](#skills-reference)** | All available skills | | **[Knowledge System](#knowledge-system)** | Architecture, ADRs, patterns | @@ -92,18 +92,18 @@ An **Engineering OS** that provides: | Doc | Purpose | |-----|---------| | **[Foundations Standards](knowledge/foundations/standards.md)** | Air-gapped, DoD, engineering principles | -| **[Orchestrator Guide](docs/guides/ORCHESTRATOR_GUIDE.md)** | Complete orchestrator usage | +| **[Harness Guide](docs/guides/HARNESS_GUIDE.md)** | Complete harness usage | | **[Complete Docs](docs/README.md)** | Full documentation index | --- -## Orchestrator Usage +## Harness Usage -The orchestrator provides **workspace-level automation** with repository routing and knowledge injection. +The harness provides **workspace-level automation** with repository routing and knowledge injection. ### Single Repository -**Direct skill invocation (no orchestrator):** +**Direct skill invocation (no harness):** ```bash # 1. Navigate to repository @@ -136,18 +136,18 @@ claude ### Multi-Repository with Orchestrator -**Orchestrator-based (recommended for production):** +**Harness-based (recommended for production):** ```bash # Step 1: Test routing -python3 -m orchestrator test SEMI-1413 +python3 -m harness test SEMI-1413 # Output: # Routed SEMI-1413 → semi # Loaded knowledge: architecture, patterns, conventions # Step 2: Generate implementation instructions -python3 -m orchestrator implement SEMI-1413 +python3 -m harness implement SEMI-1413 # Output: # Knowledge context prepared: /tmp/knowledge_context_xyz.md @@ -162,7 +162,7 @@ python3 -m orchestrator implement SEMI-1413 # (Opens Claude Code and runs the skill with knowledge context) ``` -**What the orchestrator adds:** +**What the harness adds:** - **Auto-routing**: SEMI-1413 → em-semi (via Jira component) - **Knowledge injection**: em-semi architecture/patterns - **Foundations enforcement**: Air-gapped, 80% coverage, DoD @@ -186,14 +186,14 @@ jira: ```bash # Implement multiple issues across repositories -python3 -m orchestrator multi-repo SEMI-1413 T2D-890 RT-567 +python3 -m harness multi-repo SEMI-1413 T2D-890 RT-567 # Output: # SEMI-1413 → em-semi # T2D-890 → em-talk2data # RT-567 → em-runtime # -# Generated 3 instruction sets (see /tmp/orchestrator_instructions_*.sh) +# Generated 3 instruction sets (see /tmp/harness_instructions_*.sh) ``` --- @@ -256,7 +256,7 @@ knowledge/ ### Automatic Sync ```bash -# Runs automatically before orchestrator +# Runs automatically before harness ./sync_knowledge.sh # Or manually @@ -378,7 +378,7 @@ jira: ./sync_knowledge.sh # 3. Test routing -python3 -m orchestrator test YOUR-ISSUE-123 +python3 -m harness test YOUR-ISSUE-123 # 4. Verify knowledge loaded # Should show: "Loaded knowledge for your-repo" @@ -389,7 +389,7 @@ python3 -m orchestrator test YOUR-ISSUE-123 ## Requirements - **Claude Code** v2.1.81+ (for skills) -- **Python 3.8+** (for orchestrator) +- **Python 3.8+** (for harness) - **Git** (for repositories) - **Jira MCP** (optional, for real Jira data) @@ -414,11 +414,11 @@ export JIRA_API_TOKEN=your_api_token ### Implement Single Issue ```bash -# Quick (no orchestrator) +# Quick (no harness) cd /path/to/your/repo && /autonomous-implement SEMI-1413 # Full (with knowledge) -python3 -m orchestrator implement SEMI-1413 +python3 -m harness implement SEMI-1413 # Follow instructions ``` @@ -470,7 +470,7 @@ ls -la knowledge/repositories/semi/ ```bash # Test routing -python3 -m orchestrator test SEMI-1413 +python3 -m harness test SEMI-1413 # Check Jira component mapping cat workspace.yaml | grep -A 10 "component_mapping:" @@ -488,7 +488,7 @@ See [CRITICAL_FIX_BRANCHING.md](CRITICAL_FIX_BRANCHING.md) for details. 1. **[5 min]** Read [Quickstart Guide](docs/guides/QUICKSTART.md) 2. **[10 min]** Try `/autonomous-implement` on a real issue -3. **[15 min]** Set up orchestrator for your workspace +3. **[15 min]** Set up harness for your workspace 4. **[Optional]** Configure Jira MCP for real data --- diff --git a/adapters/base.py b/adapters/base.py index 2deaf87..1486059 100644 --- a/adapters/base.py +++ b/adapters/base.py @@ -32,7 +32,7 @@ class RepositoryAdapter(ABC): Each repository implements this interface to provide standardized access to build, test, lint, and metadata operations. - The adapter pattern allows the orchestrator to remain agnostic + The adapter pattern allows the harness to remain agnostic to repository-specific details while maintaining consistent operations. """ diff --git a/docs/README.md b/docs/README.md index 50dd166..54bf13a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -9,7 +9,7 @@ Complete documentation for AI Software Factory. | Doc | Description | |-----|-------------| | **[Quick Start](guides/QUICKSTART.md)** | Get started quickly | -| **[Orchestrator Guide](guides/ORCHESTRATOR_GUIDE.md)** | Complete orchestrator usage | +| **[Harness Guide](guides/HARNESS_GUIDE.md)** | Complete harness usage | | **[Main README](../README.md)** | Overview and installation | --- @@ -51,7 +51,7 @@ Platform engineering standards and requirements: How to use the factory effectively: - **[Quickstart](guides/QUICKSTART.md)** - Get started -- **[Orchestrator Guide](guides/ORCHESTRATOR_GUIDE.md)** - Complete orchestrator usage +- **[Harness Guide](guides/HARNESS_GUIDE.md)** - Complete harness usage - **[Testing Guide](guides/TESTING_GUIDE.md)** - Multi-agent system testing **Common Tasks:** @@ -78,7 +78,7 @@ docs/ ├── README.md # This file └── guides/ # User guides ├── QUICKSTART.md # Quick start - ├── ORCHESTRATOR_GUIDE.md # Orchestrator usage + ├── HARNESS_GUIDE.md # Harness usage └── TESTING_GUIDE.md # Testing multi-agent knowledge/foundations/ @@ -103,7 +103,7 @@ Knowledge automatically extracted from repositories: - Zero maintenance (read-only generated artifacts) ```bash -# Automatic sync before orchestrator runs +# Automatic sync before harness runs ensure_knowledge_fresh() # Manual sync anytime diff --git a/docs/guides/ORCHESTRATOR_GUIDE.md b/docs/guides/HARNESS_GUIDE.md similarity index 90% rename from docs/guides/ORCHESTRATOR_GUIDE.md rename to docs/guides/HARNESS_GUIDE.md index b5c9dd0..e8d26df 100644 --- a/docs/guides/ORCHESTRATOR_GUIDE.md +++ b/docs/guides/HARNESS_GUIDE.md @@ -1,4 +1,4 @@ -# Orchestrator Guide +# Harness Guide **Complete guide to workspace-level orchestration with knowledge injection.** @@ -21,10 +21,10 @@ ```bash # Test routing -python3 -m orchestrator test SEMI-1413 +python3 -m harness test SEMI-1413 # Generate instructions -python3 -m orchestrator implement SEMI-1413 +python3 -m harness implement SEMI-1413 # Follow the printed instructions in Claude Code ``` @@ -42,7 +42,7 @@ python3 -m orchestrator implement SEMI-1413 cd ~/Documents/Development/em-semi # 2. Start Claude Code -claude --plugin-dir ~/Documents/Development/EM-AISoftwareFactory/.claude/plugins/em-software-factory +claude --plugin-dir ~/Documents/Development/EM-AISoftwareFactory//Users/malamunisamy/Documents/Development/EM-AISoftwareFactory # 3. Run skill /autonomous-implement SEMI-1413 @@ -50,7 +50,7 @@ claude --plugin-dir ~/Documents/Development/EM-AISoftwareFactory/.claude/plugins **Pros:** - Simple, direct -- Fast (no orchestrator overhead) +- Fast (no harness overhead) **Cons:** - No automatic knowledge injection @@ -65,7 +65,7 @@ claude --plugin-dir ~/Documents/Development/EM-AISoftwareFactory/.claude/plugins ```bash # From workspace root -python3 -m orchestrator implement SEMI-1413 +python3 -m harness implement SEMI-1413 # Output: # Routed SEMI-1413 → semi @@ -73,7 +73,7 @@ python3 -m orchestrator implement SEMI-1413 # # To execute: # 1. cd ~/Documents/Development/EM-AISoftwareFactory -# 2. claude --plugin-dir .claude/plugins/em-software-factory +# 2. claude --plugin-dir /Users/malamunisamy/Documents/Development/EM-AISoftwareFactory # 3. cd ~/Documents/Development/em-semi # 4. /autonomous-implement SEMI-1413 --context-file /tmp/knowledge_context_xyz.md @@ -99,7 +99,7 @@ python3 -m orchestrator implement SEMI-1413 # Example: ARCH-500 needs changes in runtime AND runtime-ui # Step 1: Check which repos -python3 -m orchestrator multi-repo ARCH-500 +python3 -m harness multi-repo ARCH-500 # Output: # Routed ARCH-500 to repositories: runtime, runtime-ui @@ -109,8 +109,8 @@ python3 -m orchestrator multi-repo ARCH-500 # 2. runtime-ui (depends on runtime) # # Instructions generated: -# /tmp/orchestrator_instructions_ARCH-500-runtime.sh -# /tmp/orchestrator_instructions_ARCH-500-runtime-ui.sh +# /tmp/harness_instructions_ARCH-500-runtime.sh +# /tmp/harness_instructions_ARCH-500-runtime-ui.sh # Step 2: Execute in order # (Follow instructions for runtime first, then runtime-ui) @@ -123,7 +123,7 @@ python3 -m orchestrator multi-repo ARCH-500 ```bash # Implement 5 issues across 3 repositories -python3 -m orchestrator multi-repo SEMI-1413 SEMI-1414 T2D-890 RT-567 UI-123 +python3 -m harness multi-repo SEMI-1413 SEMI-1414 T2D-890 RT-567 UI-123 # Output: # Routing summary: @@ -250,7 +250,7 @@ The AI reads this context and: ./sync_knowledge.sh # Automatic sync -# Runs before orchestrator implement/multi-repo commands +# Runs before harness implement/multi-repo commands # Check what was extracted ls -la knowledge/repositories/semi/ @@ -431,7 +431,7 @@ jira: ./sync_knowledge.sh # 3. Test routing -python3 -m orchestrator test NEWREPO-123 +python3 -m harness test NEWREPO-123 # Should output: # Routed NEWREPO-123 → new-repo @@ -502,11 +502,11 @@ When implementing analytics features: ```bash # Normally routes based on Jira component -python3 -m orchestrator implement SEMI-1413 +python3 -m harness implement SEMI-1413 # Routes to: semi # Force different repository (future enhancement) -python3 -m orchestrator implement SEMI-1413 --repo runtime +python3 -m harness implement SEMI-1413 --repo runtime # Routes to: runtime (override) ``` @@ -531,13 +531,13 @@ grep -r "event sourcing" knowledge/repositories/semi/ ```bash # Test routing without implementation -python3 -m orchestrator test SEMI-1413 +python3 -m harness test SEMI-1413 # Verbose output -python3 -m orchestrator test SEMI-1413 --verbose +python3 -m harness test SEMI-1413 --verbose # Test multiple -python3 -m orchestrator test SEMI-1413 T2D-890 RT-567 +python3 -m harness test SEMI-1413 T2D-890 RT-567 # Output shows routing decision for each ``` @@ -558,7 +558,7 @@ vim workspace.yaml # Add: YourComponent: your-repo # Test -python3 -m orchestrator test YOUR-123 +python3 -m harness test YOUR-123 ``` ### "No knowledge found" @@ -579,8 +579,8 @@ ls -la ~/Documents/Development/em-your-repo/README.md ```bash # The context file is temporary -# Run the orchestrator implement command again -python3 -m orchestrator implement SEMI-1413 +# Run the harness implement command again +python3 -m harness implement SEMI-1413 # It will create a new context file # Use it immediately (files may be cleaned up) @@ -607,9 +607,9 @@ ls -la ~/Documents/Development/ | Scenario | Command | Knowledge | Routing | |----------|---------|-----------|---------| | **Quick single-repo** | `cd repo && /autonomous-implement` | No | Manual (you cd) | -| **Single-repo + knowledge** | `orchestrator implement` | Yes | Auto | -| **Multi-repo** | `orchestrator multi-repo` | Yes | Auto | -| **Batch** | `orchestrator multi-repo ISSUE1 ISSUE2...` | Yes | Auto | +| **Single-repo + knowledge** | `harness implement` | Yes | Auto | +| **Multi-repo** | `harness multi-repo` | Yes | Auto | +| **Batch** | `harness multi-repo ISSUE1 ISSUE2...` | Yes | Auto | ### Knowledge Injection @@ -622,4 +622,4 @@ ls -la ~/Documents/Development/ --- -**The orchestrator transforms "which repo?" and "what patterns?" into "just implement it correctly."** +**The harness transforms "which repo?" and "what patterns?" into "just implement it correctly."** diff --git a/docs/guides/QUICKSTART.md b/docs/guides/QUICKSTART.md index 6623ce4..2d64a3a 100644 --- a/docs/guides/QUICKSTART.md +++ b/docs/guides/QUICKSTART.md @@ -31,7 +31,7 @@ claude --plugin-dir /path/to/EM-AISoftwareFactory - **Claude Code** v2.1.81+ - **Git** repository -- **Python 3.8+** (for orchestrator) +- **Python 3.8+** (for harness) --- @@ -78,7 +78,7 @@ That's it! The skill will: ```bash cd ~/Documents/Development/EM-AISoftwareFactory -python3 -m orchestrator test SEMI-1413 +python3 -m harness test SEMI-1413 ``` Output shows which repository the issue routes to and knowledge loaded. @@ -86,12 +86,12 @@ Output shows which repository the issue routes to and knowledge loaded. ### 2. Generate Instructions ```bash -python3 -m orchestrator implement SEMI-1413 +python3 -m harness implement SEMI-1413 ``` ### 3. Follow the Instructions -The orchestrator prints exact commands to run in Claude Code with knowledge context. +The harness prints exact commands to run in Claude Code with knowledge context. **Benefits:** - Auto-routes to correct repository @@ -227,7 +227,7 @@ The system uses mock data if Jira MCP is not configured. Issue keys must match p ## Next Steps 1. Try `/autonomous-implement` on a real issue -2. Read [Orchestrator Guide](ORCHESTRATOR_GUIDE.md) +2. Read [Harness Guide](HARNESS_GUIDE.md) 3. Set up Jira MCP for real data (optional) --- @@ -235,5 +235,5 @@ The system uses mock data if Jira MCP is not configured. Issue keys must match p ## Learn More - **Full README:** [../../README.md](../../README.md) -- **Orchestrator Guide:** [ORCHESTRATOR_GUIDE.md](ORCHESTRATOR_GUIDE.md) +- **Harness Guide:** [HARNESS_GUIDE.md](HARNESS_GUIDE.md) - **Engineering Standards:** [../../knowledge/foundations/standards.md](../../knowledge/foundations/standards.md) diff --git a/orchestrator/README.md b/harness/README.md similarity index 91% rename from orchestrator/README.md rename to harness/README.md index 2bad213..ab9e516 100644 --- a/orchestrator/README.md +++ b/harness/README.md @@ -1,4 +1,4 @@ -# Orchestrator - Workspace-Level Orchestration +# Harness - Workspace-Level Orchestration **Thin orchestration layer that enhances existing skills with repository knowledge and multi-repo coordination.** @@ -6,7 +6,7 @@ ## What It Does -The orchestrator **delegates** to existing `/autonomous-implement` skill while providing: +The harness **delegates** to existing `/autonomous-implement` skill while providing: 1. **Repository Routing** - Auto-routes Jira issues to correct repository 2. **Knowledge Injection** - Provides repo-specific architecture/patterns to skills @@ -40,7 +40,7 @@ The orchestrator **delegates** to existing `/autonomous-implement` skill while p └─────────────────────────────────────────────────────────────┘ ``` -**Key Insight:** The orchestrator does NOT reimplement the SDLC workflow. It enhances existing skills with knowledge context. +**Key Insight:** The harness does NOT reimplement the SDLC workflow. It enhances existing skills with knowledge context. --- @@ -73,26 +73,26 @@ Invokes `/autonomous-implement` with enriched context: ```bash # Implement single issue (auto-route) -python -m orchestrator implement ABI-123 +python -m harness implement ABI-123 # Implement in specific repository -python -m orchestrator implement ABI-123 --repo runtime +python -m harness implement ABI-123 --repo runtime # Multi-repository implementation -python -m orchestrator multi-repo SDK-456 --repos sdk,runtime,runtime-ui +python -m harness multi-repo SDK-456 --repos sdk,runtime,runtime-ui # View repository knowledge -python -m orchestrator knowledge --repo runtime -python -m orchestrator knowledge --list +python -m harness knowledge --repo runtime +python -m harness knowledge --list -# Test orchestrator components -python -m orchestrator test ABI-123 +# Test harness components +python -m harness test ABI-123 ``` ### Programmatic API ```python -from orchestrator import Executor, Router, KnowledgeEngine +from harness import Executor, Router, KnowledgeEngine from pathlib import Path import yaml @@ -184,7 +184,7 @@ link_cross_repo_prs(results) ## Knowledge Injection Mechanism -The orchestrator creates a temporary knowledge context file: +The harness creates a temporary knowledge context file: ```markdown # Repository Knowledge Context @@ -272,13 +272,13 @@ cd ~/em-runtime ```bash # From workspace root (any directory): -python -m orchestrator implement ABI-123 +python -m harness implement ABI-123 # Auto-routes to em-runtime # Injects runtime architecture/patterns # Enforces Foundations standards # Multi-repo issue: -python -m orchestrator multi-repo SDK-456 +python -m harness multi-repo SDK-456 # Detects affected repos: sdk, runtime, runtime-ui # Executes in parallel with repo-specific knowledge # Links PRs together @@ -345,8 +345,8 @@ jira: ## Testing ```bash -# Test orchestrator components -python -m orchestrator test ABI-123 +# Test harness components +python -m harness test ABI-123 # Output: # Testing Router... @@ -369,10 +369,10 @@ python -m orchestrator test ABI-123 ### Example 1: Single Repository ```bash -$ python -m orchestrator implement ABI-123 +$ python -m harness implement ABI-123 ============================================================ -AI Software Factory - Workspace Orchestrator +AI Software Factory - Workspace Harness ============================================================ Fetching issue: ABI-123 @@ -410,10 +410,10 @@ Repositories: 1 ### Example 2: Multi-Repository ```bash -$ python -m orchestrator multi-repo SDK-456 +$ python -m harness multi-repo SDK-456 ============================================================ -AI Software Factory - Multi-Repo Orchestrator +AI Software Factory - Multi-Repo Harness ============================================================ Fetching issue: SDK-456 diff --git a/orchestrator/__init__.py b/harness/__init__.py similarity index 85% rename from orchestrator/__init__.py rename to harness/__init__.py index 61a4251..8b977e0 100644 --- a/orchestrator/__init__.py +++ b/harness/__init__.py @@ -7,7 +7,7 @@ - Executor: Delegates to /autonomous-implement skill with knowledge context - Multi-repo coordination: Handles cross-repository dependencies -The orchestrator is a THIN layer that enhances existing skills with: +The harness is a THIN layer that enhances existing skills with: - Repository-specific knowledge injection - Multi-repository routing and coordination - Foundations standards enforcement @@ -37,9 +37,9 @@ def ensure_knowledge_fresh(verbose: bool = False): """ - Ensure knowledge packs are up-to-date before orchestrator runs. + Ensure knowledge packs are up-to-date before harness runs. - This is called automatically by orchestrator entry points. + This is called automatically by harness entry points. Only re-extracts if source documentation has changed. Args: diff --git a/harness/__main__.py b/harness/__main__.py new file mode 100644 index 0000000..e62aeea --- /dev/null +++ b/harness/__main__.py @@ -0,0 +1,13 @@ +""" +Entry point for running the harness as a module. + +Usage: + python -m harness implement ABI-123 + python -m harness knowledge --repo runtime + python -m harness test ABI-123 +""" + +from .cli import main + +if __name__ == '__main__': + main() diff --git a/harness/checkpoint.py b/harness/checkpoint.py new file mode 100644 index 0000000..1f2ecea --- /dev/null +++ b/harness/checkpoint.py @@ -0,0 +1,151 @@ +"""Checkpoint — write after each successful step so harness can resume after crash. + +Checkpoint file: ``{repo_path}/.harness-results/checkpoint.json`` + +File schema:: + + { + "run_id": "run_20240726_abc123", + "issue_key": "ABI-123", + "completed_steps": ["research", "plan", "implement"], + "last_step": "implement", + "timestamp": "2024-07-26T12:00:00Z" + } + +Atomic write is achieved by writing to a ``.tmp`` sibling file and then +calling ``os.replace()`` so readers never observe a partial state. + +Typical usage:: + + from pathlib import Path + from harness.checkpoint import Checkpoint + + cp = Checkpoint(repo_path) + + # On startup, resume from previous crash if checkpoint exists. + completed = cp.completed_steps() + + for step in all_steps: + if cp.should_skip(step): + print(f"[checkpoint] skipping already-completed step: {step}") + continue + + run_step(step) + completed.append(step) + cp.write(run_id, issue_key, completed) + + cp.clear() +""" + +import json +import os +from datetime import datetime, timezone +from pathlib import Path +from typing import List, Optional + + +class Checkpoint: + """Read/write harness checkpoints for a single repository. + + Parameters + ---------- + repo_path: + Root directory of the repository being operated on. The checkpoint + file is stored at ``{repo_path}/.harness-results/checkpoint.json``. + """ + + _SUBDIR = ".harness-results" + _FILENAME = "checkpoint.json" + _TMP_FILENAME = ".checkpoint.json.tmp" + + def __init__(self, repo_path: Path) -> None: + self._base_dir = Path(repo_path) / self._SUBDIR + self._path = self._base_dir / self._FILENAME + self._tmp_path = self._base_dir / self._TMP_FILENAME + + # ------------------------------------------------------------------ + # Public API + # ------------------------------------------------------------------ + + def write(self, run_id: str, issue_key: str, completed_steps: List[str]) -> None: + """Atomically write a checkpoint record. + + Writes to a ``.tmp`` file first, then uses ``os.replace()`` for an + atomic rename so the checkpoint file is never in a partial state. + + Parameters + ---------- + run_id: + Unique identifier for the current harness run. + issue_key: + Jira / issue key associated with the run. + completed_steps: + Ordered list of step names that have been completed successfully. + """ + self._base_dir.mkdir(parents=True, exist_ok=True) + + record = { + "run_id": run_id, + "issue_key": issue_key, + "completed_steps": list(completed_steps), + "last_step": completed_steps[-1] if completed_steps else None, + "timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + } + + # Write to tmp first, then atomically replace the real file. + self._tmp_path.write_text(json.dumps(record, indent=2)) + os.replace(self._tmp_path, self._path) + + def read(self) -> Optional[dict]: + """Return the checkpoint dict, or ``None`` if no checkpoint exists. + + Returns + ------- + dict or None + Parsed checkpoint record, or ``None`` if the file does not exist + or cannot be decoded. + """ + try: + content = self._path.read_text().strip() + if not content: + return None + return json.loads(content) + except (FileNotFoundError, json.JSONDecodeError, OSError): + return None + + def clear(self) -> None: + """Delete the checkpoint file. + + Should be called at run end (whether success or failure) to prevent + a stale checkpoint from being picked up by the next run. Safe to + call even if no checkpoint exists. + """ + try: + self._path.unlink() + except FileNotFoundError: + pass + # Also clean up any leftover tmp file. + try: + self._tmp_path.unlink() + except FileNotFoundError: + pass + + def completed_steps(self) -> List[str]: + """Return the list of already-completed step names, or ``[]``. + + Convenience wrapper around :meth:`read` that always returns a list. + """ + record = self.read() + if record is None: + return [] + return record.get("completed_steps", []) + + def should_skip(self, step_name: str) -> bool: + """Return ``True`` if *step_name* is already recorded as completed. + + Parameters + ---------- + step_name: + Name of the step to check (e.g. ``"research"``, ``"plan"``). + """ + return step_name in self.completed_steps() diff --git a/harness/circuit_breaker.py b/harness/circuit_breaker.py new file mode 100644 index 0000000..60f5466 --- /dev/null +++ b/harness/circuit_breaker.py @@ -0,0 +1,208 @@ +"""Circuit breaker — prevents runaway retries when a gate is systemically broken. + +State persists across harness runs in ``provenance/circuit_breakers.json``:: + + { + "linter": { "state": "open", "consecutive_failures": 5, + "opened_at": "2024-07-26T12:00:00Z", "last_error": "..." }, + "tests": { "state": "closed", "consecutive_failures": 0 }, + "evals": { "state": "closed", "consecutive_failures": 2 }, + "code-review": { "state": "closed", "consecutive_failures": 0 } + } + +A gate transitions **closed → open** after :data:`OPEN_THRESHOLD` consecutive +failures. Once open, :meth:`CircuitBreaker.check` raises +:class:`CircuitOpenError` immediately so the harness does not keep trying a +broken gate. + +Use :meth:`CircuitBreaker.reset` (or fix the underlying problem and call +:meth:`CircuitBreaker.record_success`) to close an open circuit. + +Typical usage:: + + from pathlib import Path + from harness.circuit_breaker import CircuitBreaker, CircuitOpenError + + cb = CircuitBreaker(provenance_dir=Path("provenance")) + + # Before running a gate: + try: + cb.check("linter") + except CircuitOpenError as exc: + print(f"Skipping linter gate — circuit open: {exc}") + mark_run_failed() + return + + # After the gate finishes: + if gate_passed: + cb.record_success("linter") + else: + cb.record_failure("linter", error=stderr_tail) +""" + +import json +import os +from datetime import datetime, timezone +from pathlib import Path +from typing import Dict, Optional + + +OPEN_THRESHOLD = 5 # consecutive failures required to open a circuit + + +class CircuitOpenError(Exception): + """Raised by :meth:`CircuitBreaker.check` when the circuit is open. + + The exception message includes the gate name, failure count, time opened, + and the last recorded error to aid diagnosis. + """ + + +class CircuitBreaker: + """Persistent, cross-run circuit breaker for harness gates. + + Parameters + ---------- + provenance_dir: + Directory where provenance artefacts are stored. The state file is + written as ``{provenance_dir}/circuit_breakers.json``. + """ + + _FILENAME = "circuit_breakers.json" + _TMP_FILENAME = ".circuit_breakers.json.tmp" + + def __init__(self, provenance_dir: Path) -> None: + self._dir = Path(provenance_dir) + self._path = self._dir / self._FILENAME + self._tmp_path = self._dir / self._TMP_FILENAME + + # ------------------------------------------------------------------ + # Public API + # ------------------------------------------------------------------ + + def check(self, gate: str) -> None: + """Raise :class:`CircuitOpenError` if *gate*'s circuit is open. + + Parameters + ---------- + gate: + Gate identifier (e.g. ``"linter"``, ``"tests"``). + + Raises + ------ + CircuitOpenError + When the gate's circuit is in the ``"open"`` state. + """ + state = self._load() + entry = state.get(gate, {}) + if entry.get("state") == "open": + raise CircuitOpenError( + f"Circuit for gate '{gate}' is OPEN after " + f"{entry.get('consecutive_failures', OPEN_THRESHOLD)} consecutive failures " + f"(opened at {entry.get('opened_at', 'unknown')}). " + f"Last error: {entry.get('last_error', 'N/A')}. " + f"Fix the underlying issue, then call reset('{gate}') to re-enable." + ) + + def record_failure(self, gate: str, error: str = "") -> None: + """Increment the failure counter for *gate*. + + Transitions the circuit to ``"open"`` when the counter reaches + :data:`OPEN_THRESHOLD`. + + Parameters + ---------- + gate: + Gate identifier. + error: + Optional short description / tail of the error output for + diagnostic purposes. + """ + state = self._load() + entry = state.setdefault(gate, {"state": "closed", "consecutive_failures": 0}) + entry["consecutive_failures"] = entry.get("consecutive_failures", 0) + 1 + entry["last_error"] = error + + if entry["consecutive_failures"] >= OPEN_THRESHOLD and entry["state"] != "open": + entry["state"] = "open" + entry["opened_at"] = _utc_now() + print( + f"[circuit-breaker] OPENED gate '{gate}' after " + f"{entry['consecutive_failures']} consecutive failures." + ) + else: + entry["state"] = "closed" + + self._save(state) + + def record_success(self, gate: str) -> None: + """Reset the failure counter for *gate* and close the circuit. + + Parameters + ---------- + gate: + Gate identifier. + """ + state = self._load() + entry = state.setdefault(gate, {"state": "closed", "consecutive_failures": 0}) + was_open = entry.get("state") == "open" + entry["consecutive_failures"] = 0 + entry["state"] = "closed" + entry.pop("opened_at", None) + entry.pop("last_error", None) + self._save(state) + + if was_open: + print(f"[circuit-breaker] CLOSED gate '{gate}' after successful run.") + + def reset(self, gate: str) -> None: + """Manually reset *gate* to closed state with zero failures. + + Use this after fixing an underlying infrastructure problem that caused + the circuit to open. + + Parameters + ---------- + gate: + Gate identifier. + """ + state = self._load() + state[gate] = {"state": "closed", "consecutive_failures": 0} + self._save(state) + print(f"[circuit-breaker] Manually RESET gate '{gate}'.") + + def all_states(self) -> Dict[str, dict]: + """Return the full state dictionary for all tracked gates. + + Returns + ------- + dict + Mapping of gate name → state entry. Keys within each entry: + ``state``, ``consecutive_failures``, and optionally ``opened_at`` + and ``last_error``. + """ + return self._load() + + # ------------------------------------------------------------------ + # Internal helpers + # ------------------------------------------------------------------ + + def _load(self) -> Dict: + """Load and return the state dict from disk; returns ``{}`` on any error.""" + try: + content = self._path.read_text().strip() + if not content: + return {} + return json.loads(content) + except (FileNotFoundError, json.JSONDecodeError, OSError): + return {} + + def _save(self, state: Dict) -> None: + """Atomically write *state* to the state file.""" + self._dir.mkdir(parents=True, exist_ok=True) + self._tmp_path.write_text(json.dumps(state, indent=2)) + os.replace(self._tmp_path, self._path) + + +def _utc_now() -> str: + return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") diff --git a/harness/cli.py b/harness/cli.py new file mode 100755 index 0000000..417b847 --- /dev/null +++ b/harness/cli.py @@ -0,0 +1,821 @@ +#!/usr/bin/env python3 +""" +Harness CLI — Workspace-level orchestration for the Dark Factory. + +Commands +-------- + implement Implement a Jira issue (skill mode or harness mode) + watch Tail live events for an active run + queue Show active + queued runs + cancel Cancel an active run + resume Resume a run from its last checkpoint + circuit-breaker Show or reset gate circuit breaker state + tui Launch the terminal dashboard + server Start the observability server + cost Show cost breakdown by repo / step + runs List recent runs with filters + provenance Query provenance logs and export RL datasets + sprint Execute multiple issues from a sprint + knowledge View repository knowledge + test Smoke-test harness components + +Usage examples +-------------- + python -m harness implement ABI-123 --harness + python -m harness watch ABI-123 + python -m harness queue + python -m harness cancel ABI-123 + python -m harness resume run_20240726_abc123 + python -m harness circuit-breaker status + python -m harness circuit-breaker reset linter + python -m harness tui + python -m harness server + python -m harness cost --days 7 + python -m harness runs --repo runtime --outcome failed + python -m harness provenance stats + python -m harness provenance export --output rl_dataset.json +""" + +import json +import sys +import argparse +import time +from pathlib import Path +import yaml +from typing import List, Optional + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from harness.router import Router +from harness.executor import Executor +from harness.harness import Harness +from harness.knowledge import KnowledgeEngine +from harness import jira_mcp + +_FACTORY_ROOT = Path(__file__).parent.parent +_PROVENANCE_DIR = _FACTORY_ROOT / "provenance" + + +def load_workspace_config() -> dict: + """Load workspace.yaml configuration.""" + workspace_file = Path(__file__).parent.parent / 'workspace.yaml' + + if not workspace_file.exists(): + print(f"❌ Error: workspace.yaml not found at {workspace_file}") + print(" Create workspace.yaml with repository configuration first.") + sys.exit(1) + + with open(workspace_file) as f: + return yaml.safe_load(f) + + +def _resolve_repository(args, workspace_config) -> str: + """Route issue to a repository unless --repo was specified explicitly.""" + if args.repo: + return args.repo + + router = Router(workspace_config) + component = jira_mcp.get_issue_component(args.issue_key) + repository = jira_mcp.get_repository_for_issue( + args.issue_key, + workspace_config.get('jira', {}).get('component_mapping', {}), + ) + if not repository: + repository = 'runtime' + print(f"🎯 Repository: {repository} (default — unknown component)") + else: + print(f"🎯 Repository: {repository} (routed from {component})") + return repository + + +def cmd_implement(args): + """Implement a single Jira issue with workspace-level orchestration.""" + + print(f"\n{'='*60}") + print(f"AI Software Factory - Workspace Harness") + mode = "harness" if getattr(args, 'harness', False) else "skill" + print(f"Mode: {mode}") + print(f"{'='*60}\n") + + workspace_config = load_workspace_config() + factory_root = Path(__file__).parent.parent + + print(f"📋 Issue: {args.issue_key}") + repository = _resolve_repository(args, workspace_config) + print() + + if getattr(args, 'harness', False): + # ── Harness mode: step-by-step loop with provenance ────────────── + harness = Harness( + factory_root, + workspace_config, + max_gate_attempts=getattr(args, 'max_gate_attempts', 3), + auto_merge=getattr(args, 'auto_merge', False), + ) + result = harness.implement(issue_key=args.issue_key, repository=repository) + print(result.summary()) + sys.exit(0 if result.overall_outcome in ("success", "partial") else 1) + else: + # ── Skill mode: delegate to /autonomous-implement (original) ───── + executor = Executor(factory_root, workspace_config) + result = executor.execute_single_repo( + issue_key=args.issue_key, + repository=repository, + ) + print(result.summary() if hasattr(result, 'summary') else str(result)) + sys.exit(0 if result.success else 1) + + +def cmd_multi_repo(args): + """Implement issue across multiple repositories.""" + + print(f"\n{'='*60}") + print(f"AI Software Factory - Multi-Repo Orchestrator") + print(f"{'='*60}\n") + + # Load configuration + workspace_config = load_workspace_config() + factory_root = Path(__file__).parent.parent + + # Fetch Jira issue + print(f"📋 Fetching issue: {args.issue_key}") + issue = get_jira_issue(args.issue_key) + print(f" Summary: {issue['summary']}\n") + + # Determine affected repositories + if args.repos: + repositories = args.repos.split(',') + print(f"🎯 Repositories: {', '.join(repositories)} (explicit)") + else: + router = Router(workspace_config) + repositories = router.get_affected_repositories(issue) + print(f"🎯 Repositories: {', '.join(repositories)} (auto-detected)") + + print() + + # Initialize executor + executor = Executor(factory_root, workspace_config) + + # Execute across repositories + result = executor.execute_multi_repo( + issue_key=args.issue_key, + repositories=repositories + ) + + # Print summary + print(result.summary()) + + # Exit with appropriate code + sys.exit(0 if result.overall_success else 1) + + +def cmd_provenance(args): + """Query provenance logs and export RL datasets.""" + from harness.provenance import ProvenanceLogger + + factory_root = Path(__file__).parent.parent + prov_dir = factory_root / "provenance" + + if not prov_dir.exists(): + print("No provenance data found. Run with --harness first.") + sys.exit(1) + + logger = ProvenanceLogger(prov_dir) + + if args.sub == "stats": + print(f"\n{'='*60}") + print("Provenance Statistics") + print(f"{'='*60}\n") + + success_rate = logger.success_rate() + avg_attempts = logger.avg_attempts_to_pass() + gate_failures = logger.gate_failure_rates() + + print(f" Success rate : {success_rate*100:.1f}%") + print(f" Avg gate attempts: {avg_attempts:.2f}") + print() + print(" Gate failure rates:") + for gate, rate in sorted(gate_failures.items(), key=lambda x: -x[1]): + bar = "█" * int(rate * 20) + print(f" {gate:<14} {rate*100:5.1f}% {bar}") + print() + + elif args.sub == "export": + output = Path(args.output) + n = logger.export_rl_dataset(output) + print(f"✅ Exported {n} trajectories → {output}") + + elif args.sub == "runs": + runs_dir = prov_dir / "runs" + summaries = sorted(runs_dir.glob("*.summary.json"), reverse=True) + print(f"\n{'='*60}") + print(f"Recent Runs ({len(summaries)} total)") + print(f"{'='*60}\n") + import json + for f in summaries[:20]: + s = json.loads(f.read_text()) + icon = {"success": "✅", "partial": "⚠️ ", "failed": "❌"}.get(s["overall_outcome"], "?") + print(f" {icon} {s['run_id']} {s['issue_key']:<12} {s['repository']:<14} " + f"gates:{s['gate_attempts']} reward:{s['reward']:.1f}") + print() + + +def cmd_sprint(args): + """Execute multiple issues from a sprint (delegates to /autonomous-sprint).""" + + print(f"\n{'='*60}") + print(f"AI Software Factory - Sprint Orchestrator") + print(f"{'='*60}\n") + + print("🔄 Delegating to /autonomous-sprint skill...") + print(f" JQL: {args.jql}\n") + + # TODO: This should invoke /autonomous-sprint skill with JQL + # For now, just show what would happen + + print("⚠️ Sprint orchestration via CLI not yet implemented.") + print(" Use /autonomous-sprint skill directly instead:") + print(f" /autonomous-sprint --jql \"{args.jql}\"") + + sys.exit(1) + + +def cmd_knowledge(args): + """Display repository knowledge.""" + + workspace_config = load_workspace_config() + factory_root = Path(__file__).parent.parent + + # Initialize knowledge engine + knowledge_root = factory_root / 'knowledge' + knowledge_engine = KnowledgeEngine(str(knowledge_root)) + + if args.list: + # List available repositories + repos = workspace_config.get('repositories', []) + print(f"\n{'='*60}") + print("Available Repositories") + print(f"{'='*60}\n") + for repo in repos: + print(f" - {repo['name']}: {repo.get('display_name', repo['name'])}") + print() + return + + if not args.repo: + print("❌ Error: --repo required (or use --list to see available repos)") + sys.exit(1) + + # Load knowledge for repository + print(f"\n{'='*60}") + print(f"Repository Knowledge: {args.repo}") + print(f"{'='*60}\n") + + knowledge = knowledge_engine.get_repository_knowledge(args.repo) + + for category, content in knowledge.items(): + print(f"\n## {category.upper()}") + print(f"{'-'*60}") + if content: + # Show first 500 chars + preview = content[:500] + ('...' if len(content) > 500 else '') + print(preview) + else: + print("(No content available)") + + print() + + +def cmd_test(args): + """Test harness components without execution.""" + + print(f"\n{'='*60}") + print(f"Orchestrator Component Test") + print(f"{'='*60}\n") + + workspace_config = load_workspace_config() + factory_root = Path(__file__).parent.parent + + # Test router + print("Testing Router...") + router = Router(workspace_config) + component = jira_mcp.get_issue_component(args.issue_key) + test_issue = { + 'key': args.issue_key, + 'components': [component] if component else [], + 'description': '', + 'labels': [], + } + repo = router.route_issue(test_issue) + print(f" ✅ Routed {args.issue_key} → {repo}\n") + + # Test knowledge engine + print("Testing Knowledge Engine...") + knowledge_root = factory_root / 'knowledge' + knowledge_engine = KnowledgeEngine(str(knowledge_root)) + knowledge = knowledge_engine.get_repository_knowledge(repo) + print(f" ✅ Loaded knowledge for {repo}:") + for category in knowledge.keys(): + length = len(knowledge[category]) + print(f" - {category}: {length} chars") + print() + + # Test foundations + print("Testing Foundations Standards...") + foundations = knowledge_engine.get_foundations_guidance('standards') + if foundations: + print(f" ✅ Loaded foundations standards: {len(foundations.get('standards', ''))} chars") + else: + print(f" ⚠️ No foundations standards found") + print() + + print("✅ All component tests passed!\n") + + +def _find_run_id(query: str) -> Optional[str]: + """ + Resolve a run_id or issue_key to a concrete run_id from the provenance index. + Returns the most recent matching run_id, or None. + """ + index = _PROVENANCE_DIR / "index.jsonl" + if not index.exists(): + return None + matches = [] + for line in index.read_text().splitlines(): + try: + s = json.loads(line) + except json.JSONDecodeError: + continue + if s.get("run_id") == query or s.get("issue_key") == query: + matches.append(s) + if not matches: + return None + return matches[-1]["run_id"] # most recent + + +def cmd_watch(args): + """Tail live events for an active run (by run_id or issue_key).""" + run_id = _find_run_id(args.target) + if not run_id: + print(f"❌ No run found for '{args.target}'") + sys.exit(1) + + run_file = _PROVENANCE_DIR / "runs" / f"{run_id}.jsonl" + if not run_file.exists(): + print(f"❌ Provenance file not found: {run_file}") + sys.exit(1) + + print(f"👀 Watching {run_id} (Ctrl-C to stop)\n") + with open(run_file) as fh: + # Print history + for line in fh: + _print_event(line) + # Tail + while True: + line = fh.readline() + if line: + _print_event(line) + if '"event": "run_end"' in line or '"event": "error"' in line: + print("\n✅ Run finished.") + break + else: + try: + time.sleep(0.5) + except KeyboardInterrupt: + break + + +def _print_event(line: str) -> None: + """Pretty-print a single provenance JSONL line.""" + try: + e = json.loads(line.strip()) + except json.JSONDecodeError: + return + ev = e.get("event", "?") + ts = e.get("timestamp", "")[:19].replace("T", " ") + if ev == "step": + icon = "✅" if e.get("success") else "❌" + cost = f" ${e.get('cost_usd', 0):.4f}" if e.get("cost_usd") else "" + print(f" {icon} [{ts}] step:{e.get('step')} {e.get('duration_ms', 0)/1000:.1f}s{cost}") + elif ev == "gate": + icon = "✅" if e.get("passed") else "❌" + flaky = " ⚡flaky" if e.get("flaky") else "" + print(f" {icon} [{ts}] gate:{e.get('gate')} att#{e.get('attempt')}{flaky}") + elif ev == "gate_loop_complete": + icon = "✅" if e.get("passed") else "⚠️ " + print(f" {icon} [{ts}] gate_loop: passed={e.get('passed')} attempts={e.get('attempts')}") + elif ev == "run_end": + print(f"\n 🏁 [{ts}] run_end: outcome={e.get('overall_outcome')} cost=${e.get('cost_usd',0):.4f}") + elif ev == "error": + print(f" ❌ [{ts}] error: {e.get('error', '')[:120]}") + + +def cmd_queue(args): + """Show active and queued runs.""" + # Try server API first + try: + import urllib.request + port = 8089 + with urllib.request.urlopen(f"http://localhost:{port}/api/queue", timeout=2) as r: + data = json.loads(r.read()) + active = data.get("active", []) + waiting = data.get("waiting", []) + except Exception: + active, waiting = [], [] + + if not active and not waiting: + print("No active or queued runs.") + return + + print(f"\n{'='*50}") + print(f"Active ({len(active)}):") + for r in active: + print(f" 🔄 {r.get('run_id','?')[:20]} {r.get('issue_key','?')} {r.get('repository','?')}") + + if waiting: + print(f"\nWaiting ({len(waiting)}):") + for r in waiting: + print(f" ⏳ {r.get('run_id','?')[:20]} {r.get('issue_key','?')} {r.get('repository','?')}") + print() + + +def cmd_cancel(args): + """Cancel an active run.""" + run_id = _find_run_id(args.target) or args.target + try: + import urllib.request + req = urllib.request.Request( + f"http://localhost:8089/api/runs/{run_id}/cancel", + method="POST", + ) + with urllib.request.urlopen(req, timeout=3) as r: + result = json.loads(r.read()) + print(f"✅ Cancelled: {result}") + except Exception as e: + print(f"❌ Could not cancel via server ({e}). Is the server running?") + print(" Start it with: python -m harness server") + + +def cmd_resume(args): + """Resume a run from its checkpoint.""" + run_id = args.run_id + index = _PROVENANCE_DIR / "index.jsonl" + if not index.exists(): + print("❌ No provenance data found.") + sys.exit(1) + + # Find the run in the index to get issue_key + repo + match = None + for line in index.read_text().splitlines(): + try: + s = json.loads(line) + if s.get("run_id") == run_id: + match = s + except json.JSONDecodeError: + continue + + if not match: + print(f"❌ Run '{run_id}' not found in provenance index.") + sys.exit(1) + + issue_key = match["issue_key"] + repository = match["repository"] + print(f"♻️ Resuming {run_id} ({issue_key} → {repository})") + + workspace_config = load_workspace_config() + h = Harness( + _FACTORY_ROOT, + workspace_config, + max_gate_attempts=getattr(args, "max_gate_attempts", 3), + ) + result = h.implement(issue_key, repository) + print(result.summary()) + sys.exit(0 if result.overall_outcome in ("success", "partial") else 1) + + +def cmd_circuit_breaker(args): + """Show or reset circuit breaker state.""" + from harness.circuit_breaker import CircuitBreaker + cb = CircuitBreaker(_PROVENANCE_DIR) + + if args.cb_action == "status": + states = cb.all_states() + print(f"\n{'='*48}") + print("Circuit Breaker Status") + print(f"{'='*48}") + for gate, st in states.items(): + state = st.get("state", "closed") + fails = st.get("consecutive_failures", 0) + icon = "🔴 OPEN " if state == "open" else "🟢 closed" + opened = st.get("opened_at", "")[:19] if state == "open" else "" + print(f" {icon} {gate:<14} failures={fails} {opened}") + print() + + elif args.cb_action == "reset": + gate = args.gate + cb.reset(gate) + print(f"✅ Circuit breaker reset: {gate} → closed (0 failures)") + + +def cmd_tui(args): + """Launch the terminal dashboard.""" + try: + from harness.tui import main as tui_main + tui_main() + except ImportError: + print("❌ rich is required for the TUI. Install with: pip install rich") + sys.exit(1) + + +def cmd_server(args): + """Start the observability server (foreground).""" + try: + import uvicorn + from harness.server import app + port = int(__import__("os").environ.get("HARNESS_SERVER_PORT", "8089")) + print(f"🌐 Dark Factory server → http://localhost:{port}") + print(f" Dashboard: http://localhost:{port}/") + print(f" Metrics: http://localhost:{port}/metrics") + uvicorn.run(app, host="0.0.0.0", port=port) + except ImportError: + print("❌ fastapi and uvicorn required: pip install fastapi uvicorn[standard]") + sys.exit(1) + + +def cmd_cost(args): + """Show cost breakdown by repo and step.""" + from harness.provenance import ProvenanceLogger + import datetime + + logger = ProvenanceLogger(_PROVENANCE_DIR) + days = getattr(args, "days", 7) + cutoff = datetime.datetime.utcnow() - datetime.timedelta(days=days) + index = _PROVENANCE_DIR / "index.jsonl" + + if not index.exists(): + print("No provenance data found.") + return + + by_repo: dict = {} + by_step: dict = {} + total_cost = 0.0 + + for line in index.read_text().splitlines(): + try: + s = json.loads(line) + except json.JSONDecodeError: + continue + ts = s.get("timestamp", "") + try: + run_ts = datetime.datetime.fromisoformat(ts.rstrip("Z")) + except ValueError: + continue + if run_ts < cutoff: + continue + + run_cost = s.get("cost_usd", 0.0) + repo = s.get("repository", "unknown") + by_repo[repo] = by_repo.get(repo, 0.0) + run_cost + total_cost += run_cost + + # Per-step breakdown from run events + for event in logger.read_run(s["run_id"]): + if event.get("event") not in ("step", "gate"): + continue + step_name = event.get("step") or event.get("gate") or "?" + step_cost = event.get("cost_usd", 0.0) + by_step[step_name] = by_step.get(step_name, 0.0) + step_cost + + print(f"\n{'='*50}") + print(f"Cost breakdown — last {days} day(s)") + print(f"{'='*50}") + print(f"\nTotal: ${total_cost:.4f}\n") + print("By repository:") + for repo, cost in sorted(by_repo.items(), key=lambda x: -x[1]): + print(f" {repo:<20} ${cost:.4f}") + print("\nBy step:") + for step, cost in sorted(by_step.items(), key=lambda x: -x[1])[:15]: + print(f" {step:<30} ${cost:.4f}") + print() + + +def cmd_runs(args): + """List recent runs with optional filters.""" + index = _PROVENANCE_DIR / "index.jsonl" + if not index.exists(): + print("No provenance data found.") + return + + repo_filter = getattr(args, "repo", None) + outcome_filter = getattr(args, "outcome", None) + limit = getattr(args, "limit", 20) + + summaries = [] + for line in index.read_text().splitlines(): + try: + s = json.loads(line) + except json.JSONDecodeError: + continue + if repo_filter and s.get("repository") != repo_filter: + continue + if outcome_filter and s.get("overall_outcome") != outcome_filter: + continue + summaries.append(s) + + summaries = summaries[-limit:] # most recent N + + print(f"\n{'='*80}") + print(f"{'Run ID':<24} {'Issue':<12} {'Repo':<16} {'Outcome':<8} {'Gates':<6} {'Cost':>8} {'Timestamp'}") + print(f"{'='*80}") + for s in reversed(summaries): + icon = {"success": "✅", "partial": "⚠️ ", "failed": "❌"}.get(s.get("overall_outcome", ""), "?") + run_id = s.get("run_id", "?")[:22] + issue = s.get("issue_key", "?")[:10] + repo = s.get("repository", "?")[:14] + gates = s.get("gate_attempts", 0) + cost = s.get("cost_usd", 0.0) + ts = s.get("timestamp", "")[:16].replace("T", " ") + print(f"{icon} {run_id:<22} {issue:<12} {repo:<16} {gates:<6} ${cost:>7.4f} {ts}") + print() + + +def main(): + """Main CLI entry point.""" + parser = argparse.ArgumentParser( + description='AI Software Factory - Workspace-level harness', + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + # Implement single issue (auto-route to repository) + python -m harness.cli implement ABI-123 + + # Implement in specific repository + python -m harness.cli implement ABI-123 --repo runtime + + # Implement across multiple repositories + python -m harness.cli multi-repo SDK-456 --repos sdk,runtime,runtime-ui + + # Execute sprint (delegates to /autonomous-sprint) + python -m harness.cli sprint --jql "sprint in openSprints()" + + # View repository knowledge + python -m harness.cli knowledge --repo runtime + python -m harness.cli knowledge --list + + # Test harness components + python -m harness.cli test ABI-123 + """ + ) + + subparsers = parser.add_subparsers(dest='command', help='Commands') + + # implement command + implement = subparsers.add_parser( + 'implement', + help='Implement single Jira issue in one repository' + ) + implement.add_argument('issue_key', help='Jira issue key (e.g., ABI-123)') + implement.add_argument( + '--repo', + help='Explicit repository name (default: auto-route)' + ) + implement.add_argument( + '--harness', + action='store_true', + default=False, + help=( + 'Use harness mode: call each skill individually, run gate loop ' + 'deterministically, log every step to provenance/. ' + 'Default: delegate everything to /autonomous-implement.' + ), + ) + implement.add_argument( + '--max-gate-attempts', + type=int, + default=3, + metavar='N', + help='Max gate loop retries in harness mode (default: 3)', + ) + implement.add_argument( + '--auto-merge', + action='store_true', + default=False, + help='Auto-merge PR via gh CLI when all gates pass (harness mode only)', + ) + implement.set_defaults(func=cmd_implement) + + # multi-repo command + multi = subparsers.add_parser( + 'multi-repo', + help='Implement issue across multiple repositories' + ) + multi.add_argument('issue_key', help='Jira issue key') + multi.add_argument( + '--repos', + help='Comma-separated repository names (default: auto-detect)' + ) + multi.set_defaults(func=cmd_multi_repo) + + # sprint command + sprint = subparsers.add_parser( + 'sprint', + help='Execute multiple issues from sprint' + ) + sprint.add_argument( + '--jql', + required=True, + help='JQL query for issues' + ) + sprint.set_defaults(func=cmd_sprint) + + # knowledge command + knowledge = subparsers.add_parser( + 'knowledge', + help='View repository knowledge' + ) + knowledge.add_argument('--repo', help='Repository name') + knowledge.add_argument('--list', action='store_true', help='List all repositories') + knowledge.set_defaults(func=cmd_knowledge) + + # test command + test = subparsers.add_parser( + 'test', + help='Test harness components' + ) + test.add_argument('issue_key', help='Jira issue key for testing') + test.set_defaults(func=cmd_test) + + # provenance command + prov = subparsers.add_parser( + 'provenance', + help='Query provenance logs and export RL datasets', + ) + prov_sub = prov.add_subparsers(dest='sub', help='Provenance sub-commands') + + prov_sub.add_parser('stats', help='Show aggregate statistics (success rate, gate failure rates)') + prov_sub.add_parser('runs', help='List recent runs with outcomes and rewards') + + prov_export = prov_sub.add_parser('export', help='Export RL training dataset (JSON)') + prov_export.add_argument( + '--output', + default='provenance/rl_dataset.json', + help='Output file path (default: provenance/rl_dataset.json)', + ) + prov.set_defaults(func=cmd_provenance) + + # watch command + watch = subparsers.add_parser('watch', help='Tail live events for a run') + watch.add_argument('target', help='run_id or issue_key (e.g. ABI-123 or run_20240726_abc)') + watch.set_defaults(func=cmd_watch) + + # queue command + subparsers.add_parser('queue', help='Show active and queued runs').set_defaults(func=cmd_queue) + + # cancel command + cancel = subparsers.add_parser('cancel', help='Cancel an active run') + cancel.add_argument('target', help='run_id or issue_key') + cancel.set_defaults(func=cmd_cancel) + + # resume command + resume = subparsers.add_parser('resume', help='Resume a run from its last checkpoint') + resume.add_argument('run_id', help='run_id to resume (e.g. run_20240726_abc123)') + resume.set_defaults(func=cmd_resume) + + # circuit-breaker command + cb = subparsers.add_parser('circuit-breaker', help='Show or reset gate circuit breaker state') + cb_sub = cb.add_subparsers(dest='cb_action') + cb_sub.add_parser('status', help='Show all circuit breaker states') + cb_reset = cb_sub.add_parser('reset', help='Manually reset a tripped circuit breaker') + cb_reset.add_argument('gate', choices=['linter', 'tests', 'evals', 'code-review']) + cb.set_defaults(func=cmd_circuit_breaker) + + # tui command + subparsers.add_parser('tui', help='Launch the terminal dashboard').set_defaults(func=cmd_tui) + + # server command + subparsers.add_parser('server', help='Start the observability server (foreground)').set_defaults(func=cmd_server) + + # cost command + cost_cmd = subparsers.add_parser('cost', help='Show cost breakdown by repo and step') + cost_cmd.add_argument('--days', type=int, default=7, help='Look-back window in days (default: 7)') + cost_cmd.set_defaults(func=cmd_cost) + + # runs command + runs_cmd = subparsers.add_parser('runs', help='List recent runs with optional filters') + runs_cmd.add_argument('--repo', help='Filter by repository name') + runs_cmd.add_argument('--outcome', choices=['success', 'partial', 'failed'], help='Filter by outcome') + runs_cmd.add_argument('--limit', type=int, default=20, help='Maximum rows to show (default: 20)') + runs_cmd.set_defaults(func=cmd_runs) + + # Parse and execute + args = parser.parse_args() + + if not args.command: + parser.print_help() + sys.exit(1) + + # Execute command + args.func(args) + + +if __name__ == '__main__': + main() diff --git a/harness/dashboard/index.html b/harness/dashboard/index.html new file mode 100644 index 0000000..ab77437 --- /dev/null +++ b/harness/dashboard/index.html @@ -0,0 +1,1135 @@ + + + + + + Dark Factory + + + +
+ + +
+ 🏭 Dark Factory +
+ Active: — + Queue: — + Success (7d): — + Cost today: — + Runs (7d): — +
+ Live +
+ + +
+ ⚠ Circuit breaker OPEN: + — gate is disabled until manually reset. +
+ + +
+ + +
+
+ Active Runs + 0 +
+
+
✅No active runs
+
+
+ + +
+
+ Gate Health (7-day) +
+
+
📊No data
+
+
+ + +
+
+
+ Recent Runs +
+ + + +
+
+
+ + + + + + + + + + + + + + + + + +
Run IDIssueRepoDurationGate Att.CostTimestampStatus
Loading...
+
+
+
+ +
+ +
+ + +
+
+
+
+
Run Detail
+
+
+ +
+
+
Loading...
+
+
+
+ + + + diff --git a/orchestrator/executor.py b/harness/executor.py similarity index 70% rename from orchestrator/executor.py rename to harness/executor.py index 39a3040..a0faecb 100644 --- a/orchestrator/executor.py +++ b/harness/executor.py @@ -16,8 +16,6 @@ from dataclasses import dataclass from pathlib import Path import subprocess -import json -import os import tempfile import time @@ -259,36 +257,28 @@ def _invoke_autonomous_implement( Returns: Dictionary with execution results """ - # Create knowledge context file + # Create knowledge context file inside repo_path so subprocess claude can read it context_file = self._create_knowledge_context_file( knowledge_context=knowledge_context, foundations_standards=foundations_standards, - repo_config=repo_config + repo_config=repo_config, + repo_path=repo_path, ) - try: - # Method 1: Use Skill tool directly (if available in this context) - # This is the ideal approach when running inside Claude Code - if self._is_running_in_claude_code(): - return self._invoke_skill_via_tool(issue_key, context_file, repo_path) - - # Method 2: Subprocess call to claude CLI - # This works when orchestrator runs as standalone script - return self._invoke_skill_via_subprocess(issue_key, context_file, repo_path) - - finally: - # Clean up temp file - if context_file.exists(): - context_file.unlink() + return self._invoke_skill_via_subprocess(issue_key, context_file, repo_path) def _create_knowledge_context_file( self, knowledge_context: Dict[str, str], foundations_standards: str, - repo_config: Dict + repo_config: Dict, + repo_path: Path, ) -> Path: """ - Create temporary file with knowledge context for skill. + Create knowledge context file inside the target repo directory. + + Placing it inside repo_path ensures the subprocess claude session can + read it without an out-of-allowed-directory permission prompt. Args: knowledge_context: Repository knowledge @@ -299,7 +289,7 @@ def _create_knowledge_context_file( Path to temporary context file """ context = f"""# Repository Knowledge Context -# This context is automatically injected by the orchestrator +# This context is automatically injected by the harness ## Repository: {repo_config['name']} **Display Name:** {repo_config.get('display_name', repo_config['name'])} @@ -349,132 +339,93 @@ def _create_knowledge_context_file( 6. Run gitleaks to ensure no secrets """ - # Write to temporary file - with tempfile.NamedTemporaryFile( - mode='w', - suffix='.md', - prefix='knowledge_context_', - delete=False - ) as f: - f.write(context) - return Path(f.name) - - def _invoke_skill_via_tool( + # Write into the repo dir so the subprocess claude session can read it + # without hitting an out-of-allowed-directory permission prompt. + # _invoke_skill_via_subprocess deletes it after claude exits. + context_file = repo_path / f'.knowledge_context_{repo_config["name"]}.md' + context_file.write_text(context) + return context_file + + def _invoke_skill_via_subprocess( self, issue_key: str, context_file: Path, repo_path: Path ) -> Dict: """ - Invoke skill using the Skill tool (when running in Claude Code). + Invoke /autonomous-implement skill by shelling out to the claude CLI. + + Runs claude headlessly (-p) with the factory plugin dir so all skills + are available, then passes the skill invocation as the initial prompt. Args: issue_key: Jira issue key - context_file: Path to knowledge context file - repo_path: Repository path + context_file: Path to knowledge context file (persists until skill completes) + repo_path: Repository path (used as cwd for the claude process) Returns: Execution result dictionary """ - # This would use the Skill tool if we're running inside Claude Code - # For now, we'll use subprocess as fallback - return self._invoke_skill_via_subprocess(issue_key, context_file, repo_path) + prompt = f"/autonomous-implement {issue_key} --context-file {context_file}" - def _invoke_skill_via_subprocess( - self, - issue_key: str, - context_file: Path, - repo_path: Path - ) -> Dict: - """ - Create instructions file for manual skill invocation. - - When orchestrator runs standalone (not in Claude Code), it cannot - directly invoke skills. Instead, it: - 1. Prepares knowledge context - 2. Creates instruction file - 3. Prints commands for user to run in Claude Code + cmd = [ + 'claude', + '--plugin-dir', str(self.factory_root), + '--dangerously-skip-permissions', + '-p', prompt, + ] - Args: - issue_key: Jira issue key - context_file: Path to knowledge context file - repo_path: Repository path + print(f"\n🚀 Launching claude to implement {issue_key} in {repo_path.name}...") + print(f" Plugin: {self.factory_root}") + print(f" Context: {context_file}\n") - Returns: - Execution result dictionary with instructions - """ - # Create instructions file - instructions_file = Path(tempfile.gettempdir()) / f"orchestrator_instructions_{issue_key}.sh" - - instructions = f"""#!/bin/bash -# Orchestrator Execution Instructions for {issue_key} -# Generated: {time.strftime('%Y-%m-%d %H:%M:%S')} - -# Repository: {repo_path} -# Knowledge Context: {context_file} - -echo "============================================================" -echo "Orchestrator: Ready to implement {issue_key}" -echo "============================================================" -echo "" -echo "Repository: {repo_path}" -echo "Knowledge context prepared at: {context_file}" -echo "" -echo "To execute this implementation, run:" -echo "" -echo " cd {repo_path}" -echo " claude --plugin-dir {self.factory_root}/.claude/plugins/em-software-factory" -echo "" -echo "Then in Claude Code, run:" -echo "" -echo " /autonomous-implement {issue_key} --context-file {context_file}" -echo "" -echo "============================================================" - -# For automated execution in Claude Code environment: -# cd {repo_path} -# /autonomous-implement {issue_key} --context-file {context_file} -""" + try: + result = subprocess.run( + cmd, + cwd=str(repo_path), + text=True, + timeout=3600, # 1-hour ceiling for large issues + ) - with open(instructions_file, 'w') as f: - f.write(instructions) - - instructions_file.chmod(0o755) # Make executable - - # Print instructions to console - print("\n" + "="*60) - print(f"📋 Implementation Instructions for {issue_key}") - print("="*60) - print(f"\n✅ Knowledge context prepared: {context_file}") - print(f"✅ Repository: {repo_path}") - print(f"\n📝 Instructions saved to: {instructions_file}") - print("\n" + "-"*60) - print("To execute, run these commands:") - print("-"*60) - print(f"\n1. Start Claude Code with plugin:") - print(f" cd {self.factory_root}") - print(f" claude --plugin-dir .claude/plugins/em-software-factory") - print(f"\n2. Navigate to repository:") - print(f" cd {repo_path}") - print(f"\n3. Run autonomous-implement:") - print(f" /autonomous-implement {issue_key} --context-file {context_file}") - print("\n" + "="*60 + "\n") - - return { - 'success': True, - 'output': f'Instructions created at {instructions_file}', - 'pr_url': None, - 'branch_name': None, - 'error': None, - 'instructions_file': str(instructions_file), - 'context_file': str(context_file), - 'message': 'Manual execution required - see instructions above' - } - - def _is_running_in_claude_code(self) -> bool: - """Check if running inside Claude Code environment.""" - # Simple heuristic - check for Claude Code environment markers - return 'CLAUDE_CODE' in os.environ or 'ANTHROPIC_API_KEY' in os.environ + success = result.returncode == 0 + + if not success: + return { + 'success': False, + 'error': f'claude exited with code {result.returncode}', + 'pr_url': None, + 'branch_name': None, + 'output': '', + } + + return { + 'success': True, + 'output': 'autonomous-implement completed', + 'pr_url': None, + 'branch_name': None, + 'error': None, + } + + except FileNotFoundError: + return { + 'success': False, + 'error': 'claude CLI not found — ensure it is on PATH (brew install claude-code)', + 'pr_url': None, + 'branch_name': None, + 'output': '', + } + except subprocess.TimeoutExpired: + return { + 'success': False, + 'error': 'claude timed out after 1 hour', + 'pr_url': None, + 'branch_name': None, + 'output': '', + } + finally: + # Clean up context file now that claude has finished + if context_file.exists(): + context_file.unlink() def _get_repo_config(self, repository: str) -> Dict: """Get repository configuration from workspace config.""" diff --git a/harness/harness.py b/harness/harness.py new file mode 100644 index 0000000..e146980 --- /dev/null +++ b/harness/harness.py @@ -0,0 +1,619 @@ +""" +Harness — owns the full implementation loop, calling individual skills as steps. + +Integrated with: + - scrubber : secrets removed from output before provenance writes + - locks : per-repo file lock prevents concurrent runs on the same repo + - checkpoint : survives mid-run crashes; resumes from the last completed step + - circuit_breaker : skips a gate that has failed N consecutive cross-run times + - watchdog : background thread alerts/kills hung steps + - server : registers each run for live observability (if server is up) +""" + +from __future__ import annotations + +import json +import re +import subprocess +import time +import uuid +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + +from .checkpoint import Checkpoint +from .circuit_breaker import CircuitBreaker, CircuitOpenError +from .locks import LockError, RepoLock +from .provenance import GateRecord, ProvenanceLogger, StepRecord +from .scrubber import scrub +from .watchdog import Watchdog + +# Server integration is optional. We probe lazily to avoid import-time noise +# when fastapi/uvicorn are not installed. +def _server_call(fn_name: str, *args, **kwargs) -> None: + """Call a server function if the server module is importable. Silent no-op otherwise.""" + try: + import importlib, io, contextlib + with contextlib.redirect_stderr(io.StringIO()): # suppress install-hint prints + srv = importlib.import_module(".server", package="harness") + getattr(srv, fn_name)(*args, **kwargs) + except Exception: + pass + + +# --------------------------------------------------------------------------- +# Result type +# --------------------------------------------------------------------------- + +@dataclass +class HarnessResult: + run_id: str + issue_key: str + repo: str + overall_outcome: str # "success" | "partial" | "failed" + gate_attempts: int + steps: List[StepRecord] + gate_results: List[GateRecord] + pr_url: Optional[str] + duration_ms: float + cost_usd: float + provenance_path: str + + def summary(self) -> str: + outcome_icon = { + "success": "✅ SUCCESS", + "partial": "⚠️ PARTIAL", + "failed": "❌ FAILED", + }.get(self.overall_outcome, self.overall_outcome) + + lines = [ + f"\n{'='*64}", + f"Harness Result: {self.issue_key} [{self.repo}]", + f"{'='*64}", + f"Outcome : {outcome_icon}", + f"Duration : {self.duration_ms/1000:.1f}s", + f"Gate loop : {self.gate_attempts} attempt(s)", + f"Cost : ${self.cost_usd:.4f}", + "", + "Steps:", + ] + for s in self.steps: + icon = "✅" if s.success else "❌" + cost = f" ${s.cost_usd:.4f}" if s.cost_usd else "" + lines.append(f" {icon} {s.step} ({s.duration_ms/1000:.1f}s){cost}") + if s.error: + lines.append(f" ↳ {s.error}") + + if self.gate_results: + lines += ["", "Gates (final attempt):"] + seen: Dict[str, GateRecord] = {} + for g in self.gate_results: + seen[g.gate] = g + for gate, g in seen.items(): + icon = "✅" if g.passed else "❌" + lines.append(f" {icon} {gate}") + + if self.pr_url: + lines.append(f"\nPR: {self.pr_url}") + + lines += [f"\nProvenance: {self.provenance_path}", f"{'='*64}\n"] + return "\n".join(lines) + + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- + +_GATE_ORDER = ("linter", "tests", "evals", "code-review") + +_GATE_PROMPTS = { + "linter": "/run-linter --output {result_file}", + "tests": "/run-tests --output {result_file}", + "evals": "/run-evals {issue_key} --output {result_file}", + "code-review": "/run-code-review --output {result_file}", +} + +_RESULT_FILES = {gate: f".harness-results/{gate}.json" for gate in _GATE_ORDER} + +# Steps that produce code output (watch for secrets) +_SCRUB_STEPS = {"research", "plan", "implement", "create-pr"} + + +# --------------------------------------------------------------------------- +# Harness +# --------------------------------------------------------------------------- + +class Harness: + """ + Deterministic implementation loop with per-step provenance, reliability + primitives, and live observability via the embedded server. + """ + + def __init__( + self, + factory_root: Path, + workspace_config: Dict[str, Any], + *, + max_gate_attempts: int = 3, + auto_merge: bool = False, + provenance_dir: Optional[Path] = None, + skill_timeout: int = 3600, + knowledge_engine=None, + ): + self.factory_root = Path(factory_root) + self.workspace_config = workspace_config + self.workspace_root = Path(workspace_config["workspace"]["root"]) + self.max_gate_attempts = max_gate_attempts + self.auto_merge = auto_merge + self.skill_timeout = skill_timeout + + if knowledge_engine is None: + from .knowledge import KnowledgeEngine + knowledge_engine = KnowledgeEngine(str(self.factory_root / "knowledge")) + self._knowledge = knowledge_engine + + prov_dir = provenance_dir or (self.factory_root / "provenance") + self.provenance = ProvenanceLogger(prov_dir) + self.circuit_breaker = CircuitBreaker(prov_dir) + self._prov_dir = prov_dir + + # ------------------------------------------------------------------ + # Public API + # ------------------------------------------------------------------ + + def implement(self, issue_key: str, repository: str) -> HarnessResult: + """ + Run the full implementation loop for one issue in one repository. + + Steps + ----- + 1. research-codebase (skipped if checkpoint says done) + 2. create-plan + 3. implement-plan + 4. gate loop — linter → tests → evals → code-review (up to N retries) + circuit breaker skips any gate that is "open" + 5. create-pr + 6. auto-merge (if policy allows and all gates passed) + """ + run_id = f"run_{int(time.time())}_{uuid.uuid4().hex[:8]}" + t_start = time.time() + + repo_cfg = self._repo_config(repository) + repo_path = self.workspace_root / repo_cfg["path"] + + if not repo_path.exists(): + raise FileNotFoundError( + f"Repository path not found: {repo_path}\n" + f"Clone the repo or update workspace.yaml." + ) + + (repo_path / ".harness-results").mkdir(exist_ok=True) + + # Ensure the observability server is running (silent if fastapi not installed) + _server_call("ensure_server_running", self._prov_dir, self.factory_root) # server is optional + + knowledge = self._knowledge.get_repository_knowledge(repository) + foundations = self._knowledge.get_foundations_guidance("standards") + ctx_file = self._write_knowledge_context(knowledge, foundations, repo_cfg, repo_path) + + checkpoint = Checkpoint(repo_path) + # Resume: if there's an existing checkpoint for THIS issue, reuse the run_id + existing = checkpoint.read() + if existing and existing.get("issue_key") == issue_key: + run_id = existing["run_id"] + print(f"\n♻️ Resuming {run_id} (completed: {existing.get('completed_steps', [])})") + elif existing: + print(f"\n⚠️ Stale checkpoint for {existing.get('issue_key')} found — clearing") + checkpoint.clear() + + steps: List[StepRecord] = [] + gate_results: List[GateRecord] = [] + pr_url: Optional[str] = None + overall_outcome = "failed" + gate_attempt = 0 + watchdog: Optional[Watchdog] = None + + try: + # Acquire exclusive lock on the repo — waits up to 5 min if busy + with RepoLock(repo_path, run_id=run_id, issue_key=issue_key): + self.provenance.start_run(run_id, issue_key, repository, str(repo_path)) + print(f"\n🏭 Harness run {run_id} | {issue_key} → {repository}") + + _server_call("register_run", run_id, issue_key, repository) + + # Start watchdog + watchdog = Watchdog( + run_id=run_id, + on_warn=self._on_warn, + on_kill=self._on_kill, + ) + watchdog.start() + + # ── Step 1: Research ────────────────────────────────── + if not checkpoint.should_skip("research"): + step = self._skill( + f"/research-codebase {issue_key} --context-file {ctx_file}", + repo_path, run_id, "research", watchdog=watchdog, + ) + steps.append(step) + self.provenance.log_step(run_id, step) + if step.success: + checkpoint.write(run_id, issue_key, ["research"]) + + # ── Step 2: Plan ────────────────────────────────────── + if not checkpoint.should_skip("plan"): + step = self._skill( + f"/create-plan {issue_key} --context-file {ctx_file}", + repo_path, run_id, "plan", watchdog=watchdog, + ) + steps.append(step) + self.provenance.log_step(run_id, step) + if not step.success: + raise RuntimeError(f"Planning failed: {step.error}") + checkpoint.write(run_id, issue_key, checkpoint.completed_steps() + ["plan"]) + + # ── Step 3: Implement ───────────────────────────────── + if not checkpoint.should_skip("implement"): + step = self._skill( + f"/implement-plan {issue_key}", + repo_path, run_id, "implement", watchdog=watchdog, + ) + steps.append(step) + self.provenance.log_step(run_id, step) + if not step.success: + raise RuntimeError(f"Implementation failed: {step.error}") + checkpoint.write(run_id, issue_key, checkpoint.completed_steps() + ["implement"]) + + # ── Step 4: Gate loop ───────────────────────────────── + gates_passed = False + while gate_attempt < self.max_gate_attempts: + gate_attempt += 1 + print(f"\n🔄 Gate loop — attempt {gate_attempt}/{self.max_gate_attempts}") + failures: List[GateRecord] = [] + + for gate_name in _GATE_ORDER: + # Circuit breaker check + try: + self.circuit_breaker.check(gate_name) + except CircuitOpenError as ce: + print(f" ⚡ {gate_name} — circuit open, skipping ({ce})") + self.provenance.log_error(run_id, f"circuit_open:{gate_name}:{ce}") + continue + + gr = self._gate(gate_name, issue_key, repo_path, run_id, gate_attempt, watchdog) + gate_results.append(gr) + self.provenance.log_gate(run_id, gr) + + if gr.passed: + self.circuit_breaker.record_success(gate_name) + print(f" ✅ {gate_name}") + else: + self.circuit_breaker.record_failure(gate_name, gr.error or "") + print(f" ❌ {gate_name}") + failures.append(gr) + + if not failures: + gates_passed = True + self.provenance.log_gate_loop(run_id, passed=True, attempts=gate_attempt) + print("✅ All gates passed") + break + + if gate_attempt < self.max_gate_attempts: + fix_ctx = self._write_failure_context(failures, repo_path, gate_attempt) + step = self._skill( + f"/fix-failures --failures-file {fix_ctx}", + repo_path, run_id, f"fix.attempt{gate_attempt}", watchdog=watchdog, + ) + steps.append(step) + self.provenance.log_step(run_id, step) + else: + self.provenance.log_gate_loop( + run_id, passed=False, attempts=gate_attempt, failures=failures + ) + + # ── Step 5: Create PR ───────────────────────────────── + label_flag = "" if gates_passed else "--label NEEDS-REVIEW" + step = self._skill( + f"/create-pr {label_flag}".strip(), + repo_path, run_id, "create-pr", watchdog=watchdog, + ) + steps.append(step) + self.provenance.log_step(run_id, step) + + # ── Step 6: Auto-merge ──────────────────────────────── + if gates_passed and self.auto_merge: + merge_step = self._merge(pr_url, repo_path) + steps.append(merge_step) + self.provenance.log_step(run_id, merge_step) + + overall_outcome = "success" if gates_passed else "partial" + + except LockError as le: + self.provenance.log_error(run_id, f"lock_timeout:{le}") + print(f"\n⏳ Could not acquire repo lock: {le}") + except Exception as exc: + self.provenance.log_error(run_id, str(exc)) + print(f"\n❌ Harness error: {exc}") + finally: + if watchdog: + watchdog.stop() + checkpoint.clear() + ctx_file.unlink(missing_ok=True) + _server_call("complete_run", run_id, overall_outcome) + + duration_ms = (time.time() - t_start) * 1000 + total_cost = sum(s.cost_usd for s in steps) + sum(g.cost_usd for g in gate_results) + + self.provenance.finish_run( + run_id=run_id, + issue_key=issue_key, + repository=repository, + overall_outcome=overall_outcome, + gate_attempts=gate_attempt, + steps=steps, + gate_results=gate_results, + pr_url=pr_url, + duration_ms=duration_ms, + ) + + return HarnessResult( + run_id=run_id, + issue_key=issue_key, + repo=repository, + overall_outcome=overall_outcome, + gate_attempts=gate_attempt, + steps=steps, + gate_results=gate_results, + pr_url=pr_url, + duration_ms=duration_ms, + cost_usd=round(total_cost, 6), + provenance_path=str(self.provenance._run_path(run_id)), + ) + + # ------------------------------------------------------------------ + # Skill invocation + # ------------------------------------------------------------------ + + def _skill( + self, + prompt: str, + repo_path: Path, + run_id: str, + step_name: str, + *, + watchdog: Optional[Watchdog] = None, + ) -> StepRecord: + """ + Call a skill via `claude -p` subprocess. + + Uses Popen so the watchdog can kill the process if it hangs. + Parses token/cost from stdout. Scrubs secrets before storing output. + """ + t0 = time.time() + print(f" ▶ {step_name}") + + cmd = [ + "claude", + "--plugin-dir", str(self.factory_root), + "--dangerously-skip-permissions", + "-p", prompt, + ] + try: + proc = subprocess.Popen( + cmd, + cwd=str(repo_path), + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + if watchdog: + watchdog.set_step(step_name, proc) + + try: + stdout, stderr = proc.communicate(timeout=self.skill_timeout) + except subprocess.TimeoutExpired: + proc.kill() + stdout, stderr = proc.communicate() + return StepRecord( + step=step_name, success=False, + duration_ms=(time.time() - t0) * 1000, + error=f"Timed out after {self.skill_timeout}s", + ) + + duration_ms = (time.time() - t0) * 1000 + success = proc.returncode == 0 + tokens_in, tokens_out, cost_usd = _parse_tokens(stdout or "") + + # Scrub secrets from output before storing + output_text = stdout or "" + if step_name in _SCRUB_STEPS or step_name.startswith("fix."): + output_text = scrub(output_text) + + return StepRecord( + step=step_name, + success=success, + duration_ms=duration_ms, + output_preview=output_text, + error=scrub(stderr) if (stderr and not success) else None, + tokens_in=tokens_in, + tokens_out=tokens_out, + cost_usd=cost_usd, + ) + + except FileNotFoundError: + return StepRecord( + step=step_name, success=False, duration_ms=0, + error="claude CLI not found — install with: brew install claude-code", + ) + + # ------------------------------------------------------------------ + # Gate execution + # ------------------------------------------------------------------ + + def _gate( + self, + gate: str, + issue_key: str, + repo_path: Path, + run_id: str, + attempt: int, + watchdog: Optional[Watchdog], + ) -> GateRecord: + """Run a single gate skill and read its structured JSON result file.""" + t0 = time.time() + result_file = repo_path / _RESULT_FILES[gate] + result_file.unlink(missing_ok=True) + + prompt = _GATE_PROMPTS[gate].format(issue_key=issue_key, result_file=result_file) + step = self._skill(prompt, repo_path, run_id, f"gate.{gate}.attempt{attempt}", watchdog=watchdog) + + duration_ms = (time.time() - t0) * 1000 + + if result_file.exists(): + try: + outputs: Dict[str, Any] = json.loads(result_file.read_text()) + passed = bool(outputs.get("passed", False)) + except json.JSONDecodeError: + outputs = {"raw": step.output_preview[:200]} + passed = step.success + else: + outputs = {"raw": step.output_preview[:200]} + passed = step.success + + return GateRecord( + gate=gate, + passed=passed, + attempt=attempt, + outputs=outputs, + duration_ms=duration_ms, + error=step.error, + tokens_in=step.tokens_in, + tokens_out=step.tokens_out, + cost_usd=step.cost_usd, + ) + + # ------------------------------------------------------------------ + # Merge + # ------------------------------------------------------------------ + + def _merge(self, pr_url: Optional[str], repo_path: Path) -> StepRecord: + t0 = time.time() + if not pr_url: + return StepRecord(step="auto-merge", success=False, duration_ms=0, + error="No PR URL available for auto-merge") + try: + proc = subprocess.run( + ["gh", "pr", "merge", "--squash", "--auto", pr_url], + cwd=str(repo_path), text=True, capture_output=True, timeout=120, + ) + success = proc.returncode == 0 + return StepRecord( + step="auto-merge", success=success, + duration_ms=(time.time() - t0) * 1000, + output_preview=proc.stdout, + error=proc.stderr if not success else None, + ) + except FileNotFoundError: + return StepRecord(step="auto-merge", success=False, duration_ms=0, + error="gh CLI not found") + + # ------------------------------------------------------------------ + # Watchdog callbacks + # ------------------------------------------------------------------ + + def _on_warn(self, step: str, elapsed: float) -> None: + mins = elapsed / 60 + print(f"\n⚠️ Watchdog: step '{step}' has been running {mins:.0f}m — still in progress") + + def _on_kill(self, step: str, elapsed: float) -> None: + mins = elapsed / 60 + print(f"\n🔴 Watchdog: killing step '{step}' after {mins:.0f}m (hard limit reached)") + + # ------------------------------------------------------------------ + # Helpers + # ------------------------------------------------------------------ + + def _write_failure_context(self, failures: List[GateRecord], repo_path: Path, attempt: int) -> Path: + ctx = repo_path / f".harness-results/failures-{attempt}.json" + ctx.write_text(json.dumps( + [{"gate": f.gate, "outputs": f.outputs} for f in failures], indent=2 + )) + return ctx + + def _write_knowledge_context( + self, + knowledge: Dict[str, str], + foundations: Dict[str, str], + repo_cfg: Dict[str, Any], + repo_path: Path, + ) -> Path: + content = f"""# Repository Knowledge Context +# Auto-generated by the Harness — do not edit + +## Repository: {repo_cfg['name']} +**Language:** {repo_cfg.get('language', 'unknown')} +**Build System:** {repo_cfg.get('build_system', 'unknown')} + +## Architecture +{knowledge.get('architecture', 'No architecture documentation available.')} + +## Coding Patterns +{knowledge.get('patterns', 'No coding patterns documented.')} + +## Conventions +{knowledge.get('conventions', 'No conventions documented.')} + +## Dependencies +{knowledge.get('dependencies', 'No dependency information available.')} + +## Foundations Standards +{foundations.get('standards', '')} +""" + ctx_path = repo_path / f".knowledge_context_{repo_cfg['name']}.md" + ctx_path.write_text(content) + return ctx_path + + def _repo_config(self, repository: str) -> Dict[str, Any]: + for repo in self.workspace_config.get("repositories", []): + if repo["name"] == repository: + return repo + raise ValueError(f"Repository '{repository}' not found in workspace.yaml") + + +# --------------------------------------------------------------------------- +# Token / cost parsing +# --------------------------------------------------------------------------- + +# Claude CLI emits token/cost info in various formats depending on version. +# Try multiple patterns; return zeros if none match. +_TOKEN_PATTERNS = [ + # "Tokens: 1,234 input, 567 output" + re.compile(r"Tokens?:\s*([\d,]+)\s*input[,\s]+([\d,]+)\s*output", re.I), + # "1,234 in / 567 out" + re.compile(r"([\d,]+)\s*in\s*/\s*([\d,]+)\s*out", re.I), + # "input_tokens: 1234" / "output_tokens: 567" + re.compile(r"input_tokens[\":\s]+([\d,]+).*?output_tokens[\":\s]+([\d,]+)", re.I | re.S), +] +_COST_PATTERN = re.compile(r"Cost[:\s]+\$?([\d.]+)", re.I) + + +def _parse_tokens(text: str) -> Tuple[int, int, float]: + """Parse tokens_in, tokens_out, cost_usd from claude CLI stdout.""" + tokens_in = tokens_out = 0 + cost_usd = 0.0 + + for pat in _TOKEN_PATTERNS: + m = pat.search(text) + if m: + tokens_in = int(m.group(1).replace(",", "")) + tokens_out = int(m.group(2).replace(",", "")) + break + + m = _COST_PATTERN.search(text) + if m: + try: + cost_usd = float(m.group(1)) + except ValueError: + pass + + return tokens_in, tokens_out, cost_usd diff --git a/orchestrator/jira_mcp.py b/harness/jira_mcp.py similarity index 88% rename from orchestrator/jira_mcp.py rename to harness/jira_mcp.py index 109a7d5..a30159a 100644 --- a/orchestrator/jira_mcp.py +++ b/harness/jira_mcp.py @@ -2,7 +2,7 @@ Jira Integration for Orchestrator Provides functions to extract Jira component and route to correct repository. -The orchestrator delegates actual Jira fetching to skills (which use MCP). +The harness delegates actual Jira fetching to skills (which use MCP). """ from typing import Dict, Optional @@ -12,7 +12,7 @@ def get_issue_component(issue_key: str) -> Optional[str]: """ Infer Jira component from issue key prefix for routing. - The orchestrator doesn't fetch full issue data - it just needs to know + The harness doesn't fetch full issue data - it just needs to know which repository to route to. The actual issue fetching happens in skills via MCP (mcp__atlassian__jira_get_issue). @@ -67,7 +67,7 @@ def get_repository_for_issue(issue_key: str, component_mapping: Dict[str, str]) # Integration note: -# The orchestrator's job is to route issues to the correct repository +# The harness's job is to route issues to the correct repository # and prepare knowledge context. The actual Jira data fetching happens # in the skills themselves via MCP: # @@ -77,5 +77,5 @@ def get_repository_for_issue(issue_key: str, component_mapping: Dict[str, str]) # fields: ['summary', 'description', 'issuetype', 'status'] # }) # -# This separation keeps the orchestrator lightweight and lets skills +# This separation keeps the harness lightweight and lets skills # handle the full Jira integration with all available MCP tools. diff --git a/orchestrator/knowledge.py b/harness/knowledge.py similarity index 99% rename from orchestrator/knowledge.py rename to harness/knowledge.py index 8aa88a4..82970b1 100644 --- a/orchestrator/knowledge.py +++ b/harness/knowledge.py @@ -2,7 +2,7 @@ Knowledge Engine - Retrieves semantic knowledge for planning and implementation. The knowledge engine loads architecture, patterns, conventions, and decisions -from centralized knowledge packs and provides them to the orchestrator. +from centralized knowledge packs and provides them to the harness. """ from pathlib import Path diff --git a/harness/locks.py b/harness/locks.py new file mode 100644 index 0000000..9b256a5 --- /dev/null +++ b/harness/locks.py @@ -0,0 +1,181 @@ +"""Per-repo file lock to prevent concurrent harness runs on the same repository. + +Uses POSIX ``fcntl.flock`` for kernel-enforced mutual exclusion. The lock +file lives at ``{repo_path}/.harness-results/repo.lock``; its JSON content +identifies the current holder so operators can inspect stale locks. + +Typical usage:: + + from pathlib import Path + from harness.locks import RepoLock + + with RepoLock(repo_path, run_id="run_20240726_abc123", issue_key="ABI-42"): + # safe to operate on the repo + ... +""" + +import fcntl +import json +import os +import time +from datetime import datetime, timezone +from pathlib import Path +from typing import Optional + + +class LockError(Exception): + """Raised when a lock cannot be acquired within the configured timeout.""" + + +class RepoLock: + """Exclusive per-repo lock implemented with ``fcntl.flock``. + + Parameters + ---------- + repo_path: + Root of the repository being operated on. + run_id: + Unique identifier for this harness run (written into the lock record). + issue_key: + Jira / issue key associated with this run (written into the lock record). + timeout: + Maximum seconds to wait before raising :class:`LockError`. Default + is 300 seconds (5 minutes). + + Lock record (JSON written inside the lock file):: + + { + "run_id": "run_20240726_abc123", + "issue_key": "ABI-42", + "started_at": "2024-07-26T12:00:00Z", + "pid": 12345 + } + """ + + _POLL_INTERVAL = 5 # seconds between acquisition attempts + _LOCK_SUBDIR = ".harness-results" + _LOCK_FILENAME = "repo.lock" + + def __init__( + self, + repo_path: Path, + run_id: str, + issue_key: str, + timeout: int = 300, + ) -> None: + self._repo_path = Path(repo_path) + self._run_id = run_id + self._issue_key = issue_key + self._timeout = timeout + self._lock_path = self._repo_path / self._LOCK_SUBDIR / self._LOCK_FILENAME + self._lock_fh = None # file handle kept open while lock is held + + # ------------------------------------------------------------------ + # Context manager protocol + # ------------------------------------------------------------------ + + def __enter__(self) -> "RepoLock": + self._acquire() + return self + + def __exit__(self, exc_type, exc_val, exc_tb) -> None: + self._release() + return False # do not suppress exceptions + + # ------------------------------------------------------------------ + # Internal helpers + # ------------------------------------------------------------------ + + def _ensure_dir(self) -> None: + self._lock_path.parent.mkdir(parents=True, exist_ok=True) + + def _acquire(self) -> None: + """Block until the lock is acquired or *timeout* is exceeded.""" + self._ensure_dir() + deadline = time.monotonic() + self._timeout + attempt = 0 + + while True: + fh = open(self._lock_path, "a+") # open for append+read; creates if missing + try: + fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB) + # Lock acquired — write our record and keep the file open. + fh.seek(0) + fh.truncate() + record = { + "run_id": self._run_id, + "issue_key": self._issue_key, + "started_at": datetime.now(timezone.utc).isoformat(), + "pid": os.getpid(), + } + fh.write(json.dumps(record, indent=2)) + fh.flush() + self._lock_fh = fh + print( + f"[lock] acquired for {self._issue_key} (run {self._run_id})" + f" on {self._repo_path}" + ) + return + except BlockingIOError: + # Lock is held by another process. + fh.close() + remaining = deadline - time.monotonic() + if remaining <= 0: + raise LockError( + f"Could not acquire lock on {self._lock_path} " + f"within {self._timeout}s. " + f"Current holder: {read_lock_record(self._repo_path)}" + ) + attempt += 1 + holder = read_lock_record(self._repo_path) + holder_info = ( + f"held by run={holder['run_id']} pid={holder['pid']}" + if holder + else "held (unknown holder)" + ) + print( + f"[lock] waiting for repo lock ({holder_info}); " + f"queue position ~{attempt}, " + f"{int(remaining)}s remaining …" + ) + time.sleep(min(self._POLL_INTERVAL, remaining)) + + def _release(self) -> None: + """Release the lock and remove the holder record.""" + if self._lock_fh is None: + return + try: + self._lock_fh.seek(0) + self._lock_fh.truncate() + fcntl.flock(self._lock_fh, fcntl.LOCK_UN) + finally: + self._lock_fh.close() + self._lock_fh = None + print( + f"[lock] released for {self._issue_key} (run {self._run_id})" + ) + + +def read_lock_record(repo_path: Path) -> Optional[dict]: + """Return the current lock holder record, or ``None`` if unlocked or unreadable. + + Parameters + ---------- + repo_path: + Root of the repository whose lock file should be read. + + Returns + ------- + dict or None + Parsed JSON record with keys ``run_id``, ``issue_key``, + ``started_at``, ``pid``; or ``None`` if the file does not exist or + contains no valid JSON. + """ + lock_path = Path(repo_path) / RepoLock._LOCK_SUBDIR / RepoLock._LOCK_FILENAME + try: + content = lock_path.read_text().strip() + if not content: + return None + return json.loads(content) + except (FileNotFoundError, json.JSONDecodeError, OSError): + return None diff --git a/orchestrator/planner.py b/harness/planner.py similarity index 100% rename from orchestrator/planner.py rename to harness/planner.py diff --git a/harness/provenance.py b/harness/provenance.py new file mode 100644 index 0000000..6190bba --- /dev/null +++ b/harness/provenance.py @@ -0,0 +1,384 @@ +""" +Provenance logger — streams structured JSONL events for every harness step. + +Two purposes: + 1. Audit trail — immutable record of what ran, when, inputs/outputs, outcome. + 2. RL feedback — reward-annotated trajectories consumable by a training loop. + +Directory layout: + provenance/ + runs/ + {run_id}.jsonl ← one event per line, streamed as they happen + {run_id}.summary.json ← written atomically at run end + index.jsonl ← one summary per line across all runs +""" + +from __future__ import annotations + +import datetime +import json +import shutil +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Any, Dict, List, Optional + + +# --------------------------------------------------------------------------- +# Data classes +# --------------------------------------------------------------------------- + +@dataclass +class StepRecord: + step: str + success: bool + duration_ms: float + output_preview: str = "" + error: Optional[str] = None + tokens_in: int = 0 + tokens_out: int = 0 + cost_usd: float = 0.0 + + +@dataclass +class GateRecord: + gate: str + passed: bool + attempt: int + outputs: Dict[str, Any] + duration_ms: float + error: Optional[str] = None + tokens_in: int = 0 + tokens_out: int = 0 + cost_usd: float = 0.0 + + +# --------------------------------------------------------------------------- +# Logger +# --------------------------------------------------------------------------- + +class ProvenanceLogger: + """ + Writes provenance events to JSONL files immediately (no buffering). + Partial runs are always recoverable. + """ + + def __init__(self, provenance_dir: Path): + self.dir = Path(provenance_dir) + self.runs_dir = self.dir / "runs" + self.runs_dir.mkdir(parents=True, exist_ok=True) + self.index_path = self.dir / "index.jsonl" + + # ------------------------------------------------------------------ + # Public event API + # ------------------------------------------------------------------ + + def start_run(self, run_id: str, issue_key: str, repository: str, repo_path: str) -> None: + self._append(self._run_path(run_id), { + "event": "run_start", + "run_id": run_id, + "issue_key": issue_key, + "repository": repository, + "repo_path": repo_path, + "timestamp": _ts(), + }) + + def log_step(self, run_id: str, step: StepRecord) -> None: + self._append(self._run_path(run_id), { + "event": "step", + "run_id": run_id, + "step": step.step, + "success": step.success, + "duration_ms": step.duration_ms, + "output_preview": step.output_preview[:500] if step.output_preview else "", + "error": step.error, + "tokens_in": step.tokens_in, + "tokens_out": step.tokens_out, + "cost_usd": step.cost_usd, + "timestamp": _ts(), + "reward": 1.0 if step.success else 0.0, + }) + + def log_gate(self, run_id: str, gate: GateRecord) -> None: + # Detect flakiness: passed on this attempt, but previous attempt for this + # gate (if any) failed — and no fix step ran between them in this run. + flaky = self._is_flaky(run_id, gate) + self._append(self._run_path(run_id), { + "event": "gate", + "run_id": run_id, + "gate": gate.gate, + "passed": gate.passed, + "attempt": gate.attempt, + "outputs": gate.outputs, + "duration_ms": gate.duration_ms, + "error": gate.error, + "tokens_in": gate.tokens_in, + "tokens_out": gate.tokens_out, + "cost_usd": gate.cost_usd, + "flaky": flaky, + "timestamp": _ts(), + "reward": 1.0 if gate.passed else 0.0, + }) + + def log_gate_loop( + self, + run_id: str, + passed: bool, + attempts: int, + failures: Optional[List[GateRecord]] = None, + ) -> None: + self._append(self._run_path(run_id), { + "event": "gate_loop_complete", + "run_id": run_id, + "passed": passed, + "attempts": attempts, + "remaining_failures": [ + {"gate": f.gate, "outputs": f.outputs} for f in (failures or []) + ], + "timestamp": _ts(), + # Efficiency reward: fewer attempts = better; zero if didn't pass + "reward": (1.0 / attempts) if passed else 0.0, + }) + + def log_error(self, run_id: str, error: str) -> None: + self._append(self._run_path(run_id), { + "event": "error", + "run_id": run_id, + "error": error, + "timestamp": _ts(), + "reward": 0.0, + }) + + def finish_run( + self, + run_id: str, + issue_key: str, + repository: str, + overall_outcome: str, + gate_attempts: int, + steps: List[StepRecord], + gate_results: List[GateRecord], + pr_url: Optional[str], + duration_ms: float, + ) -> None: + reward = {"success": 1.0, "partial": 0.5, "failed": 0.0}.get(overall_outcome, 0.0) + total_cost = sum(s.cost_usd for s in steps) + sum(g.cost_usd for g in gate_results) + total_tokens_in = sum(s.tokens_in for s in steps) + sum(g.tokens_in for g in gate_results) + total_tokens_out = sum(s.tokens_out for s in steps) + sum(g.tokens_out for g in gate_results) + + summary = { + "run_id": run_id, + "issue_key": issue_key, + "repository": repository, + "overall_outcome": overall_outcome, + "gate_attempts": gate_attempts, + "steps_total": len(steps), + "steps_succeeded": sum(1 for s in steps if s.success), + "gates_total": len(gate_results), + "gates_passed": sum(1 for g in gate_results if g.passed), + "gate_breakdown": _gate_breakdown(gate_results), + "pr_url": pr_url, + "duration_ms": duration_ms, + "cost_usd": round(total_cost, 6), + "tokens_in": total_tokens_in, + "tokens_out": total_tokens_out, + "timestamp": _ts(), + "reward": reward, + "human_feedback": None, + "human_rating": None, + } + + # Atomic summary file + summary_path = self.runs_dir / f"{run_id}.summary.json" + summary_path.write_text(json.dumps(summary, indent=2)) + + # Final event in run log + self._append(self._run_path(run_id), {"event": "run_end", **summary}) + + # Global index + self._append(self.index_path, summary) + + # ------------------------------------------------------------------ + # Read / export + # ------------------------------------------------------------------ + + def read_run(self, run_id: str) -> List[Dict[str, Any]]: + path = self._run_path(run_id) + if not path.exists(): + return [] + return [json.loads(line) for line in path.read_text().splitlines() if line.strip()] + + def export_rl_dataset(self, output_path: Path) -> int: + """ + Export all completed runs as RL training trajectories. + + Each trajectory contains: + - Metadata (run_id, issue, repo, overall reward) + - Ordered step events with per-step rewards + - Human feedback slot (if filled) + + Returns number of trajectories written. + """ + trajectories = [] + for summary_file in sorted(self.runs_dir.glob("*.summary.json")): + run_id = summary_file.stem.replace(".summary", "") + events = self.read_run(run_id) + summary = json.loads(summary_file.read_text()) + + trajectories.append({ + "run_id": run_id, + "issue_key": summary["issue_key"], + "repository": summary["repository"], + "overall_outcome": summary["overall_outcome"], + "overall_reward": summary["reward"], + "human_feedback": summary.get("human_feedback"), + "human_rating": summary.get("human_rating"), + "gate_breakdown": summary.get("gate_breakdown", {}), + "trajectory": [ + e for e in events if e["event"] in ("step", "gate", "gate_loop_complete") + ], + }) + + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(json.dumps(trajectories, indent=2)) + return len(trajectories) + + # ------------------------------------------------------------------ + # Analysis helpers (loop engineering) + # ------------------------------------------------------------------ + + def total_cost(self, run_id: str) -> float: + """Sum cost_usd across all events in a run.""" + return sum( + e.get("cost_usd", 0.0) + for e in self.read_run(run_id) + if e.get("event") in ("step", "gate") + ) + + def flaky_gates(self, run_id: str) -> List[str]: + """ + Return gate names that flipped pass→fail→pass without a fix step in between. + A gate is flaky if it passed on attempt N > 1 with no fix step between the + previous failed attempt and this passing attempt. + """ + events = self.read_run(run_id) + flaky = [ + e["gate"] for e in events + if e.get("event") == "gate" and e.get("flaky") + ] + return list(dict.fromkeys(flaky)) # deduplicated, order-preserving + + def rotate(self, max_age_days: int = 90) -> int: + """ + Move run JSONL + summary files older than max_age_days to provenance/archive/. + Returns count of files archived. + """ + archive_dir = self.dir / "archive" + archive_dir.mkdir(exist_ok=True) + cutoff = datetime.datetime.utcnow() - datetime.timedelta(days=max_age_days) + archived = 0 + + for summary_file in self.runs_dir.glob("*.summary.json"): + try: + summary = json.loads(summary_file.read_text()) + ts_str = summary.get("timestamp", "") + ts = datetime.datetime.fromisoformat(ts_str.rstrip("Z")) + if ts < cutoff: + run_id = summary_file.stem.replace(".summary", "") + for path in [summary_file, self._run_path(run_id)]: + if path.exists(): + shutil.move(str(path), str(archive_dir / path.name)) + archived += 1 + except Exception: + continue # skip unreadable files + + return archived + + def gate_failure_rates(self) -> Dict[str, float]: + """ + Compute per-gate failure rate across all runs. + Used to prioritise which gates need more fix capacity. + """ + counts: Dict[str, int] = {} + failures: Dict[str, int] = {} + + for summary_file in self.runs_dir.glob("*.summary.json"): + run_id = summary_file.stem.replace(".summary", "") + for event in self.read_run(run_id): + if event.get("event") != "gate": + continue + gate = event["gate"] + counts[gate] = counts.get(gate, 0) + 1 + if not event["passed"]: + failures[gate] = failures.get(gate, 0) + 1 + + return { + gate: failures.get(gate, 0) / total + for gate, total in counts.items() + } + + def avg_attempts_to_pass(self) -> float: + """Average gate loop attempts across successful runs.""" + attempts = [] + for f in self.runs_dir.glob("*.summary.json"): + s = json.loads(f.read_text()) + if s["overall_outcome"] == "success": + attempts.append(s["gate_attempts"]) + return sum(attempts) / len(attempts) if attempts else 0.0 + + def success_rate(self) -> float: + """Fraction of runs that ended in overall_outcome == 'success'.""" + summaries = list(self.runs_dir.glob("*.summary.json")) + if not summaries: + return 0.0 + successes = sum( + 1 for f in summaries + if json.loads(f.read_text()).get("overall_outcome") == "success" + ) + return successes / len(summaries) + + # ------------------------------------------------------------------ + # Internals + # ------------------------------------------------------------------ + + def _is_flaky(self, run_id: str, gate: GateRecord) -> bool: + """ + True if this gate passed (attempt > 1) and the previous attempt for the + same gate failed, with no fix step logged between them. + """ + if not gate.passed or gate.attempt <= 1: + return False + events = self.read_run(run_id) + prev_gate_failed = False + fix_between = False + for e in events: + ev = e.get("event") + if ev == "gate" and e.get("gate") == gate.gate and e.get("attempt") == gate.attempt - 1: + prev_gate_failed = not e.get("passed", True) + if ev == "step" and e.get("step", "").startswith("fix.") and prev_gate_failed: + fix_between = True + return prev_gate_failed and not fix_between + + def _run_path(self, run_id: str) -> Path: + return self.runs_dir / f"{run_id}.jsonl" + + def _append(self, path: Path, event: Dict[str, Any]) -> None: + with open(path, "a") as fh: + fh.write(json.dumps(event) + "\n") + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _ts() -> str: + return datetime.datetime.utcnow().isoformat() + "Z" + + +def _gate_breakdown(gate_results: List[GateRecord]) -> Dict[str, Dict]: + breakdown: Dict[str, Dict] = {} + for g in gate_results: + if g.gate not in breakdown: + breakdown[g.gate] = {"attempts": 0, "final": "pass"} + breakdown[g.gate]["attempts"] += 1 + breakdown[g.gate]["final"] = "pass" if g.passed else "fail" + return breakdown diff --git a/orchestrator/reporter.py b/harness/reporter.py similarity index 100% rename from orchestrator/reporter.py rename to harness/reporter.py diff --git a/orchestrator/router.py b/harness/router.py similarity index 100% rename from orchestrator/router.py rename to harness/router.py diff --git a/harness/scrubber.py b/harness/scrubber.py new file mode 100644 index 0000000..8b95a36 --- /dev/null +++ b/harness/scrubber.py @@ -0,0 +1,51 @@ +"""Scrub secrets from text before writing to provenance.""" + +import re + +PATTERNS = [ + # API keys / tokens / passwords in key=value form + r'(?i)(api[_-]?key|token|secret|password|credential|auth)\s*[:=]\s*\S+', + # Bearer tokens + r'Bearer\s+[A-Za-z0-9\-._~+/]+=*', + # GitHub PATs + r'ghp_[A-Za-z0-9]{36}', + r'github_pat_[A-Za-z0-9_]{82}', + # OpenAI / Anthropic key patterns + r'sk-ant-[A-Za-z0-9\-_]{90,}', + r'sk-[A-Za-z0-9\-]{40,}', + # Long base64-like strings (>=32 chars) — kept last so more-specific patterns above take priority + r'[A-Za-z0-9+/]{32,}={0,2}', +] + +# Pre-compile all patterns for performance. +_COMPILED = [re.compile(p) for p in PATTERNS] + +_REDACTED = "[REDACTED]" + + +def scrub(text: str) -> str: + """Replace secret-like patterns with [REDACTED]. Returns cleaned text. + + Each pattern is applied in order; earlier (more-specific) patterns are + replaced before the broad base64 catch-all so that context words such as + ``api_key=`` are not left dangling after the value is removed. + + Parameters + ---------- + text: + Arbitrary string that may contain secrets (log lines, subprocess + output, JSON blobs, etc.). + + Returns + ------- + str + A copy of *text* with every matched secret replaced by ``[REDACTED]``. + If *text* is empty or ``None`` the original value is returned unchanged. + """ + if not text: + return text + + for compiled in _COMPILED: + text = compiled.sub(_REDACTED, text) + + return text diff --git a/harness/server.py b/harness/server.py new file mode 100644 index 0000000..4184365 --- /dev/null +++ b/harness/server.py @@ -0,0 +1,774 @@ +""" +Dark Factory observability server. + +FastAPI application providing: + - REST API for run history, stats, gate health, circuit breakers + - SSE streaming for live run tailing + - Prometheus metrics (text format, no prometheus_client dependency) + - Static dashboard at / + +Usage: + # Start standalone + python -m harness.server + + # Or from harness code: + from harness.server import ensure_server_running + ensure_server_running(provenance_dir, factory_root) +""" + +from __future__ import annotations + +import asyncio +import json +import os +import signal +import subprocess +import sys +import time +from collections import defaultdict +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, AsyncIterator, Dict, List, Optional + +# --------------------------------------------------------------------------- +# Dependency check +# --------------------------------------------------------------------------- + +try: + import fastapi + from fastapi import FastAPI, HTTPException, Query + from fastapi.responses import FileResponse, JSONResponse, StreamingResponse + from fastapi.middleware.cors import CORSMiddleware + import uvicorn +except ImportError: + print( + "ERROR: fastapi and uvicorn are required. Install with:\n" + " pip install fastapi uvicorn[standard]\n" + "or:\n" + " uv add fastapi uvicorn" + ) + sys.exit(1) + +# --------------------------------------------------------------------------- +# Configuration +# --------------------------------------------------------------------------- + +_SERVER_VERSION = "1.0.0" +_PORT = int(os.environ.get("HARNESS_SERVER_PORT", "8089")) +_SERVER_START_TIME = time.time() + +# Resolve provenance directory: env var > default relative to this file +_DEFAULT_PROVENANCE_DIR = Path(__file__).parent.parent / "provenance" +_PROVENANCE_DIR: Path = Path( + os.environ.get("HARNESS_PROVENANCE_DIR", str(_DEFAULT_PROVENANCE_DIR)) +) + +# Dashboard static files +_DASHBOARD_DIR = Path(__file__).parent / "dashboard" + +# --------------------------------------------------------------------------- +# Active runs registry +# --------------------------------------------------------------------------- + +# Dict of run_id → {"issue_key", "repository", "status", "pid", "started_at", ...} +_active_runs: Dict[str, dict] = {} +_active_subprocesses: Dict[str, subprocess.Popen] = {} + + +def register_run(run_id: str, issue_key: str, repository: str, pid: Optional[int] = None) -> None: + """Called by harness instances when a run starts.""" + _active_runs[run_id] = { + "run_id": run_id, + "issue_key": issue_key, + "repository": repository, + "status": "running", + "current_step": None, + "pid": pid, + "started_at": datetime.now(timezone.utc).isoformat(), + } + + +def update_run(run_id: str, **kwargs) -> None: + """Update metadata for an active run (e.g., current_step).""" + if run_id in _active_runs: + _active_runs[run_id].update(kwargs) + + +def complete_run(run_id: str, outcome: str) -> None: + """Mark a run complete and remove from active tracking.""" + _active_runs.pop(run_id, None) + _active_subprocesses.pop(run_id, None) + + +# --------------------------------------------------------------------------- +# Provenance helpers +# --------------------------------------------------------------------------- + +def _runs_dir() -> Path: + return _PROVENANCE_DIR / "runs" + + +def _run_jsonl_path(run_id: str) -> Path: + return _runs_dir() / f"{run_id}.jsonl" + + +def _run_summary_path(run_id: str) -> Path: + return _runs_dir() / f"{run_id}.summary.json" + + +def _index_path() -> Path: + return _PROVENANCE_DIR / "index.jsonl" + + +def _circuit_breakers_path() -> Path: + return _PROVENANCE_DIR / "circuit_breakers.json" + + +def _read_index() -> List[dict]: + """Read all summaries from index.jsonl, newest first.""" + idx = _index_path() + if not idx.exists(): + return [] + lines = idx.read_text().splitlines() + results = [] + for line in lines: + line = line.strip() + if line: + try: + results.append(json.loads(line)) + except json.JSONDecodeError: + pass + return list(reversed(results)) # newest first + + +def _read_summary(run_id: str) -> Optional[dict]: + path = _run_summary_path(run_id) + if not path.exists(): + return None + try: + return json.loads(path.read_text()) + except (json.JSONDecodeError, OSError): + return None + + +def _read_run_events(run_id: str) -> List[dict]: + path = _run_jsonl_path(run_id) + if not path.exists(): + return [] + events = [] + for line in path.read_text().splitlines(): + line = line.strip() + if line: + try: + events.append(json.loads(line)) + except json.JSONDecodeError: + pass + return events + + +def _parse_ts(ts: Optional[str]) -> Optional[datetime]: + if not ts: + return None + try: + return datetime.fromisoformat(ts.replace("Z", "+00:00")) + except ValueError: + return None + + +def _days_ago(n: int) -> datetime: + from datetime import timedelta + return datetime.now(timezone.utc) - timedelta(days=n) + + +# --------------------------------------------------------------------------- +# Stats helpers +# --------------------------------------------------------------------------- + +def _compute_stats() -> dict: + summaries = _read_index() + + today = datetime.now(timezone.utc).date() + cutoff_7d = _days_ago(7) + + total = len(summaries) + successes = sum(1 for s in summaries if s.get("overall_outcome") == "success") + success_rate = (successes / total) if total else 0.0 + + # Cost estimation: ~$0.01 per 1000 tokens; use duration as proxy + # In reality, cost is not tracked in provenance; return 0 if no cost field + def _cost(s: dict) -> float: + return float(s.get("cost_usd", 0.0)) + + runs_today = [ + s for s in summaries + if _parse_ts(s.get("timestamp")) and _parse_ts(s.get("timestamp")).date() == today + ] + runs_7d = [ + s for s in summaries + if _parse_ts(s.get("timestamp")) and _parse_ts(s.get("timestamp")) >= cutoff_7d + ] + + avg_gate_attempts = ( + sum(s.get("gate_attempts", 0) for s in summaries) / total + if total else 0.0 + ) + + return { + "success_rate": round(success_rate, 3), + "avg_cost_usd": round(sum(_cost(s) for s in summaries), 4), + "avg_gate_attempts": round(avg_gate_attempts, 2), + "runs_today": len(runs_today), + "cost_today_usd": round(sum(_cost(s) for s in runs_today), 4), + "runs_7d": len(runs_7d), + "cost_7d_usd": round(sum(_cost(s) for s in runs_7d), 4), + "total_runs": total, + } + + +def _compute_gate_health() -> dict: + """Per-gate pass rate, avg duration, circuit state over last 7 days.""" + cutoff = _days_ago(7) + + gate_stats: Dict[str, dict] = defaultdict(lambda: { + "pass": 0, "fail": 0, "duration_ms_sum": 0.0, "duration_count": 0 + }) + + summaries = _read_index() + for s in summaries: + ts = _parse_ts(s.get("timestamp")) + if ts and ts < cutoff: + continue + run_id = s.get("run_id") + if not run_id: + continue + for event in _read_run_events(run_id): + if event.get("event") != "gate": + continue + gate = event.get("gate", "unknown") + gs = gate_stats[gate] + if event.get("passed"): + gs["pass"] += 1 + else: + gs["fail"] += 1 + dur = event.get("duration_ms") + if dur is not None: + gs["duration_ms_sum"] += float(dur) + gs["duration_count"] += 1 + + # Read circuit breaker state + cb_state: dict = {} + cb_path = _circuit_breakers_path() + if cb_path.exists(): + try: + cb_state = json.loads(cb_path.read_text()) + except (json.JSONDecodeError, OSError): + pass + + result = {} + for gate, gs in gate_stats.items(): + total = gs["pass"] + gs["fail"] + pass_rate = gs["pass"] / total if total else 0.0 + avg_dur = gs["duration_ms_sum"] / gs["duration_count"] if gs["duration_count"] else 0.0 + circuit = cb_state.get(gate, {}).get("state", "closed") + result[gate] = { + "gate": gate, + "pass_rate": round(pass_rate, 3), + "total_attempts": total, + "passes": gs["pass"], + "failures": gs["fail"], + "avg_duration_ms": round(avg_dur, 1), + "circuit_state": circuit, + } + + return result + + +# --------------------------------------------------------------------------- +# Prometheus metrics renderer +# --------------------------------------------------------------------------- + +def _render_prometheus() -> str: + """ + Build Prometheus text format from provenance data. + Does NOT import prometheus_client. + """ + lines: List[str] = [] + + def metric_line(name: str, labels: dict, value: float) -> str: + if labels: + label_str = ",".join(f'{k}="{v}"' for k, v in labels.items()) + return f"{name}{{{label_str}}} {value}" + return f"{name} {value}" + + # ── dark_factory_runs_total ───────────────────────────────────────── + lines += [ + "# HELP dark_factory_runs_total Total harness runs by outcome and repo", + "# TYPE dark_factory_runs_total counter", + ] + counts: Dict[tuple, int] = defaultdict(int) + for s in _read_index(): + key = (s.get("overall_outcome", "unknown"), s.get("repository", "unknown")) + counts[key] += 1 + for (outcome, repo), n in sorted(counts.items()): + lines.append(metric_line("dark_factory_runs_total", {"outcome": outcome, "repo": repo}, n)) + + # ── dark_factory_gate_results_total ──────────────────────────────── + lines += [ + "# HELP dark_factory_gate_results_total Gate pass/fail counts", + "# TYPE dark_factory_gate_results_total counter", + ] + gate_counts: Dict[tuple, int] = defaultdict(int) + for s in _read_index(): + run_id = s.get("run_id") + if not run_id: + continue + for event in _read_run_events(run_id): + if event.get("event") != "gate": + continue + gate = event.get("gate", "unknown") + result = "pass" if event.get("passed") else "fail" + gate_counts[(gate, result)] += 1 + for (gate, result), n in sorted(gate_counts.items()): + lines.append(metric_line("dark_factory_gate_results_total", {"gate": gate, "result": result}, n)) + + # ── dark_factory_run_duration_seconds histogram ──────────────────── + lines += [ + "# HELP dark_factory_run_duration_seconds Harness run duration", + "# TYPE dark_factory_run_duration_seconds histogram", + ] + buckets = [60, 120, 300, 600, 900, 1800, float("inf")] + bucket_labels = ["60", "120", "300", "600", "900", "1800", "+Inf"] + repo_durations: Dict[str, List[float]] = defaultdict(list) + for s in _read_index(): + dur_s = float(s.get("duration_ms", 0)) / 1000.0 + repo = s.get("repository", "unknown") + repo_durations[repo].append(dur_s) + for repo, durations in sorted(repo_durations.items()): + bucket_counts = [0] * len(buckets) + for dur in durations: + for i, b in enumerate(buckets): + if dur <= b: + bucket_counts[i] += 1 + cum = 0 + for i, (b_label, b_count) in enumerate(zip(bucket_labels, bucket_counts)): + cum += b_count + lines.append(metric_line( + "dark_factory_run_duration_seconds_bucket", + {"repo": repo, "le": b_label}, cum + )) + total_dur = sum(durations) + lines.append(metric_line("dark_factory_run_duration_seconds_sum", {"repo": repo}, round(total_dur, 3))) + lines.append(metric_line("dark_factory_run_duration_seconds_count", {"repo": repo}, len(durations))) + + # ── dark_factory_active_runs gauge ───────────────────────────────── + lines += [ + "# HELP dark_factory_active_runs Currently running harness runs", + "# TYPE dark_factory_active_runs gauge", + f"dark_factory_active_runs {len(_active_runs)}", + ] + + # ── dark_factory_cost_dollars_total counter ──────────────────────── + lines += [ + "# HELP dark_factory_cost_dollars_total Total API cost in USD", + "# TYPE dark_factory_cost_dollars_total counter", + ] + repo_costs: Dict[str, float] = defaultdict(float) + for s in _read_index(): + repo = s.get("repository", "unknown") + repo_costs[repo] += float(s.get("cost_usd", 0.0)) + for repo, cost in sorted(repo_costs.items()): + lines.append(metric_line("dark_factory_cost_dollars_total", {"repo": repo}, round(cost, 6))) + + # ── dark_factory_circuit_breaker_open gauge ──────────────────────── + lines += [ + "# HELP dark_factory_circuit_breaker_open Circuit breaker state (1=open/broken, 0=closed/healthy)", + "# TYPE dark_factory_circuit_breaker_open gauge", + ] + cb_state: dict = {} + cb_path = _circuit_breakers_path() + if cb_path.exists(): + try: + cb_state = json.loads(cb_path.read_text()) + except (json.JSONDecodeError, OSError): + pass + for gate, info in sorted(cb_state.items()): + is_open = 1 if info.get("state") == "open" else 0 + lines.append(metric_line("dark_factory_circuit_breaker_open", {"gate": gate}, is_open)) + + return "\n".join(lines) + "\n" + + +# --------------------------------------------------------------------------- +# SSE streaming +# --------------------------------------------------------------------------- + +async def _tail_run_events(run_id: str) -> AsyncIterator[str]: + """ + Yield SSE-formatted lines for a run's JSONL file. + Emits existing lines, then tails for new lines until run_end or disconnect. + """ + path = _run_jsonl_path(run_id) + position = 0 + + # Wait up to 5s for the file to appear (run may have just started) + for _ in range(10): + if path.exists(): + break + await asyncio.sleep(0.5) + + if not path.exists(): + yield f"data: {json.dumps({'error': 'run not found', 'run_id': run_id})}\n\n" + return + + finished = False + while not finished: + try: + with open(path, "r") as fh: + fh.seek(position) + while True: + line = fh.readline() + if not line: + break + line = line.strip() + if not line: + continue + position = fh.tell() + try: + event = json.loads(line) + except json.JSONDecodeError: + continue + yield f"data: {json.dumps(event)}\n\n" + if event.get("event") in ("run_end", "error"): + finished = True + break + except OSError: + pass + + if not finished: + await asyncio.sleep(0.5) + + +# --------------------------------------------------------------------------- +# FastAPI app +# --------------------------------------------------------------------------- + +app = FastAPI( + title="Dark Factory Observability Server", + version=_SERVER_VERSION, + docs_url="/docs", + redoc_url="/redoc", +) + +app.add_middleware( + CORSMiddleware, + allow_origins=["*"], + allow_methods=["*"], + allow_headers=["*"], +) + + +# ── Health ───────────────────────────────────────────────────────────────── + +@app.get("/health", tags=["meta"]) +async def health() -> dict: + return { + "status": "ok", + "uptime_s": round(time.time() - _SERVER_START_TIME, 1), + "active_runs_count": len(_active_runs), + "server_version": _SERVER_VERSION, + } + + +# ── Static dashboard ──────────────────────────────────────────────────────── + +@app.get("/", include_in_schema=False) +async def dashboard(): + index = _DASHBOARD_DIR / "index.html" + if index.exists(): + return FileResponse(str(index), media_type="text/html") + return JSONResponse( + {"error": "Dashboard not found", "hint": "Expected at harness/dashboard/index.html"}, + status_code=404, + ) + + +# ── Runs ─────────────────────────────────────────────────────────────────── + +@app.get("/api/runs", tags=["runs"]) +async def list_runs( + repo: Optional[str] = Query(None), + outcome: Optional[str] = Query(None), + limit: int = Query(50, ge=1, le=500), + offset: int = Query(0, ge=0), +) -> dict: + summaries = _read_index() + + if repo: + summaries = [s for s in summaries if s.get("repository") == repo] + if outcome: + summaries = [s for s in summaries if s.get("overall_outcome") == outcome] + + total = len(summaries) + page = summaries[offset: offset + limit] + + # Enrich with active-run info + active_ids = set(_active_runs.keys()) + for s in page: + s["is_active"] = s.get("run_id") in active_ids + + return {"total": total, "offset": offset, "limit": limit, "runs": page} + + +@app.get("/api/runs/{run_id}", tags=["runs"]) +async def get_run(run_id: str) -> dict: + summary = _read_summary(run_id) + events = _read_run_events(run_id) + + if not summary and not events: + # May be an in-progress run without a summary yet + active = _active_runs.get(run_id) + if not active: + raise HTTPException(status_code=404, detail=f"Run {run_id!r} not found") + return { + "run_id": run_id, + "summary": active, + "events": events, + "is_active": True, + } + + return { + "run_id": run_id, + "summary": summary, + "events": events, + "is_active": run_id in _active_runs, + } + + +@app.get("/api/runs/{run_id}/events", tags=["runs"]) +async def stream_run_events(run_id: str) -> StreamingResponse: + return StreamingResponse( + _tail_run_events(run_id), + media_type="text/event-stream", + headers={ + "Cache-Control": "no-cache", + "X-Accel-Buffering": "no", + }, + ) + + +@app.post("/api/runs", tags=["runs"], status_code=202) +async def spawn_run(body: dict) -> dict: + """ + Spawn a harness run as a background subprocess. + Body: {"issue_key": "ABI-123", "repository": "runtime"} + Returns: {"run_id": "run_..."} + """ + issue_key = body.get("issue_key") + repository = body.get("repository", "runtime") + if not issue_key: + raise HTTPException(status_code=400, detail="issue_key is required") + + # Build run_id now so we can return it immediately + import uuid as _uuid + run_id = f"run_{int(time.time())}_{_uuid.uuid4().hex[:8]}" + + factory_root = Path(__file__).parent.parent + cmd = [ + sys.executable, "-m", "harness.cli", "implement", issue_key, + "--repo", repository, + "--harness", + ] + + try: + proc = subprocess.Popen( + cmd, + cwd=str(factory_root), + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + register_run(run_id, issue_key, repository, pid=proc.pid) + _active_subprocesses[run_id] = proc + except OSError as exc: + raise HTTPException(status_code=500, detail=f"Failed to spawn: {exc}") from exc + + return {"run_id": run_id, "issue_key": issue_key, "repository": repository, "pid": proc.pid} + + +@app.post("/api/runs/{run_id}/cancel", tags=["runs"]) +async def cancel_run(run_id: str) -> dict: + proc = _active_subprocesses.get(run_id) + active = _active_runs.get(run_id) + + if not proc and not active: + raise HTTPException(status_code=404, detail=f"Active run {run_id!r} not found") + + killed = False + if proc: + try: + os.killpg(os.getpgid(proc.pid), signal.SIGTERM) + killed = True + except (ProcessLookupError, PermissionError, OSError): + try: + proc.terminate() + killed = True + except (ProcessLookupError, OSError): + pass + elif active and active.get("pid"): + try: + os.kill(active["pid"], signal.SIGTERM) + killed = True + except (ProcessLookupError, PermissionError, OSError): + pass + + complete_run(run_id, "cancelled") + return {"run_id": run_id, "cancelled": killed} + + +# ── Queue ────────────────────────────────────────────────────────────────── + +@app.get("/api/queue", tags=["queue"]) +async def get_queue() -> dict: + active = list(_active_runs.values()) + + # "waiting" runs: look for lock files + waiting = [] + lock_dir = _PROVENANCE_DIR / "locks" + if lock_dir.exists(): + for lf in lock_dir.glob("*.lock"): + try: + info = json.loads(lf.read_text()) + waiting.append(info) + except (json.JSONDecodeError, OSError): + waiting.append({"lock_file": str(lf)}) + + return {"active": active, "waiting": waiting} + + +# ── Stats ────────────────────────────────────────────────────────────────── + +@app.get("/api/stats", tags=["stats"]) +async def get_stats() -> dict: + return _compute_stats() + + +# ── Gates ───────────────────────────────────────────────────────────────── + +@app.get("/api/gates/health", tags=["gates"]) +async def get_gate_health() -> dict: + return _compute_gate_health() + + +# ── Circuit Breakers ─────────────────────────────────────────────────────── + +@app.get("/api/circuit-breakers", tags=["circuit-breakers"]) +async def get_circuit_breakers() -> dict: + path = _circuit_breakers_path() + if not path.exists(): + return {} + try: + return json.loads(path.read_text()) + except (json.JSONDecodeError, OSError) as exc: + raise HTTPException(status_code=500, detail=str(exc)) from exc + + +@app.post("/api/circuit-breakers/{gate}/reset", tags=["circuit-breakers"]) +async def reset_circuit_breaker(gate: str) -> dict: + path = _circuit_breakers_path() + state: dict = {} + if path.exists(): + try: + state = json.loads(path.read_text()) + except (json.JSONDecodeError, OSError): + state = {} + + if gate not in state: + raise HTTPException(status_code=404, detail=f"Gate {gate!r} not found in circuit breakers") + + state[gate]["state"] = "closed" + state[gate]["reset_at"] = datetime.now(timezone.utc).isoformat() + path.write_text(json.dumps(state, indent=2)) + return {"gate": gate, "state": "closed", "message": "Circuit breaker reset"} + + +# ── Prometheus metrics ───────────────────────────────────────────────────── + +@app.get("/metrics", tags=["metrics"], response_class=fastapi.responses.PlainTextResponse) +async def prometheus_metrics() -> str: + return _render_prometheus() + + +# --------------------------------------------------------------------------- +# ensure_server_running helper (called by harness.py) +# --------------------------------------------------------------------------- + +def ensure_server_running(provenance_dir: Path, factory_root: Path) -> bool: + """ + Start the server as a background subprocess if not already running. + Writes PID to provenance_dir/.server.pid. + Returns True if started, False if already running. + """ + pid_file = Path(provenance_dir) / ".server.pid" + + # Check if already running + if pid_file.exists(): + try: + pid = int(pid_file.read_text().strip()) + os.kill(pid, 0) # signal 0 = existence check, no actual signal + # Process exists — server is running + return False + except (ValueError, ProcessLookupError, PermissionError, OSError): + pass # stale PID or not running; start it + + # Start the server + cmd = [sys.executable, "-m", "harness.server"] + env = os.environ.copy() + env["HARNESS_PROVENANCE_DIR"] = str(provenance_dir) + + try: + proc = subprocess.Popen( + cmd, + cwd=str(factory_root), + env=env, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + pid_file.parent.mkdir(parents=True, exist_ok=True) + pid_file.write_text(str(proc.pid)) + # Give it a moment to bind + time.sleep(1.0) + print(f" Dark Factory server started (PID {proc.pid}) → http://localhost:{_PORT}/") + return True + except OSError as exc: + print(f" WARNING: Could not start observability server: {exc}") + return False + + +# --------------------------------------------------------------------------- +# Entrypoint +# --------------------------------------------------------------------------- + +if __name__ == "__main__": + # Resolve provenance dir from env or default + prov_dir = Path(os.environ.get("HARNESS_PROVENANCE_DIR", str(_DEFAULT_PROVENANCE_DIR))) + prov_dir.mkdir(parents=True, exist_ok=True) + (prov_dir / "runs").mkdir(parents=True, exist_ok=True) + + # Write our own PID so ensure_server_running can detect us + pid_file = prov_dir / ".server.pid" + pid_file.write_text(str(os.getpid())) + + print(f"Dark Factory Observability Server v{_SERVER_VERSION}") + print(f" Dashboard : http://localhost:{_PORT}/") + print(f" API docs : http://localhost:{_PORT}/docs") + print(f" Metrics : http://localhost:{_PORT}/metrics") + print(f" Provenance: {prov_dir}") + + uvicorn.run( + "harness.server:app", + host="0.0.0.0", + port=_PORT, + log_level="info", + reload=False, + ) diff --git a/orchestrator/sync.py b/harness/sync.py similarity index 98% rename from orchestrator/sync.py rename to harness/sync.py index 9487cf0..08aea24 100644 --- a/orchestrator/sync.py +++ b/harness/sync.py @@ -1,7 +1,7 @@ """ Knowledge Pack Sync Module -Automatically syncs knowledge packs from repositories before orchestrator runs. +Automatically syncs knowledge packs from repositories before harness runs. Only re-extracts if source documentation has changed. """ diff --git a/harness/tui.py b/harness/tui.py new file mode 100644 index 0000000..7f5c040 --- /dev/null +++ b/harness/tui.py @@ -0,0 +1,906 @@ +""" +Dark Factory TUI — live-refreshing terminal dashboard for the AI Software Factory harness. + +Reads provenance files directly (no server dependency) and optionally fetches +active-run data from the REST server at http://localhost:8089/api if it is up. + +Usage: + python -m harness.tui + python harness/tui.py + harness tui # if wired into the CLI +""" + +from __future__ import annotations + +import datetime +import json +import os +import sys +import termios +import threading +import time +import tty +import urllib.error +import urllib.request +from pathlib import Path +from typing import Dict, List, Optional, Tuple + +# --------------------------------------------------------------------------- +# Rich import — graceful fallback +# --------------------------------------------------------------------------- + +try: + from rich.console import Console + from rich.layout import Layout + from rich.live import Live + from rich.panel import Panel + from rich.style import Style + from rich.table import Table + from rich.text import Text + from rich import box +except ImportError: + print("Error: 'rich' is not installed. Install it with:\n\n pip install rich\n", file=sys.stderr) + sys.exit(1) + +# --------------------------------------------------------------------------- +# Path discovery +# --------------------------------------------------------------------------- + +_HERE = Path(__file__).parent # harness/ +_FACTORY_ROOT = _HERE.parent # EM-AISoftwareFactory/ + +PROV_DIR: Path = ( + Path(os.environ["HARNESS_PROVENANCE_DIR"]) + if "HARNESS_PROVENANCE_DIR" in os.environ + else _FACTORY_ROOT / "provenance" +) +INDEX_PATH = PROV_DIR / "index.jsonl" +CIRCUIT_BREAKER_PATH = PROV_DIR / "circuit_breakers.json" +WORKSPACE_YAML = _FACTORY_ROOT / "workspace.yaml" + +SERVER_BASE = "http://localhost:8089/api" + +# --------------------------------------------------------------------------- +# Global state shared between refresh thread and key-binding thread +# --------------------------------------------------------------------------- + +_state: Dict = { + "active_runs": [], + "today_stats": {}, + "gate_health": {}, + "circuit_states": {}, + "recent_runs": [], + "last_refresh": None, + "show_help": False, + "overlay_text": None, # non-None string = show overlay panel + "running": True, + "force_refresh": threading.Event(), +} +_state_lock = threading.Lock() + +# --------------------------------------------------------------------------- +# Data loading +# --------------------------------------------------------------------------- + +def _read_index(limit: Optional[int] = None) -> List[dict]: + """Return parsed lines from index.jsonl, newest-first. Returns [] if missing.""" + if not INDEX_PATH.exists(): + return [] + lines = [] + try: + raw = INDEX_PATH.read_text().splitlines() + except OSError: + return [] + for line in reversed(raw): + line = line.strip() + if not line: + continue + try: + lines.append(json.loads(line)) + except json.JSONDecodeError: + pass + if limit and len(lines) >= limit: + break + return lines + + +def load_recent_runs(limit: int = 10) -> List[dict]: + """Read provenance/index.jsonl, return last `limit` summaries (newest first).""" + return _read_index(limit=limit) + + +def load_today_stats() -> dict: + """Filter index for today's runs. Return: total, success, partial, failed, cost_usd.""" + today = datetime.date.today().isoformat() + stats = {"total": 0, "success": 0, "partial": 0, "failed": 0, "cost_usd": 0.0, "avg_attempts": 0.0} + attempts_list: List[float] = [] + + for entry in _read_index(): + ts = entry.get("timestamp", "") + if not ts.startswith(today): + # Index is append-only newest at bottom; once we pass today, stop + continue + stats["total"] += 1 + outcome = entry.get("overall_outcome", "failed") + if outcome in stats: + stats[outcome] += 1 # type: ignore[literal-required] + # cost: not yet tracked in summary — placeholder based on reward heuristic + reward = entry.get("reward", 0.0) + stats["cost_usd"] += reward * 1.5 # rough proxy; real field TBD + attempts = entry.get("gate_attempts", 1) + if isinstance(attempts, (int, float)): + attempts_list.append(float(attempts)) + + if attempts_list: + stats["avg_attempts"] = sum(attempts_list) / len(attempts_list) + return stats + + +def load_gate_health(days: int = 7) -> Dict[str, dict]: + """ + Scan provenance/index.jsonl for runs in the last N days. + Per gate: pass_rate, avg_duration_ms, flakiness_rate. + """ + cutoff = (datetime.datetime.utcnow() - datetime.timedelta(days=days)).isoformat() + "Z" + + gate_data: Dict[str, dict] = {} + + runs_dir = PROV_DIR / "runs" + if not runs_dir.exists(): + return gate_data + + # Collect gate events from JSONL run files that fall in window + for summary_file in runs_dir.glob("*.summary.json"): + try: + summary = json.loads(summary_file.read_text()) + except (OSError, json.JSONDecodeError): + continue + + if summary.get("timestamp", "") < cutoff: + continue + + run_id = summary.get("run_id", "") + if not run_id: + continue + + run_file = runs_dir / f"{run_id}.jsonl" + if not run_file.exists(): + continue + + try: + events_raw = run_file.read_text().splitlines() + except OSError: + continue + + for line in events_raw: + line = line.strip() + if not line: + continue + try: + event = json.loads(line) + except json.JSONDecodeError: + continue + if event.get("event") != "gate": + continue + gate = event.get("gate", "unknown") + passed = bool(event.get("passed", False)) + attempt = int(event.get("attempt", 1)) + duration_ms = float(event.get("duration_ms", 0.0)) + + if gate not in gate_data: + gate_data[gate] = { + "total_attempts": 0, + "passes": 0, + "first_attempt_passes": 0, + "total_duration_ms": 0.0, + } + gd = gate_data[gate] + gd["total_attempts"] += 1 + if passed: + gd["passes"] += 1 + if attempt == 1: + gd["first_attempt_passes"] += 1 + gd["total_duration_ms"] += duration_ms + + # Compute rates + result: Dict[str, dict] = {} + for gate, gd in gate_data.items(): + total = gd["total_attempts"] + passes = gd["passes"] + first_passes = gd["first_attempt_passes"] + pass_rate = passes / total if total else 0.0 + # Flakiness: passed but not on first attempt (needed a retry) + flakiness = (passes - first_passes) / total if total else 0.0 + avg_dur = gd["total_duration_ms"] / total if total else 0.0 + result[gate] = { + "pass_rate": pass_rate, + "flakiness_rate": flakiness, + "avg_duration_ms": avg_dur, + "total_attempts": total, + } + return result + + +def load_circuit_states() -> Dict[str, dict]: + """Read provenance/circuit_breakers.json or return all-closed defaults.""" + defaults: Dict[str, dict] = { + gate: {"state": "closed", "consecutive_failures": 0, "opened_at": None} + for gate in ("linter", "tests", "evals", "code-review") + } + if not CIRCUIT_BREAKER_PATH.exists(): + return defaults + try: + data = json.loads(CIRCUIT_BREAKER_PATH.read_text()) + # Merge with defaults so any missing gate is closed + for gate, info in data.items(): + defaults[gate] = info + return defaults + except (OSError, json.JSONDecodeError): + return defaults + + +def _get_workspace_root() -> Optional[Path]: + """Parse workspace.yaml to get the workspace root path.""" + if not WORKSPACE_YAML.exists(): + return None + try: + import yaml # type: ignore[import-untyped] + with open(WORKSPACE_YAML) as f: + cfg = yaml.safe_load(f) + root = cfg.get("workspace", {}).get("root") + if root: + return Path(root) + except Exception: + pass + return None + + +def _try_server_queue() -> Optional[List[dict]]: + """ + Attempt GET http://localhost:8089/api/queue. + Returns list of active run dicts on success, None on failure. + """ + try: + req = urllib.request.Request(f"{SERVER_BASE}/queue", headers={"Accept": "application/json"}) + with urllib.request.urlopen(req, timeout=1.0) as resp: + return json.loads(resp.read().decode()) + except Exception: + return None + + +def load_active_runs() -> List[dict]: + """ + Try GET http://localhost:8089/api/queue first. + Fall back to scanning {workspace_root}/*/.harness-results/checkpoint.json. + """ + server_data = _try_server_queue() + if server_data is not None: + return server_data + + runs: List[dict] = [] + workspace_root = _get_workspace_root() or _FACTORY_ROOT.parent + if not workspace_root.exists(): + return runs + + for checkpoint_path in workspace_root.rglob(".harness-results/checkpoint.json"): + # Skip if older than 2 hours (stale) + try: + mtime = checkpoint_path.stat().st_mtime + if time.time() - mtime > 7200: + continue + except OSError: + continue + + try: + data = json.loads(checkpoint_path.read_text()) + except (OSError, json.JSONDecodeError): + continue + + repo_path = checkpoint_path.parent.parent + data["_repo_path"] = str(repo_path) + data["_checkpoint_mtime"] = mtime + runs.append(data) + + return runs + + +# --------------------------------------------------------------------------- +# Formatting helpers +# --------------------------------------------------------------------------- + +def _outcome_icon(outcome: str) -> str: + return {"success": "✅", "partial": "⚠️ ", "failed": "❌"}.get(outcome, " ? ") + + +def _outcome_style(outcome: str) -> str: + return {"success": "bold green", "partial": "bold yellow", "failed": "bold red"}.get(outcome, "") + + +def _short_run_id(run_id: str) -> str: + """Return a compact run_id: first 8 chars of the last UUID-like segment.""" + parts = run_id.split("_") + suffix = parts[-1] if parts else run_id + return suffix[:8] + + +def _fmt_duration(ms: Optional[float]) -> str: + if ms is None or ms <= 0: + return "—" + secs = int(ms / 1000) + if secs < 60: + return f"{secs}s" + mins = secs // 60 + secs_rem = secs % 60 + if mins < 60: + return f"{mins}m{secs_rem:02d}s" if secs_rem else f"{mins}m" + hrs = mins // 60 + mins_rem = mins % 60 + return f"{hrs}h{mins_rem:02d}m" + + +def _elapsed_since(mtime: float) -> str: + return _fmt_duration((time.time() - mtime) * 1000) + + +def _pass_bar(pass_rate: float, width: int = 20) -> Text: + """Render a text progress bar with appropriate color.""" + filled = int(pass_rate * width) + empty = width - filled + bar_text = "█" * filled + "░" * empty + + if pass_rate > 0.85: + color = "green" + elif pass_rate >= 0.70: + color = "yellow" + else: + color = "red" + + t = Text() + t.append(bar_text, style=color) + return t + + +def _circuit_icon(state: str) -> Text: + if state == "open": + return Text("⚠️ OPEN", style="bold red") + return Text("● closed", style="green") + + +# --------------------------------------------------------------------------- +# Panel / table renderers +# --------------------------------------------------------------------------- + +def _render_active_runs(active_runs: List[dict]) -> Panel: + count = len(active_runs) + title = f"Active Runs ({count})" if count else "Active Runs" + + table = Table( + box=box.SIMPLE, + show_header=False, + padding=(0, 1), + expand=True, + ) + table.add_column("icon", width=3, no_wrap=True) + table.add_column("issue", style="bold cyan", no_wrap=True) + table.add_column("step", no_wrap=True) + table.add_column("elapsed", no_wrap=True) + table.add_column("repo", style="dim", no_wrap=True) + + if not active_runs: + table.add_row("", Text("No active runs", style="dim"), "", "", "") + else: + for run in active_runs: + issue_key = run.get("issue_key", "—") + completed = run.get("completed_steps", []) + last_step = run.get("last_step", "—") or "—" + mtime = run.get("_checkpoint_mtime") + elapsed = _elapsed_since(mtime) if mtime else "—" + + # Determine icon from last step + if "queued" in last_step.lower(): + icon = Text("⏳", style="dim") + step_style = "dim" + elif "gate" in last_step.lower(): + icon = Text("🔄", style="bold yellow") + step_style = "bold yellow" + elif completed: + icon = Text("🔄", style="bold cyan") + step_style = "bold cyan" + else: + icon = Text("⏳", style="dim") + step_style = "dim" + + repo_path = run.get("_repo_path", "") + repo_name = Path(repo_path).name if repo_path else run.get("repository", "—") + + table.add_row( + icon, + issue_key, + Text(last_step, style=step_style), + elapsed, + repo_name, + ) + + return Panel(table, title=title, border_style="blue") + + +def _render_today_stats(stats: dict) -> Panel: + total = stats.get("total", 0) + success = stats.get("success", 0) + partial = stats.get("partial", 0) + failed = stats.get("failed", 0) + cost = stats.get("cost_usd", 0.0) + avg_att = stats.get("avg_attempts", 0.0) + + t = Text() + t.append(f"Runs: {total}\n", style="bold") + t.append("✅ ", style="bold green") + t.append(f"{success} ", style="bold green") + t.append("⚠️ ", style="bold yellow") + t.append(f"{partial} ", style="bold yellow") + t.append("❌ ", style="bold red") + t.append(f"{failed}\n", style="bold red") + t.append(f"Cost: ${cost:.2f}\n") + t.append(f"Avg: {avg_att:.1f} attempts") + + return Panel(t, title="Today", border_style="blue") + + +def _render_gate_health( + gate_health: Dict[str, dict], + circuit_states: Dict[str, dict], +) -> Panel: + table = Table( + box=box.SIMPLE, + show_header=False, + padding=(0, 1), + expand=True, + ) + table.add_column("gate", width=14, no_wrap=True) + table.add_column("bar", width=22, no_wrap=True) + table.add_column("pct", width=10, no_wrap=True) + table.add_column("flakiness", width=16, no_wrap=True) + table.add_column("circuit", no_wrap=True) + + known_gates = ("linter", "tests", "evals", "code-review") + all_gates = list(dict.fromkeys(list(known_gates) + list(gate_health.keys()))) + + for gate in all_gates: + info = gate_health.get(gate) + cb = circuit_states.get(gate, {"state": "closed"}) + cb_icon = _circuit_icon(cb.get("state", "closed")) + + if not info: + table.add_row( + Text(gate, style="dim"), + Text("─" * 20, style="dim"), + Text("—", style="dim"), + Text("—", style="dim"), + cb_icon, + ) + continue + + pass_rate = info["pass_rate"] + flakiness = info["flakiness_rate"] + bar = _pass_bar(pass_rate) + + pct_text = Text(f"{pass_rate*100:.0f}% pass") + if pass_rate > 0.85: + pct_text.stylize("green") + elif pass_rate >= 0.70: + pct_text.stylize("yellow") + else: + pct_text.stylize("red") + + if flakiness > 0.01: + flak_text = Text(f"⚡ flaky {flakiness*100:.0f}%", style="yellow") + else: + flak_text = Text("") + + table.add_row( + Text(gate), + bar, + pct_text, + flak_text, + cb_icon, + ) + + return Panel(table, title="Gate Health (7-day)", border_style="blue") + + +def _render_recent_runs(recent_runs: List[dict]) -> Panel: + table = Table( + box=box.SIMPLE, + show_header=False, + padding=(0, 1), + expand=True, + ) + table.add_column("icon", width=4, no_wrap=True) + table.add_column("run_id", width=12, no_wrap=True, style="dim") + table.add_column("issue", width=12, no_wrap=True, style="bold") + table.add_column("repo", width=14, no_wrap=True, style="dim") + table.add_column("dur", width=8, no_wrap=True) + table.add_column("att", width=6, no_wrap=True) + table.add_column("cost", width=8, no_wrap=True) + table.add_column("label", no_wrap=True) + + if not recent_runs: + table.add_row("", "", Text("No runs yet", style="dim"), "", "", "", "", "") + else: + for run in recent_runs: + outcome = run.get("overall_outcome", "failed") + icon = _outcome_icon(outcome) + style = _outcome_style(outcome) + + short_id = _short_run_id(run.get("run_id", "—")) + issue = run.get("issue_key", "—") + repo = run.get("repository", "—") + dur = _fmt_duration(run.get("duration_ms")) + attempts = run.get("gate_attempts") + att_str = str(attempts) if attempts is not None else "—" + + # cost: placeholder from reward + reward = run.get("reward") + cost_str = f"${reward*1.5:.2f}" if reward is not None else "—" + + # label: human_feedback or needs-review for partial + label = run.get("human_feedback") or "" + if not label and outcome == "partial": + label = "NEEDS-REVIEW" + + label_text = Text(label) + if label == "NEEDS-REVIEW": + label_text.stylize("bold yellow") + + table.add_row( + Text(icon, style=style), + short_id, + Text(issue, style=style), + repo, + dur, + att_str, + cost_str, + label_text, + ) + + return Panel(table, title="Recent Runs (last 10)", border_style="blue") + + +def _render_footer(show_help: bool) -> Panel: + if show_help: + help_text = ( + "[q/Q] quit [r/R] force refresh [w/W] watch run " + "[c/C] cancel run [?] toggle help\n\n" + "watch: streams live events from the run's JSONL file below the dashboard.\n" + "cancel: sends a cancel request to the REST server (requires server running).\n" + "Refresh interval: 2 seconds. Data read directly from provenance/ files." + ) + return Panel(Text(help_text, style="dim"), title="Help", border_style="dim") + return Panel( + Text( + "[q] quit [w] watch run [c] cancel run [r] refresh now [?] help", + style="dim", + ), + border_style="dim", + ) + + +def _render_overlay(text: str) -> Panel: + return Panel(Text(text), title="Input", border_style="bold yellow") + + +def _build_layout(state: dict) -> Layout: + layout = Layout() + layout.split_column( + Layout(name="header", size=3), + Layout(name="top", size=10), + Layout(name="gate", size=10), + Layout(name="recent", size=14), + Layout(name="footer", size=4), + ) + + # Header + ts = state.get("last_refresh") + ts_str = ts.strftime("%H:%M:%S") if ts else "—" + header_text = Text() + header_text.append("🏭 Dark Factory", style="bold white") + header_text.append(f" last refresh: {ts_str}", style="dim") + layout["header"].update(Panel(header_text, border_style="bright_blue")) + + # Top row: active runs + today stats + top = Layout() + top.split_row( + Layout(name="active", ratio=3), + Layout(name="today", ratio=2), + ) + top["active"].update(_render_active_runs(state.get("active_runs", []))) + top["today"].update(_render_today_stats(state.get("today_stats", {}))) + layout["top"].update(top) + + # Gate health + layout["gate"].update( + _render_gate_health( + state.get("gate_health", {}), + state.get("circuit_states", {}), + ) + ) + + # Recent runs + layout["recent"].update(_render_recent_runs(state.get("recent_runs", []))) + + # Footer / help / overlay + overlay = state.get("overlay_text") + if overlay: + layout["footer"].update(_render_overlay(overlay)) + else: + layout["footer"].update(_render_footer(state.get("show_help", False))) + + return layout + + +# --------------------------------------------------------------------------- +# Data refresh thread +# --------------------------------------------------------------------------- + +def _refresh_data(state: dict, lock: threading.Lock) -> None: + """Load all data and update state dict atomically.""" + active_runs = load_active_runs() + today_stats = load_today_stats() + gate_health = load_gate_health(days=7) + circuit_states = load_circuit_states() + recent_runs = load_recent_runs(limit=10) + + with lock: + state["active_runs"] = active_runs + state["today_stats"] = today_stats + state["gate_health"] = gate_health + state["circuit_states"] = circuit_states + state["recent_runs"] = recent_runs + state["last_refresh"] = datetime.datetime.now() + + +def _data_thread(state: dict, lock: threading.Lock) -> None: + """Background thread: refresh data every 2 seconds or on force_refresh event.""" + while True: + with lock: + running = state.get("running", True) + if not running: + break + _refresh_data(state, lock) + # Wait 2s but wake up if force_refresh is set + state["force_refresh"].wait(timeout=2.0) + state["force_refresh"].clear() + + +# --------------------------------------------------------------------------- +# Key input thread +# --------------------------------------------------------------------------- + +def _read_char() -> Optional[str]: + """Read one character from stdin (non-blocking, raw mode).""" + fd = sys.stdin.fileno() + old = termios.tcgetattr(fd) + try: + tty.setraw(fd) + ch = sys.stdin.read(1) + return ch + except Exception: + return None + finally: + termios.tcsetattr(fd, termios.TCSADRAIN, old) + + +def _watch_run(run_id_or_issue: str, console: Console) -> None: + """Stream live events from a run's JSONL file until Enter is pressed.""" + runs_dir = PROV_DIR / "runs" + target_path: Optional[Path] = None + + # Resolve by run_id or issue_key + candidate = runs_dir / f"{run_id_or_issue}.jsonl" + if candidate.exists(): + target_path = candidate + else: + # Search summaries for matching issue_key + for sf in sorted(runs_dir.glob("*.summary.json"), reverse=True): + try: + s = json.loads(sf.read_text()) + except Exception: + continue + if s.get("issue_key", "").lower() == run_id_or_issue.lower(): + run_id_found = s.get("run_id", sf.stem.replace(".summary", "")) + target_path = runs_dir / f"{run_id_found}.jsonl" + break + + if not target_path or not target_path.exists(): + console.print(f"[bold red]Run not found:[/bold red] {run_id_or_issue}") + console.print("Press Enter to return...") + sys.stdin.readline() + return + + console.print(f"[bold cyan]Watching:[/bold cyan] {target_path.name} (press Enter to stop)\n") + stop_event = threading.Event() + + def _tail(): + seen_bytes = 0 + while not stop_event.is_set(): + try: + data = target_path.read_bytes() + except OSError: + time.sleep(0.5) + continue + if len(data) > seen_bytes: + new_data = data[seen_bytes:].decode(errors="replace") + for line in new_data.splitlines(): + line = line.strip() + if not line: + continue + try: + event = json.loads(line) + except json.JSONDecodeError: + console.print(line) + continue + ev_type = event.get("event", "?") + ts = event.get("timestamp", "")[:19].replace("T", " ") + if ev_type == "step": + icon = "✅" if event.get("success") else "❌" + console.print( + f"[dim]{ts}[/dim] {icon} step [bold]{event.get('step')}[/bold] " + f"({_fmt_duration(event.get('duration_ms'))})" + ) + elif ev_type == "gate": + icon = "✅" if event.get("passed") else "❌" + console.print( + f"[dim]{ts}[/dim] {icon} gate [bold]{event.get('gate')}[/bold] " + f"attempt={event.get('attempt')} " + f"({_fmt_duration(event.get('duration_ms'))})" + ) + elif ev_type == "error": + console.print(f"[dim]{ts}[/dim] [bold red]error[/bold red] {event.get('error')}") + elif ev_type == "run_end": + outcome = event.get("overall_outcome", "?") + console.print( + f"[dim]{ts}[/dim] [bold]run_end[/bold] outcome={outcome} " + f"reward={event.get('reward')}" + ) + else: + console.print(f"[dim]{ts}[/dim] {ev_type}") + seen_bytes = len(data) + time.sleep(0.5) + + t = threading.Thread(target=_tail, daemon=True) + t.start() + + # Wait for Enter + try: + sys.stdin.readline() + except Exception: + pass + stop_event.set() + t.join(timeout=1) + console.print("\n[dim]Watch ended. Returning to dashboard...[/dim]") + time.sleep(1) + + +def _cancel_run(run_id_or_issue: str, console: Console) -> None: + """Send a cancel request to the REST server.""" + try: + payload = json.dumps({"run_id": run_id_or_issue, "issue_key": run_id_or_issue}).encode() + req = urllib.request.Request( + f"{SERVER_BASE}/cancel", + data=payload, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req, timeout=2.0) as resp: + body = resp.read().decode() + console.print(f"[bold green]Cancelled:[/bold green] {body}") + except urllib.error.URLError: + console.print("[bold yellow]Server not running[/bold yellow] — cannot cancel remotely.") + console.print("To stop a run manually, kill the harness process.") + + console.print("Press Enter to return...") + sys.stdin.readline() + + +def _key_thread(state: dict, lock: threading.Lock, live: "Live", console: Console) -> None: + """Read keystrokes and update state. Runs in a background daemon thread.""" + while True: + with lock: + if not state.get("running", True): + break + + ch = _read_char() + if ch is None: + continue + + if ch in ("q", "Q"): + with lock: + state["running"] = False + break + + elif ch in ("r", "R"): + state["force_refresh"].set() + + elif ch == "?": + with lock: + state["show_help"] = not state.get("show_help", False) + + elif ch in ("w", "W"): + # Pause live, prompt user, stream events + live.stop() + console.print() + console.print("[bold cyan]Watch run[/bold cyan] — enter run_id or issue_key (blank=cancel): ", end="") + # Restore line mode for input + fd = sys.stdin.fileno() + old = termios.tcgetattr(fd) + termios.tcsetattr(fd, termios.TCSADRAIN, old) + try: + val = sys.stdin.readline().strip() + except Exception: + val = "" + if val: + _watch_run(val, console) + live.start(refresh=True) + + elif ch in ("c", "C"): + live.stop() + console.print() + console.print("[bold yellow]Cancel run[/bold yellow] — enter run_id or issue_key (blank=cancel): ", end="") + fd = sys.stdin.fileno() + old = termios.tcgetattr(fd) + termios.tcsetattr(fd, termios.TCSADRAIN, old) + try: + val = sys.stdin.readline().strip() + except Exception: + val = "" + if val: + _cancel_run(val, console) + live.start(refresh=True) + + +# --------------------------------------------------------------------------- +# Main entry point +# --------------------------------------------------------------------------- + +def main() -> None: + """Launch the TUI. Called by `harness tui` CLI command.""" + console = Console() + + # Initial data load + lock = threading.Lock() + _refresh_data(_state, lock) + + # Start data refresh thread + dt = threading.Thread(target=_data_thread, args=(_state, lock), daemon=True) + dt.start() + + with Live( + _build_layout(_state), + console=console, + refresh_per_second=0.5, + screen=True, + ) as live: + + # Start key-binding thread (needs live handle for pause/resume) + kt = threading.Thread(target=_key_thread, args=(_state, lock, live, console), daemon=True) + kt.start() + + while True: + with lock: + if not _state.get("running", True): + break + + with lock: + layout = _build_layout(_state) + live.update(layout) + time.sleep(0.1) # tight loop; actual data refresh is every 2s + + # Teardown + with lock: + _state["running"] = False + + console.print("\n[bold]Dark Factory TUI exited.[/bold]") + + +if __name__ == "__main__": + main() diff --git a/harness/watchdog.py b/harness/watchdog.py new file mode 100644 index 0000000..afa316b --- /dev/null +++ b/harness/watchdog.py @@ -0,0 +1,240 @@ +"""Watchdog — background thread that alerts and escalates hung harness steps. + +The watchdog runs as a daemon thread alongside the main harness process. It +monitors how long the current step has been running and escalates in two +stages: + +1. **Warn** — calls the caller-supplied ``on_warn`` callback once per step. +2. **Kill** — calls ``on_kill``, then terminates the subprocess (if one was + registered via :meth:`Watchdog.set_step`) and raises an internal flag so + the harness knows the run has been aborted. + +Thresholds (warn → hard kill):: + + research, plan, fix: 20 min warn / 60 min kill + implement: 45 min warn / 90 min kill (implement is longest) + any gate: 15 min warn / 60 min kill + create-pr: 10 min warn / 30 min kill + +Typical usage:: + + import subprocess + from harness.watchdog import Watchdog + + def warn_handler(step: str, elapsed: float) -> None: + print(f"WARNING: step '{step}' has been running for {elapsed/60:.1f} min") + + def kill_handler(step: str, elapsed: float) -> None: + print(f"ERROR: killing step '{step}' after {elapsed/60:.1f} min") + mark_run_failed() + + watchdog = Watchdog(run_id="run_abc", on_warn=warn_handler, on_kill=kill_handler) + watchdog.start() + + proc = subprocess.Popen(["claude", ...]) + watchdog.set_step("research", proc) + proc.wait() + + watchdog.set_step("plan", proc2) + ... + + watchdog.stop() +""" + +import subprocess +import threading +import time +from typing import Callable, Optional + + +# --------------------------------------------------------------------------- +# Threshold tables (seconds) +# --------------------------------------------------------------------------- + +WARN_THRESHOLDS_S: dict = { + "research": 20 * 60, + "plan": 20 * 60, + "fix": 20 * 60, + "implement": 45 * 60, + "create-pr": 10 * 60, + "default": 15 * 60, # gates and any unrecognised step +} + +KILL_THRESHOLDS_S: dict = { + "research": 60 * 60, + "plan": 60 * 60, + "fix": 60 * 60, + "implement": 90 * 60, + "create-pr": 30 * 60, + "default": 60 * 60, +} + + +def _warn_threshold(step: str) -> float: + return WARN_THRESHOLDS_S.get(step, WARN_THRESHOLDS_S["default"]) + + +def _kill_threshold(step: str) -> float: + return KILL_THRESHOLDS_S.get(step, KILL_THRESHOLDS_S["default"]) + + +# --------------------------------------------------------------------------- +# Watchdog class +# --------------------------------------------------------------------------- + +class Watchdog: + """Background thread that monitors and escalates hung harness steps. + + Parameters + ---------- + run_id: + Unique identifier for the current harness run (used in log messages). + on_warn: + Callback invoked once per step when elapsed time exceeds the warn + threshold. Signature: ``(step_name: str, elapsed_s: float) -> None``. + on_kill: + Callback invoked when elapsed time exceeds the kill threshold. The + watchdog kills the subprocess (if set) immediately after this callback + returns. Signature: ``(step_name: str, elapsed_s: float) -> None``. + check_interval: + How often (in seconds) the background thread wakes to check elapsed + time. Default is 30 seconds. + """ + + def __init__( + self, + run_id: str, + on_warn: Callable[[str, float], None], + on_kill: Callable[[str, float], None], + check_interval: int = 30, + ) -> None: + self._run_id = run_id + self._on_warn = on_warn + self._on_kill = on_kill + self._check_interval = check_interval + + # Mutable step state — protected by _lock. + self._lock = threading.Lock() + self._step_name: Optional[str] = None + self._step_started_at: Optional[float] = None + self._proc: Optional[subprocess.Popen] = None + self._warned: bool = False + self._killed: bool = False # True once the hard-kill fires for the current step + + # Thread control + self._stop_event = threading.Event() + self._thread: Optional[threading.Thread] = None + + # ------------------------------------------------------------------ + # Public API + # ------------------------------------------------------------------ + + def start(self) -> None: + """Start the background watchdog thread. + + The thread is marked as a daemon so it does not prevent the process + from exiting if the main thread terminates unexpectedly. + """ + if self._thread is not None and self._thread.is_alive(): + return # already running + + self._stop_event.clear() + self._thread = threading.Thread( + target=self._run, + name=f"watchdog-{self._run_id}", + daemon=True, + ) + self._thread.start() + + def stop(self) -> None: + """Signal the watchdog thread to stop and wait for it to finish. + + Should be called after all steps have completed (or on error cleanup). + Safe to call even if the watchdog was never started. + """ + self._stop_event.set() + if self._thread is not None: + self._thread.join(timeout=self._check_interval + 5) + self._thread = None + + def set_step( + self, + step_name: str, + proc: Optional[subprocess.Popen] = None, + ) -> None: + """Update the step being watched. + + Resets the step timer and clears the warned/killed flags so fresh + thresholds apply to the new step. + + Parameters + ---------- + step_name: + Human-readable step name (e.g. ``"research"``, ``"implement"``). + proc: + The subprocess running this step. If provided and the kill + threshold is exceeded, the watchdog will call ``proc.kill()`` + (SIGKILL) to terminate it immediately. + """ + with self._lock: + self._step_name = step_name + self._step_started_at = time.monotonic() + self._proc = proc + self._warned = False + self._killed = False + + # ------------------------------------------------------------------ + # Background thread + # ------------------------------------------------------------------ + + def _run(self) -> None: + """Main loop executed by the background thread.""" + while not self._stop_event.is_set(): + self._stop_event.wait(timeout=self._check_interval) + if self._stop_event.is_set(): + break + self._check() + + def _check(self) -> None: + """Single iteration of the watchdog check — called every check_interval.""" + with self._lock: + if self._step_name is None or self._step_started_at is None: + return # no active step + + step = self._step_name + elapsed = time.monotonic() - self._step_started_at + warn_at = _warn_threshold(step) + kill_at = _kill_threshold(step) + proc = self._proc + already_warned = self._warned + already_killed = self._killed + + # --- Kill threshold (checked first so we don't warn+kill in same tick) --- + if elapsed >= kill_at and not already_killed: + with self._lock: + self._killed = True + + # Call handler outside the lock to avoid re-entrancy issues. + try: + self._on_kill(step, elapsed) + except Exception: + pass # watchdog must never crash the harness + + # Terminate the subprocess. + if proc is not None: + try: + proc.kill() + except (ProcessLookupError, OSError): + pass # already finished + + return # do not also warn after killing + + # --- Warn threshold --- + if elapsed >= warn_at and not already_warned: + with self._lock: + self._warned = True + + try: + self._on_warn(step, elapsed) + except Exception: + pass diff --git a/orchestrator/__main__.py b/orchestrator/__main__.py deleted file mode 100644 index 06b0228..0000000 --- a/orchestrator/__main__.py +++ /dev/null @@ -1,13 +0,0 @@ -""" -Entry point for running orchestrator as a module. - -Usage: - python -m orchestrator implement ABI-123 - python -m orchestrator knowledge --repo runtime - python -m orchestrator test ABI-123 -""" - -from .cli import main - -if __name__ == '__main__': - main() diff --git a/orchestrator/cli.py b/orchestrator/cli.py deleted file mode 100755 index 502a97c..0000000 --- a/orchestrator/cli.py +++ /dev/null @@ -1,335 +0,0 @@ -#!/usr/bin/env python3 -""" -Orchestrator CLI - Workspace-level orchestration for multi-repository workflows. - -This CLI routes Jira issues to repositories, loads repository-specific knowledge, -and delegates to /autonomous-implement skill for execution. - -Usage: - python -m orchestrator.cli implement ABI-123 - python -m orchestrator.cli implement ABI-123 --repo runtime - python -m orchestrator.cli sprint --jql "sprint in openSprints()" -""" - -import sys -import argparse -from pathlib import Path -import yaml -from typing import List, Optional - -# Add parent directory to path for imports -sys.path.insert(0, str(Path(__file__).parent.parent)) - -from orchestrator.router import Router -from orchestrator.executor import Executor -from orchestrator.knowledge import KnowledgeEngine -from orchestrator import jira_mcp - - -def load_workspace_config() -> dict: - """Load workspace.yaml configuration.""" - workspace_file = Path(__file__).parent.parent / 'workspace.yaml' - - if not workspace_file.exists(): - print(f"❌ Error: workspace.yaml not found at {workspace_file}") - print(" Create workspace.yaml with repository configuration first.") - sys.exit(1) - - with open(workspace_file) as f: - return yaml.safe_load(f) - - -def cmd_implement(args): - """Implement a single Jira issue with workspace-level orchestration.""" - - print(f"\n{'='*60}") - print(f"AI Software Factory - Workspace Orchestrator") - print(f"{'='*60}\n") - - # Load configuration - workspace_config = load_workspace_config() - factory_root = Path(__file__).parent.parent - - # Route to repository (unless explicitly specified) - print(f"📋 Issue: {args.issue_key}") - - if args.repo: - repository = args.repo - print(f"🎯 Repository: {repository} (explicit)") - else: - # Route based on issue key prefix - router = Router(workspace_config) - component = jira_mcp.get_issue_component(args.issue_key) - repository = jira_mcp.get_repository_for_issue( - args.issue_key, - workspace_config.get('jira', {}).get('component_mapping', {}) - ) - - if not repository: - repository = 'runtime' # Default - print(f"🎯 Repository: {repository} (default - unknown component)") - else: - print(f"🎯 Repository: {repository} (routed from {component})") - - print() - - # Initialize executor - executor = Executor(factory_root, workspace_config) - - # Execute in repository - result = executor.execute_single_repo( - issue_key=args.issue_key, - repository=repository - ) - - # Print summary - print(result.summary() if hasattr(result, 'summary') else str(result)) - - # Exit with appropriate code - sys.exit(0 if result.success else 1) - - -def cmd_multi_repo(args): - """Implement issue across multiple repositories.""" - - print(f"\n{'='*60}") - print(f"AI Software Factory - Multi-Repo Orchestrator") - print(f"{'='*60}\n") - - # Load configuration - workspace_config = load_workspace_config() - factory_root = Path(__file__).parent.parent - - # Fetch Jira issue - print(f"📋 Fetching issue: {args.issue_key}") - issue = get_jira_issue(args.issue_key) - print(f" Summary: {issue['summary']}\n") - - # Determine affected repositories - if args.repos: - repositories = args.repos.split(',') - print(f"🎯 Repositories: {', '.join(repositories)} (explicit)") - else: - router = Router(workspace_config) - repositories = router.get_affected_repositories(issue) - print(f"🎯 Repositories: {', '.join(repositories)} (auto-detected)") - - print() - - # Initialize executor - executor = Executor(factory_root, workspace_config) - - # Execute across repositories - result = executor.execute_multi_repo( - issue_key=args.issue_key, - repositories=repositories - ) - - # Print summary - print(result.summary()) - - # Exit with appropriate code - sys.exit(0 if result.overall_success else 1) - - -def cmd_sprint(args): - """Execute multiple issues from a sprint (delegates to /autonomous-sprint).""" - - print(f"\n{'='*60}") - print(f"AI Software Factory - Sprint Orchestrator") - print(f"{'='*60}\n") - - print("🔄 Delegating to /autonomous-sprint skill...") - print(f" JQL: {args.jql}\n") - - # TODO: This should invoke /autonomous-sprint skill with JQL - # For now, just show what would happen - - print("⚠️ Sprint orchestration via CLI not yet implemented.") - print(" Use /autonomous-sprint skill directly instead:") - print(f" /autonomous-sprint --jql \"{args.jql}\"") - - sys.exit(1) - - -def cmd_knowledge(args): - """Display repository knowledge.""" - - workspace_config = load_workspace_config() - factory_root = Path(__file__).parent.parent - - # Initialize knowledge engine - knowledge_root = factory_root / 'knowledge' - knowledge_engine = KnowledgeEngine(str(knowledge_root)) - - if args.list: - # List available repositories - repos = workspace_config.get('repositories', []) - print(f"\n{'='*60}") - print("Available Repositories") - print(f"{'='*60}\n") - for repo in repos: - print(f" - {repo['name']}: {repo.get('display_name', repo['name'])}") - print() - return - - if not args.repo: - print("❌ Error: --repo required (or use --list to see available repos)") - sys.exit(1) - - # Load knowledge for repository - print(f"\n{'='*60}") - print(f"Repository Knowledge: {args.repo}") - print(f"{'='*60}\n") - - knowledge = knowledge_engine.get_repository_knowledge(args.repo) - - for category, content in knowledge.items(): - print(f"\n## {category.upper()}") - print(f"{'-'*60}") - if content: - # Show first 500 chars - preview = content[:500] + ('...' if len(content) > 500 else '') - print(preview) - else: - print("(No content available)") - - print() - - -def cmd_test(args): - """Test orchestrator components without execution.""" - - print(f"\n{'='*60}") - print(f"Orchestrator Component Test") - print(f"{'='*60}\n") - - workspace_config = load_workspace_config() - factory_root = Path(__file__).parent.parent - - # Test router - print("Testing Router...") - router = Router(workspace_config) - test_issue = get_jira_issue(args.issue_key) - repo = router.route_issue(test_issue) - print(f" ✅ Routed {args.issue_key} → {repo}\n") - - # Test knowledge engine - print("Testing Knowledge Engine...") - knowledge_root = factory_root / 'knowledge' - knowledge_engine = KnowledgeEngine(str(knowledge_root)) - knowledge = knowledge_engine.get_repository_knowledge(repo) - print(f" ✅ Loaded knowledge for {repo}:") - for category in knowledge.keys(): - length = len(knowledge[category]) - print(f" - {category}: {length} chars") - print() - - # Test foundations - print("Testing Foundations Standards...") - foundations = knowledge_engine.get_foundations_guidance('standards') - if foundations: - print(f" ✅ Loaded foundations standards: {len(foundations.get('standards', ''))} chars") - else: - print(f" ⚠️ No foundations standards found") - print() - - print("✅ All component tests passed!\n") - - -def main(): - """Main CLI entry point.""" - parser = argparse.ArgumentParser( - description='AI Software Factory - Workspace-level orchestrator', - formatter_class=argparse.RawDescriptionHelpFormatter, - epilog=""" -Examples: - # Implement single issue (auto-route to repository) - python -m orchestrator.cli implement ABI-123 - - # Implement in specific repository - python -m orchestrator.cli implement ABI-123 --repo runtime - - # Implement across multiple repositories - python -m orchestrator.cli multi-repo SDK-456 --repos sdk,runtime,runtime-ui - - # Execute sprint (delegates to /autonomous-sprint) - python -m orchestrator.cli sprint --jql "sprint in openSprints()" - - # View repository knowledge - python -m orchestrator.cli knowledge --repo runtime - python -m orchestrator.cli knowledge --list - - # Test orchestrator components - python -m orchestrator.cli test ABI-123 - """ - ) - - subparsers = parser.add_subparsers(dest='command', help='Commands') - - # implement command - implement = subparsers.add_parser( - 'implement', - help='Implement single Jira issue in one repository' - ) - implement.add_argument('issue_key', help='Jira issue key (e.g., ABI-123)') - implement.add_argument( - '--repo', - help='Explicit repository name (default: auto-route)' - ) - implement.set_defaults(func=cmd_implement) - - # multi-repo command - multi = subparsers.add_parser( - 'multi-repo', - help='Implement issue across multiple repositories' - ) - multi.add_argument('issue_key', help='Jira issue key') - multi.add_argument( - '--repos', - help='Comma-separated repository names (default: auto-detect)' - ) - multi.set_defaults(func=cmd_multi_repo) - - # sprint command - sprint = subparsers.add_parser( - 'sprint', - help='Execute multiple issues from sprint' - ) - sprint.add_argument( - '--jql', - required=True, - help='JQL query for issues' - ) - sprint.set_defaults(func=cmd_sprint) - - # knowledge command - knowledge = subparsers.add_parser( - 'knowledge', - help='View repository knowledge' - ) - knowledge.add_argument('--repo', help='Repository name') - knowledge.add_argument('--list', action='store_true', help='List all repositories') - knowledge.set_defaults(func=cmd_knowledge) - - # test command - test = subparsers.add_parser( - 'test', - help='Test orchestrator components' - ) - test.add_argument('issue_key', help='Jira issue key for testing') - test.set_defaults(func=cmd_test) - - # Parse and execute - args = parser.parse_args() - - if not args.command: - parser.print_help() - sys.exit(1) - - # Execute command - args.func(args) - - -if __name__ == '__main__': - main() diff --git a/provenance/.gitignore b/provenance/.gitignore new file mode 100644 index 0000000..b7bceb0 --- /dev/null +++ b/provenance/.gitignore @@ -0,0 +1,4 @@ +# Provenance run logs — do not commit +runs/ +index.jsonl +rl_dataset.json diff --git a/skills/autonomous-implement/SKILL.md b/skills/autonomous-implement/SKILL.md index 9164921..ae0243c 100644 --- a/skills/autonomous-implement/SKILL.md +++ b/skills/autonomous-implement/SKILL.md @@ -21,7 +21,7 @@ Use this skill to: # Implement single issue /autonomous-implement ABI-123 -# With repository knowledge context (from orchestrator) +# With repository knowledge context (from harness) /autonomous-implement ABI-123 --context-file /tmp/knowledge_context.md # With specific branch (if already created) @@ -48,34 +48,38 @@ Use this skill to: ## Process Flow ``` -┌─────────────────────────────────────────────────────────┐ -│ 1. Fetch Jira Issue │ -│ ↓ │ -│ 2. Create Branch from main/master (CRITICAL) │ -│ ↓ │ -│ 3. Research Codebase (existing /research-codebase) │ -│ ↓ │ -│ 4. Create Plan (existing /create-plan) │ -│ ↓ │ -│ 5. Generate Evals (/eval-generator) │ -│ ↓ │ -│ 6. Implement (existing /implement-plan) │ -│ ↓ │ -│ 7. Run Evals (pytest) │ -│ ├─ PASS → Continue │ -│ └─ FAIL → Retry (max 3 attempts) │ -│ ↓ │ -│ 8. Create PR (existing /create-pr) │ -│ ↓ │ -│ 9. Code Review (existing /code-review) │ -│ ↓ │ -│ 10. Update Jira (/jira-update) │ -└─────────────────────────────────────────────────────────┘ +┌─────────────────────────────────────────────────────────────┐ +│ 1. Fetch Jira Issue │ +│ ↓ │ +│ 2. Create Branch from main/master (CRITICAL) │ +│ ↓ │ +│ 3. Research Codebase (/research-codebase) │ +│ ↓ │ +│ 4. Create Plan (/create-plan) │ +│ ↓ │ +│ 5. Generate Evals (/eval-generator) │ +│ ↓ │ +│ 6. Implement (/implement-plan) │ +│ ↓ │ +│ 7. Verify & Fix (/verify-and-fix) ← unified loop │ +│ ├─ Gate 1: Linter & static analysis │ +│ ├─ Gate 2: Existing tests (no regressions) │ +│ ├─ Gate 3: New evals (acceptance criteria) │ +│ └─ Gate 4: Code review blockers check │ +│ ├─ ALL PASS → Continue │ +│ └─ ANY FAIL → Fix → Retry (max 3 attempts) │ +│ ↓ (exhausted) │ +│ Create PR with [NEEDS-REVIEW] label │ +│ ↓ │ +│ 8. Create PR (/create-pr) │ +│ ↓ │ +│ 9. Update Jira (/jira-update) │ +└─────────────────────────────────────────────────────────────┘ ``` ## Knowledge Context Integration -When invoked by the **workspace orchestrator**, this skill receives a knowledge context file containing: +When invoked by the **workspace harness**, this skill receives a knowledge context file containing: ### Context File Contents @@ -359,61 +363,31 @@ Executes implementation: - Writes initial tests - Updates documentation -### Step 7: Run Evals +### Step 7: Verify & Fix -Execute generated eval tests: +Invoke the `/verify-and-fix` skill. This runs all four verification gates in sequence, retrying with targeted fixes on each failure, up to 3 attempts: ```bash -pytest tests/evals/${issueKey}/ -v --json-report --json-report-file=eval-results.json +/verify-and-fix ${issueKey} ``` -**Parse results:** -```javascript -const evalResults = JSON.parse(readFile('eval-results.json')) - -const summary = { - total: evalResults.summary.total, - passed: evalResults.summary.passed, - failed: evalResults.summary.failed, - duration: evalResults.summary.duration -} -``` +The four gates (in order): +1. **Linter & static analysis** — auto-fix pass first, then manual edits for remaining errors +2. **Existing tests** — full suite excluding `tests/evals/`; no regressions allowed +3. **New evals** — acceptance-criteria tests in `tests/evals/${issueKey}/` +4. **Code review blockers** — parallel review agents; only blockers are fixed (nits ignored) -**If evals fail:** -- Analyze failure reasons -- Attempt fixes (max 3 attempts) -- If still failing after 3 attempts: - - Option A: Create PR with `[NEEDS-REVIEW]` label - - Option B: Escalate to human +**On success** (all gates pass within 3 attempts): proceed to Step 8. -**Retry logic:** -```javascript -let attempt = 1 -const maxAttempts = 3 +**On failure** (all 3 attempts exhausted with remaining failures): +- If `--force-pr` flag: create PR with `[NEEDS-REVIEW]` label and failure summary +- Otherwise: stop, report failures, ask human to intervene -while (attempt <= maxAttempts) { - const results = await runEvals(issueKey) - - if (results.passed === results.total) { - // All passed! - break - } - - if (attempt < maxAttempts) { - // Analyze failures and fix - await analyzeFai lures(results.failures) - await applyFixes(results.failures) - attempt++ - } else { - // Max attempts reached - throw new Error(`Evals failed after ${maxAttempts} attempts`) - } -} -``` +See `/verify-and-fix` skill for full loop logic, fix strategies, and output schema. ### Step 8: Create Pull Request -If evals pass, use existing `/create-pr` skill: +If `/verify-and-fix` passes, use existing `/create-pr` skill: ```bash /create-pr @@ -432,24 +406,21 @@ Closes ABI-123 - Applied rate limiting to all API endpoints - Added rate limit headers to responses -## Eval Results -✓ All acceptance criteria validated - -**Functional Tests:** 4/4 passed -- ✓ Rate limiting enforces 100 req/min -- ✓ Rate limit headers included -- ✓ 429 status on limit exceeded -- ✓ Rate limit resets correctly - -**Performance Tests:** 2/2 passed -- ✓ Latency < 10ms -- ✓ Handles 1000 concurrent users - -**Quality Tests:** 2/2 passed -- ✓ Coverage 85% (target: 80%) -- ✓ Security scan passed - -Total: 8/8 tests passed ✓ +## Verification Results +All gates passed after 2 attempts. + +**Gate 1 — Linter:** ✅ No errors (auto-fixed 3 warnings on attempt 1) +**Gate 2 — Existing Tests:** ✅ 47/47 passing — no regressions +**Gate 3 — Evals (acceptance criteria):** ✅ 8/8 passed + - ✓ Rate limiting enforces 100 req/min + - ✓ Rate limit headers included + - ✓ 429 status on limit exceeded + - ✓ Rate limit resets correctly + - ✓ Latency < 10ms + - ✓ Handles 1000 concurrent users + - ✓ Coverage 85% (target: 80%) + - ✓ Security scan passed +**Gate 4 — Code Review:** ✅ Approve — no blockers ## Testing \`\`\`bash @@ -457,31 +428,20 @@ pytest tests/evals/ABI-123/ -v \`\`\` ``` -**If evals failed (and --force-pr used):** +**If `/verify-and-fix` failed (and --force-pr used):** - Add `[NEEDS-REVIEW]` label -- Include failure details in PR description -- Request manual review - -### Step 9: Automated Code Review - -Run existing `/code-review` skill on the PR: - -```bash -/code-review -``` - -Posts review comments: -- Critical issues (bugs, security) -- Warnings (code smells, performance) -- Suggestions (style, readability) +- Include per-gate failure details in PR description +- List fixes applied across all attempts +- Request manual review for remaining failures -### Step 10: Update Jira +### Step 9: Update Jira Use `/jira-update` to sync status: ```bash /jira-update ${issueKey} \ --pr-url ${prUrl} \ + --verification-passed ${verificationPassed} \ --evals-passed ${evalsPassed} \ --evals-total ${evalsTotal} \ --status "In Review" @@ -493,12 +453,15 @@ Adds comment to Jira: **Pull Request:** [PR #789](https://github.com/org/repo/pull/789) -**Eval Results:** 8/8 passed ✓ +**Verification Results (2 attempts):** +- ✅ Linter: No errors +- ✅ Existing Tests: 47/47 +- ✅ Evals: 8/8 passed +- ✅ Code Review: Approve All acceptance criteria validated through automated tests. **Next Steps:** -- Code review in progress - Merge after approval ``` @@ -514,12 +477,15 @@ Timeline: ✓ Created implementation plan (45s) ✓ Generated evals (8s) ✓ Implemented solution (3m 24s) - ✓ Ran evals - 8/8 passed (15s) + ✓ Verify & Fix — all gates passed on attempt 2/3 (1m 48s) + ✓ Linter: No errors (auto-fixed 3 warnings on attempt 1) + ✓ Existing tests: 47/47 + ✓ Evals: 8/8 passed + ✓ Code review: Approve — no blockers ✓ Created PR #789 (5s) - ✓ Automated code review (22s) ✓ Updated Jira (3s) -Total time: 4 minutes 34 seconds +Total time: 6 minutes 22 seconds **Pull Request:** https://github.com/EmergenceAI/em-talk2data/pull/789 **Jira Issue:** https://company.atlassian.net/browse/ABI-123 @@ -527,7 +493,7 @@ Total time: 4 minutes 34 seconds Status: Ready for human review and merge ``` -### Partial Success (Evals Failed) +### Partial Success (Verification Failed) ```markdown ⚠ Partially implemented ABI-123: Add API Rate Limiting @@ -537,21 +503,23 @@ Timeline: ✓ Created implementation plan (45s) ✓ Generated evals (8s) ✓ Implemented solution (3m 24s) - ⚠ Ran evals - 6/8 passed (2 failed) (15s) - ⚠ Retried fixes - 7/8 passed (1 failed) (1m 30s) - ⚠ Max retry attempts reached + ⚠ Verify & Fix — exhausted 3/3 attempts (4m 12s) + ✓ Linter: No errors + ✓ Existing tests: 47/47 + ⚠ Evals: 7/8 passed (1 failed) + ✓ Code review: Approve — no blockers ✓ Created PR #789 with [NEEDS-REVIEW] label (5s) ✓ Updated Jira (3s) -Total time: 6 minutes 22 seconds +Total time: 9 minutes 01 second **Pull Request:** https://github.com/EmergenceAI/em-talk2data/pull/789 **Jira Issue:** https://company.atlassian.net/browse/ABI-123 -**Failed Evals:** -- test_concurrent_users_performance: System degraded under 1000 users +**Remaining Failures:** +- Gate 3 (Evals): test_concurrent_users_performance — system degraded under 1000 users -Status: Needs human review to address eval failures +Status: Needs human review to address remaining failures ``` ### Failure (Cannot Proceed) @@ -586,37 +554,31 @@ pytest tests/evals/ABI-123/ -v "plan": { "duration": 45, "status": "completed" }, "evalGen": { "duration": 8, "status": "completed" }, "implement": { "duration": 204, "status": "completed" }, - "evals": { - "duration": 15, + "verifyAndFix": { + "duration": 108, "status": "completed", - "passed": 8, - "failed": 0, - "total": 8 + "passed": true, + "attempts": 2, + "gateResults": { + "linter": { "passed": true, "autoFixed": 3, "remainingErrors": 0 }, + "existingTests": { "passed": true, "total": 47, "failures": [] }, + "evals": { "passed": 8, "failed": 0, "total": 8 }, + "codeReview": { "verdict": "Approve", "blockers": [] } + } }, - "pr": { - "duration": 5, + "pr": { + "duration": 5, "status": "completed", "number": 789, "url": "https://github.com/org/repo/pull/789" }, - "review": { "duration": 22, "status": "completed" }, "jiraUpdate": { "duration": 3, "status": "completed" } }, - "totalDuration": 274, + "totalDuration": 385, "pr": { "number": 789, "url": "https://github.com/org/repo/pull/789", "status": "open" - }, - "evalResults": { - "passed": 8, - "failed": 0, - "total": 8, - "categories": { - "functional": { "passed": 4, "total": 4 }, - "performance": { "passed": 2, "total": 2 }, - "quality": { "passed": 2, "total": 2 } - } } } ``` @@ -633,20 +595,20 @@ JIRA_API_TOKEN=xxx # GitHub GITHUB_TOKEN=ghp_xxx -# Eval settings -EVAL_RETRY_LIMIT=3 -EVAL_TIMEOUT=300 # 5 minutes -FORCE_PR_ON_EVAL_FAILURE=false +# Verification loop settings (passed through to /verify-and-fix) +VERIFY_RETRY_LIMIT=3 # Max full-loop attempts (default: 3) +VERIFY_GATES=linter,tests,evals,review # Gates to run (default: all) +EVAL_TIMEOUT=300 # Seconds before eval run times out (default: 300) +FORCE_PR_ON_EVAL_FAILURE=false # Renamed from FORCE_PR_ON_EVAL_FAILURE; applies to any gate ``` **Skill-specific settings:** ```json { "autonomous-implement": { - "maxRetries": 3, + "verifyRetryLimit": 3, "evalTimeout": 300, "forcePrOnFailure": false, - "skipCodeReview": false, "autoTransitionJira": true, "targetStatus": "In Review" } @@ -677,21 +639,22 @@ Recommendation: 2. Retry: /autonomous-implement ABI-123 ``` -**Eval failures after max retries:** +**Verification failures after max retries:** ``` -Warning: Evals failed after 3 attempts +Warning: /verify-and-fix exhausted 3 attempts with remaining failures -Failed tests: -- test_concurrent_users_performance (performance degradation) +Failed gates (final attempt): +- Gate 3 (Evals): test_concurrent_users_performance — performance degradation +- (Gates 1, 2, 4 all passed) Actions taken: - Created PR #789 with [NEEDS-REVIEW] label -- Added failure details to PR description +- Added per-gate failure details and all fixes applied to PR description - Updated Jira with partial completion status Next steps: - Review performance issue manually -- Fix and re-run evals +- Fix and re-run: /verify-and-fix ABI-123 --gates evals - Update PR when passing ``` @@ -713,10 +676,11 @@ const results = await pipeline( ## Success Criteria - [x] Composes existing skills correctly -- [x] Generates and validates evals -- [x] Retries on eval failures -- [x] Creates PR only when evals pass (or with warning) -- [x] Updates Jira with full context +- [x] Generates evals from acceptance criteria +- [x] Runs all 4 verification gates via `/verify-and-fix` before PR creation +- [x] Retries with targeted fixes on any gate failure (max 3 attempts) +- [x] Creates PR only when all gates pass (or with `[NEEDS-REVIEW]` label on failure) +- [x] Updates Jira with full verification context - [x] Handles errors gracefully - [x] Provides clear progress updates - [x] Returns structured results @@ -732,14 +696,13 @@ const results = await pipeline( - `/research-codebase` - Understanding context - `/create-plan` - Tech spec generation - `/implement-plan` - Code implementation +- `/verify-and-fix` - Comprehensive verification loop (linter + tests + evals + code review) - `/create-pr` - PR creation -- `/code-review` - Automated review **New components:** - `/eval-generator` - Test generation -- Eval execution - pytest runner +- `/verify-and-fix` - Unified 4-gate retry loop (replaces inline eval-only retry + standalone code review step) - `/jira-update` - Jira synchronization -- Retry logic - Automated fix attempts **Performance:** - Typical time: 3-8 minutes per issue diff --git a/skills/fix-failures/SKILL.md b/skills/fix-failures/SKILL.md new file mode 100644 index 0000000..6e30df1 --- /dev/null +++ b/skills/fix-failures/SKILL.md @@ -0,0 +1,180 @@ +--- +name: fix-failures +description: Read gate failure context from --failures-file and apply targeted fixes per gate type +--- + +# Fix Failures + +Read gate failure context written by the harness and apply targeted fixes — one fix strategy per failed gate type. Write a fix summary to `.harness-results/fix-{attempt}.json` so the harness can log what was changed. + +## Usage + +```bash +# Called by the harness after a gate loop attempt fails +/fix-failures --failures-file .harness-results/failures-1.json + +# Called with explicit attempt number +/fix-failures --failures-file .harness-results/failures-2.json --attempt 2 +``` + +### Parameters + +- `--failures-file ` (required): Path to the failures JSON written by the harness. Contains an array of `{ gate, outputs }` objects. +- `--attempt `: Attempt number (default: 1). Used to name the fix summary file. + +--- + +## Failures File Format + +```json +[ + { + "gate": "linter", + "outputs": { + "passed": false, + "errors": ["src/foo.py:42: F401 'os' imported but unused"], + "auto_fixed": 2, + "remaining_errors": 1 + } + }, + { + "gate": "evals", + "outputs": { + "passed": false, + "failures": [ + { + "nodeid": "tests/evals/ABI-123/test_functional.py::test_rate_limit_enforced", + "acceptance_criterion": "Rate limiting enforces 100 req/min", + "message": "AssertionError: got 200 OK, expected 429" + } + ] + } + } +] +``` + +--- + +## Instructions + +### Step 1: Read failures file + +```bash +failures = JSON.parse(readFile(args['failures-file'])) +``` + +### Step 2: Apply fixes in gate order + +Process each failed gate in this order, regardless of the order they appear in the file: + +**Order: linter → tests → evals → code-review** + +#### Fix: linter + +1. Read each error — note file path and line number. +2. Run auto-fix first: + ```bash + ruff check --fix . + black . + isort . + ``` +3. For errors that remain after auto-fix: read the file at the specified line, understand the issue, apply a targeted edit. +4. Record each fix in `fixes_applied`. + +#### Fix: tests + +1. For each failing test: + a. Read the test file to understand what it expects. + b. Read the implementation file(s) the test exercises. + c. Determine: is the test correct and the implementation wrong, or did the implementation intentionally change behaviour? + d. Fix the implementation. Do NOT delete tests. + e. If the implementation intentionally changed behaviour (and it was approved in the plan), update the test assertion and add a comment explaining why. +2. Record each fix. + +#### Fix: evals + +1. For each failing eval: + a. Read the eval test to extract the acceptance criterion. + b. Read the implementation code that should satisfy it. + c. Fix the implementation. **Never edit eval test files.** + d. Record what was changed and why. +2. Do NOT modify eval tests under any circumstance. + +#### Fix: code-review + +1. For each blocker: + a. Read `file` + `line` from the blocker object. + b. Read the code at that location. + c. Apply the fix described in `description`. + d. Ignore suggestions — only fix blockers. +2. Record each fix. + +### Step 3: Write fix summary + +Write to `.harness-results/fix-{attempt}.json`: + +```json +{ + "attempt": 1, + "gates_addressed": ["linter", "evals"], + "fixes_applied": [ + { + "gate": "linter", + "file": "src/foo.py", + "line": 42, + "description": "Removed unused 'os' import" + }, + { + "gate": "evals", + "file": "src/api/rate_limiter.py", + "description": "Fixed rate limit check to use per-minute window instead of per-second" + } + ], + "files_modified": ["src/foo.py", "src/api/rate_limiter.py"], + "skipped": [], + "notes": "Linter auto-fix resolved 2 of 3 issues; 1 required manual edit." +} +``` + +### Step 4: Print summary + +``` +🔧 Fix attempt 1 + Gates addressed: linter, evals + Fixes applied: 2 + - src/foo.py:42 — removed unused import [linter] + - src/api/rate_limiter.py — fixed rate limit window [evals] + Files modified: 2 +``` + +--- + +## Output Schema + +```json +{ + "attempt": 1, + "gates_addressed": ["linter", "evals"], + "fixes_applied": [ + { + "gate": "linter | tests | evals | code-review", + "file": "src/...", + "line": 42, + "description": "what was changed and why" + } + ], + "files_modified": ["src/..."], + "skipped": ["tests/evals/ABI-123/test_functional.py"], + "notes": "optional free-text" +} +``` + +--- + +## Invariants + +- **Never modify eval test files** — they encode acceptance criteria. +- **Never delete tests** — fix the implementation instead. +- **Fix blockers only** from the code-review gate — do not address suggestions. +- **Auto-fix before manual edit** for the linter gate. +- If a fix cannot be determined (e.g., a performance issue requires architectural input), log it in `skipped` and note the reason. The harness will surface it in the failure summary. diff --git a/skills/run-code-review/SKILL.md b/skills/run-code-review/SKILL.md new file mode 100644 index 0000000..c40496f --- /dev/null +++ b/skills/run-code-review/SKILL.md @@ -0,0 +1,164 @@ +--- +name: run-code-review +description: Run a focused code review for blockers only and write a structured JSON result to --output +--- + +# Run Code Review + +Run a focused parallel code review on the current branch diff, extracting only **blockers** (bugs, security issues, linter errors, failing tests). Write a structured JSON result to `--output` for the harness. + +This is a harness-optimised variant of `/code-review`. Suggestions and nits are collected but do not affect the `passed` verdict. + +## Usage + +```bash +# Called by the harness +/run-code-review --output .harness-results/code-review.json + +# Standalone (prints full review, no output file) +/run-code-review +``` + +### Parameters + +- `--output `: File path to write the JSON result. + +--- + +## Instructions + +### Step 1: Determine diff scope + +```bash +# On a feature branch — review all changes vs main +git diff main...HEAD + +# On main — review staged changes +git diff --staged + +# Fallback — latest commit +git show HEAD +``` + +### Step 2: Assess diff size and launch parallel agents + +**Small diff** (≤100 lines / ≤3 files): launch agents 1–4 only. +**Medium/large diff** (>100 lines or >3 files): launch all agents. + +Launch in a single parallel call: + +- **Agent 1 — Test & Linter Runner**: run tests and linter for changed files; report failures +- **Agent 2 — Code Reviewer**: up to 5 issues ranked by impact; focus on bugs and incorrect logic +- **Agent 3 — Security Reviewer**: injection, auth, secrets, error-handling leaks +- **Agent 4 — Quality Reviewer**: complexity, dead code, architectural pattern violations +- **Agent 5 — Performance Reviewer** *(large diff only)*: N+1, blocking ops, memory leaks +- **Agent 6 — Dependency Reviewer** *(large diff only)*: new deps, API contract breaks, deployment safety + +### Step 3: Triage findings into blockers vs suggestions + +**Blockers** (anything that must be fixed before merge): +- Bugs or logic errors that fail in production +- Security vulnerabilities with a realistic exploit path +- Linter errors (not warnings) +- Failing tests surfaced by Agent 1 +- Clear architectural violations from the project style guide + +**Suggestions**: everything else (style, minor performance, nice-to-haves). + +### Step 4: Write JSON result to --output + +```json +{ + "passed": true, + "verdict": "Approve", + "blockers": [], + "suggestions": [ + { + "severity": "MED", + "category": "quality", + "title": "Redundant null check", + "file": "src/api/handler.py", + "line": 88, + "description": "Variable is never null at this point — check is dead code." + } + ], + "agents_run": ["test-runner", "code-reviewer", "security-reviewer", "quality-reviewer"], + "diff_stats": { "files_changed": 3, "lines_added": 87, "lines_removed": 12 } +} +``` + +Failure example: + +```json +{ + "passed": false, + "verdict": "Needs Work", + "blockers": [ + { + "severity": "HIGH", + "category": "security", + "title": "SQL injection via unsanitised input", + "file": "src/db/queries.py", + "line": 34, + "description": "user_id is interpolated directly into the SQL string. Use parameterised queries." + }, + { + "severity": "HIGH", + "category": "correctness", + "title": "Off-by-one in pagination", + "file": "src/api/list.py", + "line": 71, + "description": "offset = page * limit should be offset = (page - 1) * limit; page 1 skips first item." + } + ], + "suggestions": [], + "agents_run": ["test-runner", "code-reviewer", "security-reviewer", "quality-reviewer"], + "diff_stats": { "files_changed": 5, "lines_added": 210, "lines_removed": 45 } +} +``` + +**`passed` is `true` only when `blockers` is empty.** + +### Step 5: Print summary + +``` +🔎 Code review (4 agents) + Blockers: 0 | Suggestions: 1 + Verdict: ✅ Approve +``` + +--- + +## Output Schema + +```json +{ + "passed": true | false, + "verdict": "Approve | Needs Work", + "blockers": [ + { + "severity": "HIGH | MED | LOW", + "category": "security | correctness | performance | quality", + "title": "...", + "file": "src/...", + "line": 42, + "description": "..." + } + ], + "suggestions": [ ... ], + "agents_run": ["..."], + "diff_stats": { "files_changed": 0, "lines_added": 0, "lines_removed": 0 } +} +``` + +## Calibration + +- **Only blockers affect `passed`** — suggestions are recorded but don't gate the loop. +- Calibrate severity as a senior engineer would: internal tooling, controlled inputs, and low-blast-radius code get lower severity than user-facing authentication or payment flows. +- Do not include nits or style preferences in blockers. + +## Exit Behaviour + +- Exit 0 when `passed == true`. +- Exit 1 when `passed == false`. +- Always write the result file. diff --git a/skills/run-evals/SKILL.md b/skills/run-evals/SKILL.md new file mode 100644 index 0000000..fbf6132 --- /dev/null +++ b/skills/run-evals/SKILL.md @@ -0,0 +1,164 @@ +--- +name: run-evals +description: Run acceptance-criteria eval tests for a Jira issue and write a structured JSON result to --output +--- + +# Run Evals + +Run the acceptance-criteria eval tests generated for a specific Jira issue (`tests/evals/{issueKey}/`). Write a structured JSON result to `--output` for the harness. + +## Usage + +```bash +# Called by the harness +/run-evals ABI-123 --output .harness-results/evals.json + +# Standalone +/run-evals ABI-123 +``` + +### Parameters + +- `issue_key` (required): Jira issue key — used to locate `tests/evals/{issueKey}/` +- `--output `: File path to write the JSON result. + +--- + +## Instructions + +### Step 1: Locate eval directory + +```bash +eval_dir = "tests/evals/${issueKey}" + +if not exists(eval_dir): + # Write "no evals found" result and exit 0 (not a failure) + write_result({ + "passed": true, + "skipped": true, + "reason": f"No eval directory found at {eval_dir}", + "total": 0, + "passed_count": 0, + "failed_count": 0, + "failures": [] + }) + exit(0) +``` + +### Step 2: Run evals + +```bash +pytest tests/evals/${issueKey}/ -v \ + --json-report \ + --json-report-file=.harness-results/eval-report.json \ + 2>&1 +``` + +If `--json-report` is not available: + +```bash +pytest tests/evals/${issueKey}/ -v 2>&1 +``` + +### Step 3: Parse results + +Same parsing logic as `/run-tests` — use JSON report if available, fall back to stdout parsing. + +**Critical rule:** eval tests represent acceptance criteria. Their text is the ground truth. + +```javascript +const failures = report.tests + .filter(t => t.outcome === 'failed') + .map(t => ({ + nodeid: t.nodeid, + acceptance_criterion: extractCriterion(t.nodeid), // infer from test name + message: t.call?.longrepr || 'unknown', + })) +``` + +### Step 4: Write JSON result to --output + +```json +{ + "passed": true, + "skipped": false, + "issue_key": "ABI-123", + "eval_dir": "tests/evals/ABI-123", + "total": 8, + "passed_count": 8, + "failed_count": 0, + "failures": [], + "command": "pytest tests/evals/ABI-123/ -v" +} +``` + +Failure example: + +```json +{ + "passed": false, + "skipped": false, + "issue_key": "ABI-123", + "eval_dir": "tests/evals/ABI-123", + "total": 8, + "passed_count": 6, + "failed_count": 2, + "failures": [ + { + "nodeid": "tests/evals/ABI-123/test_functional.py::test_rate_limit_enforced", + "acceptance_criterion": "Rate limiting enforces 100 req/min", + "message": "AssertionError: got 200 OK, expected 429 Too Many Requests" + }, + { + "nodeid": "tests/evals/ABI-123/test_performance.py::test_latency_under_10ms", + "acceptance_criterion": "Latency < 10ms", + "message": "AssertionError: mean latency 18.4ms > 10ms threshold" + } + ], + "command": "pytest tests/evals/ABI-123/ -v" +} +``` + +**`passed` is `true` only when `failed_count == 0`.** + +### Step 5: Print summary + +``` +🎯 Evals: tests/evals/ABI-123/ (8 tests) + Total: 8 | Passed: 8 | Failed: 0 + Result: ✅ PASS +``` + +--- + +## Output Schema + +```json +{ + "passed": true | false, + "skipped": false, + "issue_key": "ABI-123", + "eval_dir": "tests/evals/ABI-123", + "total": 8, + "passed_count": 8, + "failed_count": 0, + "failures": [ + { + "nodeid": "...", + "acceptance_criterion": "...", + "message": "..." + } + ], + "command": "pytest tests/evals/ABI-123/ -v" +} +``` + +## Invariant + +**Never modify eval tests.** If an eval fails, the fix belongs in the implementation, not in the test file. + +## Exit Behaviour + +- Exit 0 when `passed == true` (including the `skipped == true` case). +- Exit 1 when `passed == false`. +- Always write the result file. diff --git a/skills/run-linter/SKILL.md b/skills/run-linter/SKILL.md new file mode 100644 index 0000000..05fa587 --- /dev/null +++ b/skills/run-linter/SKILL.md @@ -0,0 +1,140 @@ +--- +name: run-linter +description: Run linter and static analysis on changed files; write structured JSON result to --output file +--- + +# Run Linter + +Run linter and static analysis on the current branch's changed files. Write a structured JSON result to the path specified by `--output` so the harness can read pass/fail deterministically. + +## Usage + +```bash +# Called by the harness — always provide --output +/run-linter --output .harness-results/linter.json + +# Standalone (omit --output to just print results) +/run-linter +``` + +### Parameters + +- `--output `: File path to write the JSON result. Required when called by the harness. + +--- + +## Instructions + +### Step 1: Detect project type and linter command + +Check `Makefile`, `pyproject.toml`, `package.json`, `.github/workflows/` in that order. + +```bash +# Python — check for make target first +if Makefile has "check" or "lint" target: + make check # or make lint +else: + ruff check . + black --check . + isort --check-only . + +# JavaScript / TypeScript +npm run lint +# or: eslint src/ --max-warnings=0 + +# Go +golangci-lint run + +# Catch-all +make lint # if defined +``` + +### Step 2: Auto-fix pass (Python only) + +Before recording failures, attempt auto-fix: + +```bash +ruff check --fix . +black . +isort . +``` + +Re-run the linter after auto-fix to measure what remains. + +### Step 3: Collect results + +Parse linter output: + +``` +errors = lines matching ERROR or error-level diagnostics +warnings = lines matching WARNING (allowed — do not block) +auto_fixed = count of issues resolved by the auto-fix pass +remaining_errors = count of errors after auto-fix +``` + +### Step 4: Write JSON result to --output + +Write the result file **before** printing anything to stdout. + +```json +{ + "passed": true, + "tool": "ruff + black + isort", + "auto_fixed": 3, + "remaining_errors": 0, + "remaining_warnings": 2, + "errors": [], + "warnings": [ + "src/foo.py:12: line too long (121 > 120)" + ] +} +``` + +**`passed` is `true` only when `remaining_errors == 0`.** Warnings do not block. + +Failure example: + +```json +{ + "passed": false, + "tool": "ruff + black + isort", + "auto_fixed": 2, + "remaining_errors": 1, + "remaining_warnings": 0, + "errors": [ + "src/api/handler.py:42: F401 'os' imported but unused" + ], + "warnings": [] +} +``` + +### Step 5: Print summary to stdout + +``` +🔍 Linter: ruff + black + isort + Auto-fixed: 3 issues + Errors: 0 | Warnings: 2 + Result: ✅ PASS +``` + +--- + +## Output Schema + +```json +{ + "passed": true | false, + "tool": "string — linter(s) used", + "auto_fixed": 0, + "remaining_errors": 0, + "remaining_warnings": 0, + "errors": ["file:line: code message", ...], + "warnings": ["file:line: code message", ...] +} +``` + +## Exit Behaviour + +- Exit 0 when `passed == true` (harness reads exit code as secondary signal). +- Exit 1 when `passed == false`. +- Always write the result file even on failure. diff --git a/skills/run-tests/SKILL.md b/skills/run-tests/SKILL.md new file mode 100644 index 0000000..840b34e --- /dev/null +++ b/skills/run-tests/SKILL.md @@ -0,0 +1,162 @@ +--- +name: run-tests +description: Run the existing test suite (excluding evals/) and write a structured JSON result to --output +--- + +# Run Tests + +Run the project's existing test suite on the current branch, excluding the `tests/evals/` directory (which is handled by `/run-evals`). Write a structured JSON result to `--output` for the harness. + +## Usage + +```bash +# Called by the harness +/run-tests --output .harness-results/tests.json + +# Standalone +/run-tests +``` + +### Parameters + +- `--output `: File path to write the JSON result. + +--- + +## Instructions + +### Step 1: Discover test command + +Check in this order: + +```bash +# 1. Makefile +grep -E "^test:" Makefile → make test + +# 2. pyproject.toml (Python) +[tool.pytest.ini_options] → uv run pytest or pytest + +# 3. package.json (JS/TS) +"scripts": { "test": "..." } → npm test + +# 4. go.mod (Go) + → go test ./... + +# 5. Cargo.toml (Rust) + → cargo test + +# 6. CI workflow +.github/workflows/*.yml → extract test step command +``` + +### Step 2: Run tests, excluding evals + +```bash +# Python — exclude evals directory +pytest --ignore=tests/evals/ -v \ + --json-report \ + --json-report-file=.harness-results/pytest-report.json \ + 2>&1 + +# If --json-report plugin not installed, run without it and parse stdout +pytest --ignore=tests/evals/ -v 2>&1 + +# JS/TS +npm test -- --testPathIgnorePatterns=evals + +# Go / Rust +go test ./... # evals not applicable +``` + +### Step 3: Parse results + +From pytest JSON report (if available): + +```javascript +const report = JSON.parse(readFile('.harness-results/pytest-report.json')) +const passed = report.summary.passed +const failed = report.summary.failed +const total = report.summary.total +const failures = report.tests + .filter(t => t.outcome === 'failed') + .map(t => ({ + nodeid: t.nodeid, + message: t.call?.longrepr || t.longrepr || 'unknown error', + })) +``` + +From stdout (fallback): + +- Count lines matching `PASSED`, `FAILED`, `ERROR` +- Extract test names and error messages from failure blocks + +### Step 4: Write JSON result to --output + +```json +{ + "passed": true, + "total": 47, + "passed_count": 47, + "failed_count": 0, + "error_count": 0, + "failures": [], + "command": "pytest --ignore=tests/evals/ -v" +} +``` + +Failure example: + +```json +{ + "passed": false, + "total": 47, + "passed_count": 45, + "failed_count": 2, + "error_count": 0, + "failures": [ + { + "nodeid": "tests/services/test_auth.py::test_token_expiry", + "message": "AssertionError: expected 401, got 200" + }, + { + "nodeid": "tests/api/test_health.py::test_ready_endpoint", + "message": "ConnectionRefusedError: [Errno 111] Connection refused" + } + ], + "command": "pytest --ignore=tests/evals/ -v" +} +``` + +**`passed` is `true` only when `failed_count == 0 AND error_count == 0`.** + +### Step 5: Print summary + +``` +🧪 Existing tests: pytest --ignore=tests/evals/ + Total: 47 | Passed: 47 | Failed: 0 + Result: ✅ PASS +``` + +--- + +## Output Schema + +```json +{ + "passed": true | false, + "total": 47, + "passed_count": 47, + "failed_count": 0, + "error_count": 0, + "failures": [ + { "nodeid": "...", "message": "..." } + ], + "command": "pytest ..." +} +``` + +## Exit Behaviour + +- Exit 0 when `passed == true`. +- Exit 1 when `passed == false`. +- Always write the result file. diff --git a/skills/verify-and-fix/SKILL.md b/skills/verify-and-fix/SKILL.md new file mode 100644 index 0000000..4c75478 --- /dev/null +++ b/skills/verify-and-fix/SKILL.md @@ -0,0 +1,429 @@ +--- +name: verify-and-fix +description: Run all verification gates (linter, existing tests, evals, code review) in a retry loop until everything passes or max attempts is reached +--- + +# Verify and Fix + +Run a comprehensive, ordered verification loop that checks linter/static analysis, existing tests, new acceptance-criteria evals, and code review blockers — retrying with targeted fixes on each failure until all gates pass or the attempt limit is reached. + +This is a **reusable primitive**. It is invoked by `/autonomous-implement` after implementation, but can also be called standalone at any point during development. + +## When to Use This Skill + +- After implementing a feature to verify everything is green before opening a PR +- As a standalone check on a work-in-progress branch +- From other skills that need a verified-clean state before proceeding (e.g., `/autonomous-sprint`, `/autonomous-implement`) + +## Usage + +```bash +# Full loop — all 4 gates, max 3 attempts +/verify-and-fix ABI-123 + +# Run only specific gates +/verify-and-fix ABI-123 --gates linter,tests + +# Override retry limit +/verify-and-fix ABI-123 --max-attempts 5 + +# Skip eval gate (no evals generated yet) +/verify-and-fix ABI-123 --skip-evals +``` + +### Parameters + +- `issue_key` (required): Jira issue key — used to locate eval tests at `tests/evals/{issueKey}/` +- `--gates `: Comma-separated subset of gates to run: `linter`, `tests`, `evals`, `review`. Default: all four. +- `--max-attempts `: Override retry limit. Default: `VERIFY_RETRY_LIMIT` env var (default 3). +- `--skip-evals`: Skip Gate 3 (eval tests). Implies only gates 1, 2, 4 run. + +--- + +## Process Flow + +``` +┌─────────────────────────────────────────────────────────────┐ +│ attempt = 1 (max 3) │ +│ │ +│ Gate 1: Linter & Static Analysis │ +│ ↓ PASS │ +│ Gate 2: Existing Tests (full suite, no regressions) │ +│ ↓ PASS │ +│ Gate 3: New Evals (tests/evals/{issueKey}/) │ +│ ↓ PASS │ +│ Gate 4: Code Review Blockers Check │ +│ ↓ PASS │ +│ ALL PASS → return { passed: true, attempts: N } │ +│ │ +│ ANY FAIL → applyTargetedFixes(failures) → attempt++ │ +│ └─ attempt > maxAttempts → return { passed: false, ... } │ +└─────────────────────────────────────────────────────────────┘ +``` + +**Gate order rationale:** Linter errors are cheapest to fix and can cause test failures (import errors, syntax errors). Regressions in existing tests must be resolved before running new evals to avoid false positives. Code review is the most expensive gate (spawns parallel agents), so it runs last — only when the fast gates are already green. + +--- + +## Detailed Gate Specs + +### Gate 1: Linter & Static Analysis + +**What to run** (check project config first — `Makefile`, `pyproject.toml`, `package.json`, `.github/workflows/`): + +```bash +# Python projects +make check # if Makefile target exists +# or directly: +ruff check . +black --check . +isort --check-only . + +# TypeScript/JavaScript projects +npm run lint +# or: eslint src/ + +# Go projects +golangci-lint run + +# General +make lint # if defined +``` + +**Auto-fix pass** (run before declaring failure): + +```bash +# Python +ruff check --fix . +black . +isort . + +# JS/TS +npm run lint -- --fix +eslint src/ --fix +``` + +**Gate passes when:** Zero linter errors (warnings are allowed but logged). + +**Gate fails when:** Any linter error remains after the auto-fix pass. + +--- + +### Gate 2: Existing Tests + +**What to run:** The full existing test suite, excluding the new evals directory (`tests/evals/`). + +```bash +# Python projects +make test # if Makefile target exists +pytest --ignore=tests/evals/ -v # or directly +uv run pytest --ignore=tests/evals/ -v # if using uv + +# JavaScript/TypeScript +npm test +npm run test:unit + +# Go +go test ./... +``` + +**Gate passes when:** All pre-existing tests pass. New tests added as part of the implementation may fail here — treat them as implementation failures (fix the code, not the test). + +**Gate fails when:** Any test that existed before this branch's changes is now failing (regression), OR any new test added during implementation is failing. + +**Fix strategy:** +1. Read the failing test file and the implementation file it tests. +2. Determine: is the test correct and the implementation wrong, or did the implementation intentionally change behavior? +3. If regression: fix the implementation to restore correct behavior. +4. If intentional behavior change: update the test to match the new contract AND add a comment explaining why. +5. Do NOT delete tests to make them pass. + +--- + +### Gate 3: New Evals (Acceptance Criteria Tests) + +**What to run:** Only the evals generated for this issue. + +```bash +pytest tests/evals/${issueKey}/ -v \ + --json-report \ + --json-report-file=.verify-and-fix-eval-results.json +``` + +**Gate passes when:** All eval tests pass (100%). + +**Gate fails when:** Any eval test fails. + +**Fix strategy:** +1. Read the failing eval test to understand which acceptance criterion it validates. +2. Read the implementation code that should satisfy the criterion. +3. Fix the implementation. Do NOT modify eval tests — they represent the acceptance criteria. +4. Re-run only the failing evals to confirm the fix before moving on. + +**Parse results:** +```javascript +const results = JSON.parse(readFile('.verify-and-fix-eval-results.json')) +const summary = { + total: results.summary.total, + passed: results.summary.passed, + failed: results.summary.failed, + failedTests: results.tests.filter(t => t.outcome === 'failed').map(t => t.nodeid) +} +``` + +--- + +### Gate 4: Code Review Blockers Check + +**What to run:** The `/code-review` skill scoped to the current branch diff. Extract only **Blockers** — suggestions and nits are ignored in the loop. + +```bash +/code-review +``` + +Read the review output and filter for the `### Blockers` section. If the verdict is `Approve` or the Blockers section is empty, the gate passes. + +**Gate passes when:** Verdict is `Approve` (no blockers), OR the Blockers section lists zero items. + +**Gate fails when:** One or more blockers are listed (bugs, security issues, linter errors the code review agent found that Gate 1 missed, failing tests the review agent flagged). + +**Fix strategy:** +1. Read each blocker: file path, line reference, and description. +2. Locate the file and line. +3. Apply the fix described in the blocker. +4. Do NOT address suggestions or nits — those go in the PR description for the human reviewer. + +--- + +## Retry Loop — Full Logic + +```javascript +const maxAttempts = parseInt(process.env.VERIFY_RETRY_LIMIT || '3') +const gates = parseGatesParam(args['gates']) || ['linter', 'tests', 'evals', 'review'] + +let attempt = 1 +let allGateResults = [] +const fixLog = [] + +while (attempt <= maxAttempts) { + console.log(`\n🔄 Verification attempt ${attempt}/${maxAttempts}`) + + const gateResults = {} + let failed = false + + // Run gates in order — stop at first failure to avoid noisy output + // but still collect all failures before fixing + + if (gates.includes('linter')) { + gateResults.linter = await runLinterGate() + if (!gateResults.linter.passed) failed = true + } + + if (gates.includes('tests')) { + gateResults.existingTests = await runExistingTestsGate() + if (!gateResults.existingTests.passed) failed = true + } + + if (gates.includes('evals') && !args['skip-evals']) { + gateResults.evals = await runEvalsGate(issueKey) + if (!gateResults.evals.passed) failed = true + } + + if (gates.includes('review')) { + gateResults.codeReview = await runCodeReviewGate() + if (!gateResults.codeReview.passed) failed = true + } + + allGateResults.push({ attempt, ...gateResults }) + + if (!failed) { + return { + passed: true, + attempts: attempt, + gateResults, + fixesApplied: fixLog + } + } + + if (attempt < maxAttempts) { + const fixes = await applyTargetedFixes(gateResults) + fixLog.push(...fixes) + attempt++ + } else { + // Max attempts reached + return { + passed: false, + attempts: attempt, + gateResults, + fixesApplied: fixLog, + remainingFailures: collectFailures(gateResults) + } + } +} +``` + +--- + +## Fix Application — Targeted Per Gate + +When one or more gates fail, apply fixes in gate order before retrying: + +```javascript +async function applyTargetedFixes(gateResults) { + const fixes = [] + + if (gateResults.linter && !gateResults.linter.passed) { + // 1. Run auto-fix commands + // 2. For remaining errors: read error message, locate file:line, edit + fixes.push(...await fixLinterErrors(gateResults.linter.errors)) + } + + if (gateResults.existingTests && !gateResults.existingTests.passed) { + // 1. Read failing test + implementation + // 2. Fix implementation (or update test if intentional change) + fixes.push(...await fixTestFailures(gateResults.existingTests.failures)) + } + + if (gateResults.evals && !gateResults.evals.passed) { + // 1. Map failing eval → acceptance criterion + // 2. Fix implementation (never edit eval tests) + fixes.push(...await fixEvalFailures(gateResults.evals.failedTests)) + } + + if (gateResults.codeReview && !gateResults.codeReview.passed) { + // 1. Read each blocker's file:line + // 2. Apply fix (ignore suggestions/nits) + fixes.push(...await fixCodeReviewBlockers(gateResults.codeReview.blockers)) + } + + return fixes +} +``` + +--- + +## Output + +### All Gates Pass + +```markdown +✅ Verification passed on attempt 2/3 + +Gate Results: + ✅ Linter: No errors (auto-fixed 3 warnings on attempt 1) + ✅ Existing Tests: 47/47 passing + ✅ Evals: 8/8 passing (tests/evals/ABI-123/) + ✅ Code Review: Approve — no blockers + +Fixes Applied (attempt 1): + - Auto-fixed: ruff --fix removed 3 unused import warnings + - Fixed: src/api/rate_limiter.py:42 — removed unreachable branch flagged by ruff + +Ready to create PR. +``` + +### Max Attempts Reached + +```markdown +⚠️ Verification failed after 3/3 attempts + +Gate Results (final attempt): + ✅ Linter: No errors + ✅ Existing Tests: 47/47 passing + ⚠️ Evals: 6/8 passing (2 failed) + - test_concurrent_users_performance: latency exceeded threshold + - test_rate_limit_reset: reset timer off by 1s + ✅ Code Review: Approve — no blockers + +Fixes Applied: + - Attempt 1: Fixed linter errors (3 auto-fixed) + - Attempt 2: Fixed test_rate_limit_reset — corrected timer math + - Attempt 3: Adjusted connection pool size for throughput — test still fails + +Remaining Failures: + - test_concurrent_users_performance: needs architecture review + +Recommendation: Create PR with [NEEDS-REVIEW] label. Human review needed for performance issue. +``` + +--- + +## Output Schema + +```json +{ + "passed": true, + "attempts": 2, + "gateResults": { + "linter": { + "passed": true, + "autoFixed": 3, + "remainingErrors": 0 + }, + "existingTests": { + "passed": true, + "total": 47, + "failures": [] + }, + "evals": { + "passed": 8, + "failed": 0, + "total": 8, + "failedTests": [] + }, + "codeReview": { + "verdict": "Approve", + "blockers": [], + "suggestions": 2 + } + }, + "fixesApplied": [ + "auto-fix: ruff removed 3 unused import warnings", + "edit: src/api/rate_limiter.py:42 — removed unreachable branch" + ] +} +``` + +--- + +## Configuration + +```bash +VERIFY_RETRY_LIMIT=3 # Max full-loop attempts (default: 3) +VERIFY_GATES=linter,tests,evals,review # Which gates to run (default: all) +EVAL_TIMEOUT=300 # Seconds before eval run times out (default: 300) +``` + +--- + +## Invariants + +- **Never modify eval tests** to make them pass. Evals represent acceptance criteria. +- **Never delete existing tests** to remove failures. Fix the code instead. +- **Never skip blockers** from code review. Fix blockers; ignore suggestions and nits. +- **Auto-fix is always attempted first** for linter gate before manual edits. +- **Gate order is fixed**: linter → tests → evals → review. This order is not configurable. + +--- + +## Integration with Other Skills + +**Called by `/autonomous-implement`** after Step 6 (Implement): +```bash +/verify-and-fix ${issueKey} +# returns: { passed, attempts, gateResults, fixesApplied } +# if passed → proceed to /create-pr +# if !passed → create PR with [NEEDS-REVIEW] label, flag remaining failures +``` + +**Can also be called standalone** during development: +```bash +# After making changes, before committing +/verify-and-fix ABI-123 + +# Check only linter and tests (fast feedback) +/verify-and-fix ABI-123 --gates linter,tests + +# Full verification before opening PR manually +/verify-and-fix ABI-123 --max-attempts 5 +``` diff --git a/test_orchestrator.py b/test_orchestrator.py index e9dfc75..85f50ae 100755 --- a/test_orchestrator.py +++ b/test_orchestrator.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 """ -Test the orchestrator end-to-end. +Test the harness end-to-end. This script demonstrates: 1. Loading workspace configuration @@ -17,7 +17,7 @@ # Add to path sys.path.insert(0, str(Path(__file__).parent)) -from orchestrator import KnowledgeEngine, Router, Planner +from harness import KnowledgeEngine, Router, Planner # Load workspace config print("Loading workspace configuration...") @@ -32,7 +32,7 @@ print() # Initialize components -print("Initializing orchestrator components...") +print("Initializing harness components...") knowledge_engine = KnowledgeEngine(factory_root / 'knowledge') router = Router(workspace_config) planner = Planner(knowledge_engine, router, workspace_config) @@ -123,7 +123,7 @@ print("=" * 60) print() print("Next steps:") -print(" 1. The orchestrator successfully generated a task graph") +print(" 1. The harness successfully generated a task graph") print(" 2. Knowledge was loaded from the knowledge pack") print(" 3. Steps were generated based on issue type") print(" 4. Ready to integrate with Claude Code for execution") diff --git a/test_orchestrator_with_sync.py b/test_orchestrator_with_sync.py index f88aea8..a4a783f 100755 --- a/test_orchestrator_with_sync.py +++ b/test_orchestrator_with_sync.py @@ -2,11 +2,11 @@ """ Test Orchestrator with Auto-Sync -Demonstrates that knowledge packs are automatically synced before orchestrator runs. +Demonstrates that knowledge packs are automatically synced before harness runs. """ import sys -from orchestrator import ensure_knowledge_fresh, KnowledgeEngine, Router +from harness import ensure_knowledge_fresh, KnowledgeEngine, Router def main(): print("=" * 80) @@ -108,10 +108,10 @@ def main(): print("🎉 Orchestrator ready to use!") print() print("Next steps:") - print("1. Knowledge packs will auto-sync before each orchestrator run") + print("1. Knowledge packs will auto-sync before each harness run") print("2. Only re-extracts if README.md or docs/ changed in repos") print("3. Manual sync: ./sync_knowledge.sh") - print("4. Force sync: python3 -m orchestrator.sync --force") + print("4. Force sync: python3 -m harness.sync --force") print() if __name__ == '__main__': diff --git a/test_subprocess_claude.py b/test_subprocess_claude.py index a30e88a..4640faa 100644 --- a/test_subprocess_claude.py +++ b/test_subprocess_claude.py @@ -34,7 +34,7 @@ except subprocess.TimeoutExpired: print(" ERROR: Timeout") -# Test 3: claude with message file (similar to orchestrator) +# Test 3: claude with message file (similar to harness) print("\n3. Testing 'claude --message-file'...") message = "Hello, this is a test message" with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f: diff --git a/workspace.yaml b/workspace.yaml index 83a7310..da4df46 100644 --- a/workspace.yaml +++ b/workspace.yaml @@ -182,7 +182,7 @@ testing: # AI Factory configuration factory: - orchestrator: + harness: max_concurrent_tasks: 8 retry_on_failure: 3 timeout_minutes: 30