Configuration Validator with Auto-Optimization & Drift Detection

[github-actions]

Strategic Improvement

Validates daemon-config.json against a schema, detects misconfigurations that cause failures (timeouts too low, effort levels wrong, iteration budgets suboptimal), and auto-tunes settings based on DORA metrics and recent performance data. Runs on daemon startup and periodically during operation. With 48 hardcoded values across the codebase, ensuring runtime configuration is correct and optimal is critical for reliability.

Acceptance Criteria

• Defines JSON schema for daemon-config.json covering all sections (effort_levels, model_routing, loop, intelligence, etc.)
• Validates config on daemon startup and every 6 hours during operation
• Detects common misconfigurations: timeouts below minimum thresholds, invalid model names, iteration budgets too low/high
• Auto-tunes based on DORA metrics: if MTTR is high, increase retry thresholds; if lead time is high, increase iteration budgets
• Detects drift: compares current config to optimal baseline, alerts if deviation >20%
• Logs validation results to ~/.shipwright/config-health.jsonl
• Test suite validates schema enforcement and auto-tuning logic

Context

• Priority: P0
• Complexity: standard
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P0: Reliability & Success Rate

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	Strategic reliability improvement that validates and auto-optimizes daemon configuration, reducing failures from misconfiguration—core to daemon stability and P0 priority.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	30093728616

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (280 lines)

Implementation Plan — Configuration Validator with Auto-Optimization & Drift Detection

Issue: #809 Priority: P0 (Reliability & Success Rate) Complexity: standard
Task type: devops / bash + jq Scripts language: Bash 3.2 Schema: JSON Schema draft-07

---

Brainstorming: Socratic Design Refinement (answered in-pipeline)

Minimum viable change. One new sourceable bash library (scripts/lib/config-validator.sh) that (1) validates .claude/daemon-config.json against a jq-checkable rule set, (2) is invoked once on daemon startup and every 6h in the poll loop, and (3) appends results to ~/.shipwright/config-health.jsonl. Auto-tuning and drift detection are additive layers over the same read/validate core. Everything else (a shipped config/daemon-config.schema.json, CLI subcommands, docs) makes the MVP discoverable and testable.

Implicit requirements (not stated in the issue).

• The validator must never crash the daemon: value-level problems degrade to WARN, not exit 1. Only structural JSON corruption may hard-fail at boot.
• Auto-tune writes to the same file the daemon just read — it must use the canonical atomic mktemp → jq → mv pattern and be idempotent (no metric change ⇒ no diff).
• "Optimal baseline" does not exist yet; we must define it (shipped defaults + observed DORA) and persist to ~/.shipwright/optimization/config-baseline.json.

Time/complexity bound (validated against a real use case). Validation is O(rules) ≈ 40 jq reads over a <5 KB file → <150 ms (proxy: _smart_int does one jq read per key; 40 reads are negligible against the 60 s poll and 6 h validation cadence). Real precedent: the existing optimize_adjust_audit_intensity() in sw-self-optimize.sh already performs the same class of jq read → conditional → atomic write every optimize cycle with no measurable overhead.

---

Architecture Decision Record

Component Diagram

                       ┌──────────────────────────────────────────────┐
                       │  scripts/sw-config-validator.sh (CLI shim)    │
                       │  shipwright config {validate|tune|drift|       │
                       │                     health|baseline}           │
                       └───────────────────┬──────────────────────────┘
                                           │ sources
                                           ▼
        ┌───────────────────────────────────────────────────────────────┐
        │            scripts/lib/config-validator.sh  (core)             │
        │  cfgval_validate <cfg>      → validation result JSON           │
        │  cfgval_autotune  <cfg>     → applies DORA-informed edits       │
        │  cfgval_detect_drift <cfg>  → deviation report vs baseline      │
        │  cfgval_write_health <json> → append config-health.jsonl        │
        │  cfgval_capture_baseline    → snapshot optimal config           │
        └──┬──────────────┬───────────────┬───────────────┬──────────────┘
           │ reads         │ reads DORA     │ reads/writes  │ writes
           ▼               ▼                ▼               ▼
  config/daemon-      calculate_dora()   .claude/         ~/.shipwright/
  config.schema.json  (sw-dora.sh, or    daemon-          config-health.jsonl
  (rule source)       events.jsonl)      config.json      optimization/config-baseline.json

        ▲ hooked into                        ▲ hooked into
        │                                    │
   sw-daemon.sh daemon_start()          lib/daemon-poll.sh daemon_poll_loop()
   (startup, after load_config)         (every 6h via LAST_CONFIG_VALIDATION_EPOCH)

</details>

## ✅ Task Checklist
- [ ] Task 1: Write `config/daemon-config.schema.json` (draft-07) covering all sections. *(blocks 2)*
- [ ] Task 2: Implement rule table + `_cfgval_*` helpers in `scripts/lib/config-validator.sh`. *(needs 1; blocks 3,6)*
- [ ] Task 3: Implement `cfgval_validate` (structural gate + rule loop + event). *(needs 2; blocks 9,10,11)*
- [ ] Task 4: Implement `cfgval_write_health` atomic append to `config-health.jsonl`.
- [ ] Task 5: Implement `cfgval_capture_baseline` + `cfgval_detect_drift` (awk pct, 20% threshold, event).
- [ ] Task 6: Implement `cfgval_autotune` (DORA read, bounded idempotent jq edits, `--dry-run`, `.total<5` guard). *(needs 2,5)*
- [ ] Task 7: Write `scripts/sw-config-validator.sh` CLI shim with `main` guard. *(needs 3–6)*
- [ ] Task 8: Register `config`/`config-validator` in `scripts/sw`. *(needs 7)*
- [ ] Task 9: Hook validate into `sw-daemon.sh` `daemon_start()`; parse `config_validation.*`; extend `daemon init`. *(needs 3)*
- [ ] Task 10: Hook validate+drift+autotune into `lib/daemon-poll.sh` (6h). *(needs 3,5,6)*
- [ ] Task 11: Write `scripts/sw-config-validator-test.sh`; register in `package.json`. *(needs 3–6)*
- [ ] Task 12: E2E discoverability check via CLI router. *(needs 8)*
- [ ] Task 13: Update `.claude/CLAUDE.md`; `shipwright docs check` passes.
- [ ] Task 14: Run full `npm test`; confirm no daemon/self-optimize/compat/poll regressions.
- [ ] `config/daemon-config.schema.json` covers `effort_levels`, `model_routing`, `loop`, `intelligence`, `auto_scale`, `patrol`, `health`, `max_turns`, polling/pipeline keys. *(AC1)*
- [ ] `cfgval_validate` runs on daemon startup and every 6h in the poll loop. *(AC2)*
- [ ] Detects timeouts below minimum, invalid model names, out-of-range iteration budgets — each with a dedicated passing test. *(AC3)*
- [ ] `cfgval_autotune` raises retry thresholds on high MTTR and iteration budgets on high lead time, driven by DORA, tested both ways. *(AC4)*
- [ ] `cfgval_detect_drift` compares to `config-baseline.json`, flags deviation >20%, with boundary tests. *(AC5)*
- [ ] Validation results appended to `~/.shipwright/config-health.jsonl`. *(AC6)*

---
_Generated by `shipwright pipeline` at 2026-07-24T12:50:12Z_

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (153 lines)

Anchors confirmed: config-validator.sh/sw-config-validator.sh/daemon-config.schema.json don't yet exist (correct for design stage), while calculate_dora() (sw-dora.sh:114) emits exactly {deploy_freq, cycle_time[hrs], cfr[%], mttr[hrs], total} and the quiet-period guard in lib/daemon-poll.sh:606 is real. Here is the ADR.

---

Design: Configuration Validator with Auto-Optimization & Drift Detection

Context

The daemon and pipeline read .claude/daemon-config.json through dozens of scattered _smart_int/_smart_model/_smart_effort calls (scripts/lib/compat.sh). Every one of those calls silently falls back to a hardcoded default when a key is missing, malformed, or out of range. The failure mode this issue (#809, P0) targets is silent misconfiguration: a typo'd model name ("gpt4"), a timeout below a usable floor (max_turns.pipeline_stage=1), or an iteration budget so high it burns tokens (loop.max_restarts=99) never surfaces — it just quietly degrades reliability and success rate. There is no code that reads the config as a whole and asserts it is sane.

Constraints that shape the design, drawn from the codebase:

• Pure bash 3.2 + jq. No declare -A, no readarray, no ${var,,}, no float arithmetic in bash. Every existing lib obeys this; introducing a node/ajv runtime dependency into the hot path violates the "scripts are bash" architecture (.claude/CLAUDE.md → Architecture).
• The config is read concurrently. Many pipeline stages read it while the daemon runs. Any writer (auto-tune) must use the canonical mktemp → jq → mv atomic pattern so readers see either the old or new complete file — never a torn one.
• Reliability tooling must fail open. A validator that can crash the daemon is a net reliability loss. Precedent: daemon_self_optimize/daemon_patrol in lib/daemon-poll.sh are all || daemon_log WARN "... — continuing".
• DORA metrics already exist. calculate_dora() (sw-dora.sh:114) returns {deploy_freq, cycle_time (hrs), cfr (%), mttr (hrs), total} off ~/.shipwright/events.jsonl, returning all-zeros when the file is absent. Auto-tune reuses this — it does not own metric computation.
• Decomposition is the trend. Recent commits decomposed sw-recruit.sh/sw-pipeline.sh; folding this into the already-1690-line sw-self-optimize.sh runs against that grain and SRP.

Decision

Ship a standalone sourceable core lib + thin CLI shim + draft-07 schema, hooked into the daemon lifecycle at two points, all fail-open except structural JSON corruption at boot.

Five components, each with one reason to change:

Component	Path	Reason to change
Schema (rule source of truth)	config/daemon-config.schema.json	Config surface changes
Core lib	scripts/lib/config-validator.sh	Validation/tune/drift logic
CLI shim	scripts/sw-config-validator.sh	UX / subcommand surface
DORA reader	scripts/sw-dora.sh (reused, not owned)	Metric definition
Lifecycle hooks	sw-daemon.sh, lib/daemon-poll.sh	Wiring cadence

Validation core. Rules are encoded as a newline-delimited table (field_path|type|constraint|severity) consumed by a while read loop — the bash-3.2-safe substitute for the associative array an ajv error map would use. cfgval_validate:

1. Structural gate first: jq empty <cfg> → on failure return {ok:false} (the only hard-fail condition).
2. Rule loop: each rule uses jq getpath where null/absent ⇒ skip (optional sections don't misfire), accumulating errors[]/warnings[] into an ok:true envelope so the caller decides severity.
3. Emit emit_event "config.validated" "errors=N" "warnings=M".

Error-handling contract (the crux of the design):

• cfgval_validate returns ok:false only on unparseable JSON. All value problems ride inside ok:true.
• cfgval_autotune / cfgval_detect_drift return empty result sets, exit 0 when DORA/baseline data is absent — a reliability feature fails open.
• Daemon startup is the sole place a ok:false becomes exit 1: booting on a corrupt config means every downstream jq read silently defaults, hiding the corruption. Better to stop loudly once at boot.

Auto-tune data flow (DORA → config). In the poll quiet-period branch only (issue_count==0 && active_count==0, exactly where patrol runs — so no stage is mid-read), when self_optimize=true: read DORA via type -t calculate_dora (sourcing sw-dora.sh behind its main guard) with an inline events.jsonl fallback for self-containment. Guard .total < 5 ⇒ no change. Bounded, hard-capped, idempotent edits, each a chained atomic write. Ownership is split to prevent thrash: validator owns loop.max_restarts, max_retries, loop.extension_size, max_turns.pipeline_stage; audit intensity stays in optimize_adjust_audit_intensity.

Drift. cfgval_capture_baseline projects the tunable numeric subset to ~/.shipwright/optimization/config-baseline.json (atomic). cfgval_detect_drift computes pct = |cur-base|*100/base in awk (no bash floats), flags > threshold*100 (default 20%), emits config.drift_detected. Critical: recapture baseline after a sanctioned autotune so drift only ever flags unexpected change, not the daemon's own edits.

Observability sink. cfgval_write_health appends each validation record to ~/.shipwright/config-health.jsonl using the same flock/atomic-append convention as emit_event, swallowing all errors.

Alternatives Considered

1. Pure bash+jq rule engine (CHOSEN) — Pros: <150ms (~40 jq reads over <5KB), no new runtime dep, matches every existing lib, additive blast radius, the shipped draft-07 schema stays ajv-consumable in CI later at zero runtime cost. Cons: rule table is hand-maintained parallel to the schema (mitigated by generating both from the same section list and a test asserting they agree).
2. ajv / node JSON-Schema at runtime — Pros: single source of truth, richer error messages. Cons: node cold-start ~200–400ms per call in the daemon hot path, introduces a hard node dependency into a pure-bash daemon, violates the stated architecture. Rejected — but we ship the schema so this stays open later.
3. Fold into sw-self-optimize.sh — Pros: fewer files. Cons: that file is 1690 lines and owns learning/bandit logic; validation+drift is a distinct responsibility (SRP), and editing a heavily-tested file risks its suite. Rejected, against the decomposition trend.

Implementation Plan

Files to create:

• config/daemon-config.schema.json — draft-07, additionalProperties:true at top (config carries last_optimization etc.), strict on enums/ranges: effort_levels.* ∈ {low,medium,high,xhigh,max}; model_routing.*/model/fallback_model ∈ {opus,sonnet,haiku,fable}; pipeline_template ∈ the 8 shipped templates; integer minimum/maximum on loop.*, poll_interval, max_parallel, auto_scale.*, max_turns.pipeline_stage; intelligence.ab_test_ratio ∈ [0,1]; intelligence.anomaly_threshold > 0.
• scripts/lib/config-validator.sh — cfgval_validate|autotune|detect_drift|write_health|capture_baseline + _cfgval_* helpers + rule table; idempotent-source guard _SW_CONFIG_VALIDATOR_LOADED.
• scripts/sw-config-validator.sh — standard header, VERSION="3.3.0", [[ "${BASH_SOURCE[0]}" == "$0" ]] && main "$@" guard, subcommands validate|tune|drift|health|baseline capture|help.

---

Generated by shipwright pipeline design stage at 2026-07-24T12:53:32Z

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: opus

[github-actions]

[github-actions]