diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b15c33..95114ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,82 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [2.39.0] - 2026-06-22 + +A correctness & security hardening release. A multi-agent audit of C3 surfaced a +hook-enforcement bypass, several edit-ledger / session-store data-loss races, Windows +line-ending and subprocess bugs across the c3 tools, installer config-merge data-loss +risks, and three Oracle security gaps. All are fixed here. + +### Security + +- **Oracle `POST /api/config` was unauthenticated.** Any local process could + `POST {"api_require_auth": false}` to strip authentication off the entire Discovery + API, or repoint `ollama_base_url` to exfiltrate prompts. The endpoint now requires the + Bearer token and rejects unknown config keys (allowlisted from `DEFAULTS`). +- **Oracle `GET /api/apikey` leaked the raw token.** It returned the plaintext Bearer + token with no auth; it now returns a masked form unless a valid Bearer token is + presented (`generate`/`rotate` still reveal the new token once). +- **Oracle Discovery `project_path` was unvalidated.** Callers could read any `.c3` + project on the machine by path; project paths are now validated against discovered + projects before any read. + +### Fixed + +- **Enforcement bypass: any read-only `c3_*` call unlocked native `Edit`/`Write`.** The + PreToolUse signal fast-path allowed any native tool whenever *any* fresh c3 signal + existed, ignoring per-tool prerequisites. Write-class tools (Edit/Write/MultiEdit) now + require a `c3_edit`/`c3_edits`/`c3_agent` signal; read tools are unchanged. +- **`MultiEdit` and `NotebookEdit` bypassed enforcement and the edit ledger entirely** — + no PreToolUse/PostToolUse matcher was registered for them. Both are now enforced and + logged. +- **`c3_edit` rewrote whole LF files as CRLF on Windows.** A one-line edit flipped every + line ending; edits now preserve the file's original newline style (single, batch, and + create modes). +- **`c3_edit` batch mode wrote the file and logged a ledger entry even when zero patches + applied**, and crashed on a non-dict batch element. It now writes/logs only when a patch + actually changed the file, and returns a clear error for malformed batches. +- **Edit ledger could lose writes.** `tag_edit` did a lock-free full-file rewrite that + clobbered concurrent appends, and `log_edit` didn't take the write lock. `tag_edit` now + appends a tag patch under lock, `log_edit` is locked, edit ids carry a random suffix to + prevent hook/server collisions within the same second, and orphaned patches are logged. +- **`sessions.json` could be wiped on a corrupt/partial read.** The conversation store now + writes atomically (temp + `os.replace`) and, on a parse failure, backs up the corrupt + file instead of silently resetting the catalog to empty; `add_turn` index updates are + locked. +- **`c3_delegate(backend="claude")` was 100% broken** (a tuple-unpacking bug failed every + call). Fixed; all CLI runners now also decode subprocess output as UTF-8 (cp1252 crash + fix) and kill the full process tree on timeout. +- **JS/TS `export class/function/const` symbols were missing** from compression maps and + the file-memory index (the walker descended past the declaration). Exported symbols are + now indexed. +- **`c3_compress` rendered every class with Python `class Name:` syntax** regardless of + language; it now uses the language-appropriate declaration. +- **`c3_read(symbols=...)` could return truncated bodies** when a `}` appeared inside a + string or comment; the brace scanner now skips string/char/template literals and + comments. +- **`file_memory` lazy search index had a first-search-vs-background-update race** + (introduced with lazy init); build/update/search are now lock-guarded. +- **Installer config-merge data-loss risks.** `merge_c3_block` could corrupt `CLAUDE.md` + on out-of-order/duplicate markers; the global-`CLAUDE.md` writer could delete user + `#` headings placed after the managed block; and `upsert_toml_section` orphaned child + subtables (e.g. `[mcp_servers.c3.env]`) on re-install. All fixed; the global managed + region now uses explicit BEGIN/END markers. +- **Sticky unlocks from `c3_compress`/`c3_agent` were lost** (written only to a file no + hook reads); they now reach the enforcer's `.json` unlock map. +- Smaller fixes: `c3_read` negative/reversed/comma line specs and `lines=0`; `c3_validate` + process-tree kill + UTF-8 on Windows; empty-fact rejection in `c3_memory add`; + `context_snapshot` atomic writes + corrupt-latest fallback; `web_security` no longer + skips the host allowlist on a missing Host for mutating requests; Oracle chat/config + endpoints return JSON errors for bad bodies; Oracle MCP auth toggles apply without a + restart; the activity digest flags truncated scans; TOML `#` inside quoted values is no + longer stripped. + +### Changed + +- PreToolUse enforcement now distinguishes read-class from write-class c3 signals — a + behavior change for anyone who relied on a read-only c3 call to unlock a native write. + ## [2.38.1] - 2026-06-14 Startup reliability fixes for the MCP server and for the Oracle on Windows. diff --git a/cli/_hook_utils.py b/cli/_hook_utils.py index 53c9b87..28c60e3 100644 --- a/cli/_hook_utils.py +++ b/cli/_hook_utils.py @@ -74,9 +74,46 @@ def get_tool_output(data: dict) -> tuple: def get_tool_input_path(data: dict) -> str: - """Extract file path from tool_input, handling both Claude (file_path) and Gemini (path).""" + """Extract file path from tool_input, handling Claude (file_path), + Gemini (path), and NotebookEdit (notebook_path).""" tool_input = data.get("tool_input", {}) - return tool_input.get("file_path", "") or tool_input.get("path", "") + return ( + tool_input.get("file_path", "") + or tool_input.get("path", "") + or tool_input.get("notebook_path", "") + ) + + +def record_json_unlocks(editable: list, project_path: Path | None = None) -> None: + """Record file paths as read+edit unlocked in .c3/unlocked_files.json. + + This is the map that hook_pretool_enforce.py actually reads (the plain + .txt unlock list is not consumed by any hook). Mirrors the behaviour of + cli/hook_c3read._record_json_unlocks so c3_compress/c3_agent sticky + unlocks reach the enforcer. Fails silently on I/O errors. + """ + base = project_path if project_path is not None else Path.cwd() + json_path = base / ".c3" / "unlocked_files.json" + try: + existing: dict = {} + if json_path.exists(): + try: + existing = json.loads(json_path.read_text(encoding="utf-8")) + if not isinstance(existing, dict): + existing = {} + except Exception: + existing = {} + for fp in editable: + if not fp: + continue + normalized = str(Path(fp).resolve()) + cats = set(existing.get(normalized, [])) + cats.update({"read", "edit"}) + existing[normalized] = sorted(cats) + json_path.parent.mkdir(parents=True, exist_ok=True) + json_path.write_text(json.dumps(existing), encoding="utf-8") + except Exception: + pass def emit_additional_context(text: str, is_gemini: bool) -> None: diff --git a/cli/c3.py b/cli/c3.py index c2e678e..3952808 100644 --- a/cli/c3.py +++ b/cli/c3.py @@ -85,7 +85,7 @@ # Config CONFIG_DIR = ".c3" CONFIG_FILE = ".c3/config.json" -__version__ = "2.38.1" +__version__ = "2.39.0" def _command_deps() -> CommandDeps: @@ -4084,7 +4084,11 @@ def _upsert_toml_section(toml_path: Path, section: str, entries: dict) -> None: content = toml_path.read_text(encoding="utf-8") if toml_path.exists() else "" header = f"[{section}]" - # Strip existing section (header + its key=value lines) + # Strip existing section (header + its key=value lines). Also strip any + # dotted child subtables (e.g. "[mcp_servers.c3.env]" under + # "[mcp_servers.c3]") so they are not orphaned beneath the re-appended + # section, which would corrupt the file on re-run. + child_prefix = f"{header[:-1]}." # "[mcp_servers.c3]" -> "[mcp_servers.c3." lines = content.splitlines() new_lines: list[str] = [] skip = False @@ -4094,7 +4098,8 @@ def _upsert_toml_section(toml_path: Path, section: str, entries: dict) -> None: skip = True continue if skip and stripped.startswith("["): - skip = False + if not stripped.startswith(child_prefix): + skip = False if not skip: new_lines.append(line) @@ -4577,41 +4582,48 @@ def _ensure_global_claude_md() -> None: existing = global_md.read_text(encoding="utf-8") - if _GLOBAL_CLAUDE_MD_MARKER not in existing: - # User has their own CLAUDE.md — append C3 section - merged = existing.rstrip() + "\n\n" + _GLOBAL_CLAUDE_MD_CONTENT - global_md.write_text(merged, encoding="utf-8") - print(f"Updated {global_md} (appended C3 enforcement)") - return + # The C3-managed region is delimited by explicit BEGIN/END sentinels (the + # same ones used for project instruction docs). This is unambiguous, so + # user-written content outside the markers — including H1 headings that + # happen to mention "C3" or "Tool Discipline" — is never swallowed. + from services.claude_md import C3_BLOCK_BEGIN, C3_BLOCK_END, merge_c3_block - # C3 section exists — replace it with the latest version - # Find the C3 section boundaries: starts at the marker, ends at next # heading or EOF - start = existing.index(_GLOBAL_CLAUDE_MD_MARKER) - # Find the next top-level heading after the C3 section - rest = existing[start + len(_GLOBAL_CLAUDE_MD_MARKER):] - lines_after = rest.split("\n") - end_offset = len(rest) # default: to EOF - running = 0 - for line in lines_after: - running += len(line) + 1 - # A top-level heading that's NOT part of C3's sub-headings - if line.startswith("# ") and "C3" not in line and "Tool Discipline" not in line: - end_offset = running - len(line) - 1 - break - - end = start + len(_GLOBAL_CLAUDE_MD_MARKER) + end_offset - before = existing[:start].rstrip() - after = existing[end:].lstrip() + wrapped = f"{C3_BLOCK_BEGIN}\n{_GLOBAL_CLAUDE_MD_CONTENT.strip()}\n{C3_BLOCK_END}" - parts = [] - if before: - parts.append(before) - parts.append(_GLOBAL_CLAUDE_MD_CONTENT.strip()) - if after: - parts.append(after) + # Markers already present → surgical, marker-bounded replacement. + if C3_BLOCK_BEGIN in existing: + global_md.write_text(merge_c3_block(existing, wrapped), encoding="utf-8") + print(f"Updated {global_md} (refreshed C3 enforcement)") + return - global_md.write_text("\n\n".join(parts) + "\n", encoding="utf-8") - print(f"Updated {global_md} (refreshed C3 enforcement)") + # Legacy marker-less C3 region → one-time migration into the marked block. + # Bound the region from the legacy heading to the NEXT top-level (``# ``) + # heading. C3's own content has exactly one H1 (the legacy heading itself), + # so the next H1 reliably marks where user content resumes; we deliberately + # do NOT skip H1s containing "C3"/"Tool Discipline" (the old heuristic did, + # which is what swallowed user headings). + if _GLOBAL_CLAUDE_MD_MARKER in existing: + start = existing.index(_GLOBAL_CLAUDE_MD_MARKER) + rest = existing[start + len(_GLOBAL_CLAUDE_MD_MARKER):] + end_offset = len(rest) # default: to EOF + running = 0 + for line in rest.split("\n"): + running += len(line) + 1 + if line.startswith("# "): + end_offset = running - len(line) - 1 + break + end = start + len(_GLOBAL_CLAUDE_MD_MARKER) + end_offset + before = existing[:start].rstrip() + after = existing[end:].lstrip() + parts = [p for p in (before, wrapped, after) if p] + global_md.write_text("\n\n".join(parts) + "\n", encoding="utf-8") + print(f"Updated {global_md} (migrated C3 enforcement to markers)") + return + + # User has their own CLAUDE.md with no C3 content — append the marked block. + merged = existing.rstrip() + "\n\n" + wrapped + "\n" + global_md.write_text(merged, encoding="utf-8") + print(f"Updated {global_md} (appended C3 enforcement)") def _instruction_documents_for_project() -> list[tuple[str, str]]: @@ -5019,6 +5031,8 @@ def cmd_install_mcp(args): glob_matcher = "find_files" edit_matcher = "edit_file" write_matcher = "write_file" + # Gemini has no MultiEdit / NotebookEdit equivalents. + extra_edit_matchers = [] else: shell_matcher = "Bash" read_matcher = "Read" @@ -5026,6 +5040,9 @@ def cmd_install_mcp(args): glob_matcher = "Glob" edit_matcher = "Edit" write_matcher = "Write" + # Claude Code also exposes MultiEdit (batch edits) and NotebookEdit; + # both bypass enforcement/logging unless their matchers are registered. + extra_edit_matchers = ["MultiEdit", "NotebookEdit"] # ── PostToolUse hooks ── desired_post_hooks = [ @@ -5120,6 +5137,10 @@ def cmd_install_mcp(args): "matcher": write_matcher, "hooks": [{"type": "command", "command": hook_edit_ledger_cmd}] }, + *[ + {"matcher": m, "hooks": [{"type": "command", "command": hook_edit_ledger_cmd}]} + for m in extra_edit_matchers + ], ] # ── PreToolUse hooks (enforcement — blocks native tools without prior c3_*) ── @@ -5144,6 +5165,10 @@ def cmd_install_mcp(args): "matcher": write_matcher, "hooks": [{"type": "command", "command": hook_enforce_cmd}] }, + *[ + {"matcher": m, "hooks": [{"type": "command", "command": hook_enforce_cmd}]} + for m in extra_edit_matchers + ], ] # Merge: replace existing C3 hooks (so re-running install-mcp updates commands), diff --git a/cli/hook_edit_ledger.py b/cli/hook_edit_ledger.py index b2efc54..0353744 100644 --- a/cli/hook_edit_ledger.py +++ b/cli/hook_edit_ledger.py @@ -10,6 +10,7 @@ import json import sys +import uuid from datetime import datetime, timezone from pathlib import Path @@ -25,7 +26,7 @@ ".py", ".js", ".ts", ".tsx", ".jsx", ".go", ".rs", ".java", ".rb", ".c", ".cpp", ".h", ".cs", ".html", ".css", ".json", ".yaml", ".yml", ".toml", ".sql", ".md", ".txt", - ".sh", ".bat", ".ps1", ".r", + ".sh", ".bat", ".ps1", ".r", ".ipynb", } # How many tail lines to scan for version/seq (avoids full-file parse) @@ -95,8 +96,11 @@ def _next_seq(ledger_file: Path, now: datetime) -> int: continue eid = entry.get("id", "") if eid.startswith(prefix): + # ids may carry a random suffix ("..._001_a1b2"); take the leading + # numeric run after the prefix so same-second seq counting survives. + seq_part = eid[len(prefix):].split("_", 1)[0] try: - max_seq = max(max_seq, int(eid[len(prefix):])) + max_seq = max(max_seq, int(seq_part)) except ValueError: pass return max_seq + 1 @@ -172,7 +176,9 @@ def main(): git_pending = tracking_level != "minimal" entry = { - "id": f"edit_{now.strftime('%Y%m%d_%H%M%S')}_{_next_seq(ledger_file, now):03d}", + # Random suffix prevents id collisions when the hook process and the + # server process (services/edit_ledger.py) write within the same second. + "id": f"edit_{now.strftime('%Y%m%d_%H%M%S')}_{_next_seq(ledger_file, now):03d}_{uuid.uuid4().hex[:4]}", "timestamp": now.isoformat(), "session_id": "", "file": rel, diff --git a/cli/hook_edit_unlock.py b/cli/hook_edit_unlock.py index 006fe40..3bdd709 100644 --- a/cli/hook_edit_unlock.py +++ b/cli/hook_edit_unlock.py @@ -14,7 +14,11 @@ sys.path.insert(0, str(Path(__file__).parent.parent)) -from cli._hook_utils import emit_additional_context, log_hook_error # noqa: E402 +from cli._hook_utils import ( # noqa: E402 + emit_additional_context, + log_hook_error, + record_json_unlocks, +) EDITABLE_EXTS = { ".py", ".js", ".ts", ".tsx", ".jsx", ".go", ".rs", ".java", @@ -144,6 +148,10 @@ def main(): except Exception: pass + # Fix 2: also write the .json unlock map — the .txt list above is read + # by NO hook; hook_pretool_enforce.py only consults unlocked_files.json. + record_json_unlocks(editable) + # Emit batched nudge with all pending files # Prefer c3_edit (no unlock needed). Native Edit is also unlocked via sticky file set. if len(pending) == 1: diff --git a/cli/hook_pretool_enforce.py b/cli/hook_pretool_enforce.py index 8e7fea9..fb998e6 100644 --- a/cli/hook_pretool_enforce.py +++ b/cli/hook_pretool_enforce.py @@ -180,24 +180,30 @@ def _is_file_unlocked(project_path: Path, file_path: str, category: str) -> bool return category in cats or "both" in cats -def _check_signal_file(project_path: Path) -> tuple[bool, bool]: +def _check_signal_file(project_path: Path) -> tuple[bool, bool, str]: """Read last_c3_call.json written by hook_c3_signal.py. - Returns (recent, read_unlocked): + Returns (recent, read_unlocked, c3_tool): recent: True if a c3_* tool completed within _SIGNAL_MAX_AGE_SECS read_unlocked: True if that tool was c3_search/c3_compress/c3_filter + c3_tool: short name of the c3 tool that wrote the signal (e.g. + "c3_edit"), or "" if recent is False / unparseable. + + Fails closed: on any parse error, returns (False, False, ""). """ signal_path = project_path / _SIGNAL_FILE if not signal_path.exists(): - return False, False + return False, False, "" try: data = json.loads(signal_path.read_text(encoding="utf-8")) ts = datetime.fromisoformat(data["timestamp"]) age = (datetime.now(timezone.utc) - ts).total_seconds() recent = age <= _SIGNAL_MAX_AGE_SECS - return recent, bool(data.get("read_unlocked", False)) and recent + if not recent: + return False, False, "" + return True, bool(data.get("read_unlocked", False)), str(data.get("tool", "")) except Exception: - return False, False + return False, False, "" def _check_c3_used(project_path: Path, tool_name: str, tool_input: dict) -> tuple[bool, str]: @@ -223,10 +229,20 @@ def _check_c3_used(project_path: Path, tool_name: str, tool_input: dict) -> tupl required_cat = _TOOL_CATEGORY.get(tool_name, "read") # ── Fix 4: signal file — primary, fast, reliable ───────────────────────── - signal_recent, signal_read_unlocked = _check_signal_file(project_path) + signal_recent, signal_read_unlocked, signal_tool = _check_signal_file(project_path) if signal_recent: + # Bypass fix: for write-class tools (Edit/Write/MultiEdit), the signal + # may only unlock them when the c3 tool that wrote it actually satisfies + # this tool's prereqs (e.g. c3_edit/c3_edits/c3_agent). A read-class + # signal (c3_status, c3_search, …) must NOT unlock a native write. + if tool_name in _BLOCKED_TOOLS: + if signal_tool in allowed: + if native_target: + _record_unlock(project_path, native_target, required_cat) + return True, "signal" + # Fresh signal exists but it's not a write-prereq tool — fall through # Fix 5: Grep/Glob without file path needs a read-unlocking tool - if not native_target and tool_name in ("Grep", "Glob", "FindFiles", "SearchText"): + elif not native_target and tool_name in ("Grep", "Glob", "FindFiles", "SearchText"): if signal_read_unlocked: return True, "signal" # Signal exists but not read-unlocking (e.g. c3_memory) — fall through diff --git a/cli/tools/delegate.py b/cli/tools/delegate.py index 5e75ced..223e9e0 100644 --- a/cli/tools/delegate.py +++ b/cli/tools/delegate.py @@ -40,6 +40,7 @@ def _kill_proc_tree(proc): subprocess.run( ["taskkill", "/F", "/T", "/PID", str(proc.pid)], capture_output=True, stdin=subprocess.DEVNULL, + creationflags=subprocess.CREATE_NO_WINDOW, ) else: proc.kill() @@ -51,8 +52,10 @@ def _kill_proc_tree(proc): def _communicate_with_heartbeat(proc, timeout=45, idle_timeout=15): """communicate() replacement with idle-activity watchdog. - Monitors stderr for activity. If no stderr output for idle_timeout seconds, - kills the process early (catches MCP startup hangs). Also enforces total timeout. + Monitors both stdout and stderr for activity. If neither stream produces + output for idle_timeout seconds, kills the process early (catches MCP startup + hangs) without killing a backend that streams its answer only on stdout. + Also enforces total timeout. Returns (stdout, stderr, status) where status is 'ok', 'timeout', or 'idle_timeout'. """ @@ -71,7 +74,7 @@ def _read_stream(stream, parts, track_activity=False): except (ValueError, OSError): pass - t_out = threading.Thread(target=_read_stream, args=(proc.stdout, stdout_parts), daemon=True) + t_out = threading.Thread(target=_read_stream, args=(proc.stdout, stdout_parts, True), daemon=True) t_err = threading.Thread(target=_read_stream, args=(proc.stderr, stderr_parts, True), daemon=True) t_out.start() t_err.start() @@ -218,10 +221,17 @@ def _run_claude(task: str, context: str, cwd: str | None = None, cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL, - text=True, cwd=cwd, + text=True, encoding="utf-8", errors="replace", cwd=cwd, **_popen_kwargs(), ) - output, err = _communicate_with_heartbeat(proc, timeout=timeout, idle_timeout=idle_timeout) + output, err, status = _communicate_with_heartbeat( + proc, timeout=timeout, idle_timeout=idle_timeout, + ) + if status == "idle_timeout": + return (f"[claude:idle_timeout] No stderr activity for {idle_timeout}s " + f"(likely MCP startup hang)"), False + if status == "timeout": + return f"[claude:timeout] No response after {timeout}s", False if proc.returncode == 0 and output.strip(): return output.strip(), True return f"[claude:error] {(err or '').strip() or 'no output'}", False @@ -307,7 +317,7 @@ def _start_gemini_early(model: str, timeout: int = 45, idle_timeout: int = 15, cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, - text=True, + text=True, encoding="utf-8", errors="replace", cwd=cwd, **_popen_kwargs(), ) @@ -344,7 +354,7 @@ def _read_stream(stream, parts, track_activity=False): except (ValueError, OSError): pass - t_out = threading.Thread(target=_read_stream, args=(proc.stdout, stdout_parts), daemon=True) + t_out = threading.Thread(target=_read_stream, args=(proc.stdout, stdout_parts, True), daemon=True) t_err = threading.Thread(target=_read_stream, args=(proc.stderr, stderr_parts, True), daemon=True) t_out.start() t_err.start() @@ -448,7 +458,7 @@ def _run_gemini(task: str, context: str, model: str, cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL, - text=True, + text=True, encoding="utf-8", errors="replace", cwd=cwd, **_popen_kwargs(), ) @@ -563,7 +573,7 @@ def _run_codex(task: str, context: str, model: str, sandbox: str, cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL, - text=True, + text=True, encoding="utf-8", errors="replace", cwd=cwd, **_popen_kwargs(), ) @@ -592,25 +602,18 @@ def _run_codex_resume(follow_up: str, timeout: int = 120, """Resume last Codex session with a follow-up prompt.""" cmd = ["codex", "exec", "--skip-git-repo-check", "resume", "--last"] try: - import sys proc = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, - text=True, + text=True, encoding="utf-8", errors="replace", cwd=cwd, + **_popen_kwargs(), ) try: stdout, stderr = proc.communicate(input=follow_up, timeout=timeout) except subprocess.TimeoutExpired: - if sys.platform == "win32": - subprocess.run( - ["taskkill", "/F", "/T", "/PID", str(proc.pid)], - capture_output=True, stdin=subprocess.DEVNULL, - ) - else: - proc.kill() - proc.wait(timeout=5) + _kill_proc_tree(proc) return f"[codex:timeout] Resume timed out after {timeout}s", False if proc.returncode != 0: @@ -1166,7 +1169,10 @@ def _check_claude(): model=fallback, system=tdef["system"], temperature=tdef.get("temperature", 0.3), max_tokens=int(dcfg.get("max_tokens", 512) or 512), - ) + timeout=timeout_s) + if retry_resp is None: + # Timeout/failure on the fallback — not a valid empty answer. + continue retry_conf = _estimate_confidence(task_type, retry_resp, count_tokens(retry_resp)) if retry_conf != "low": resp = retry_resp diff --git a/cli/tools/edit.py b/cli/tools/edit.py index c78c3d4..0a32295 100644 --- a/cli/tools/edit.py +++ b/cli/tools/edit.py @@ -25,6 +25,37 @@ def _get_file_lock(path: Path) -> threading.Lock: return _file_locks[key] +def _read_preserving_newlines(path: Path) -> tuple[str, str]: + """Read a file's text and detect its dominant newline style. + + Returns (content, newline) where `content` has all line endings + normalized to ``\n`` (so existing replace logic is unchanged) and + `newline` is the EOL to write back: ``\r\n`` if CRLF dominates the + file, otherwise ``\n``. This avoids Python's text-mode write rewriting + every line to ``os.linesep`` on Windows. + """ + raw = path.read_bytes() + crlf = raw.count(b"\r\n") + lf_only = raw.count(b"\n") - crlf + newline = "\r\n" if crlf > lf_only else "\n" + content = raw.decode("utf-8") + # Normalize to \n internally so replacement matching is EOL-agnostic. + content = content.replace("\r\n", "\n").replace("\r", "\n") + return content, newline + + +def _write_preserving_newlines(path: Path, content: str, newline: str) -> None: + """Write `content` (which uses ``\n``) back using the original EOL style. + + Uses ``newline=""`` so Python performs no translation; we emit the + detected EOL explicitly so an LF-only file stays LF-only on Windows. + """ + if newline != "\n": + content = content.replace("\n", newline) + with open(path, "w", encoding="utf-8", newline="") as fh: + fh.write(content) + + # Unicode lookalike substitutions used as a fallback when the literal # old_string is not found. Strictly 1:1 (same-length) substitutions so # positions are preserved — we locate the match on the normalized string @@ -131,7 +162,10 @@ def handle_edit(file_path: str, old_string: str, new_string: str, try: path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(new_string, encoding="utf-8") + # newline="" → write content exactly as given; no os.linesep + # translation, so the caller's line endings are preserved verbatim. + with open(path, "w", encoding="utf-8", newline="") as fh: + fh.write(new_string) except Exception as e: return finalize("c3_edit", {"file": file_path}, f"Create error: {e}", "create error") @@ -161,15 +195,22 @@ def handle_edit(file_path: str, old_string: str, new_string: str, return finalize("c3_edit", {"file": file_path}, "edits must be a non-empty JSON list", "bad edits param") + if not all(isinstance(p, dict) for p in edit_list): + return finalize("c3_edit", {"file": file_path}, + "edits must be a JSON list of objects " + "({old_string, new_string, ...}); a non-object element was found", + "bad edits param") + with file_lock: try: - content = path.read_text(encoding="utf-8") + content, _newline = _read_preserving_newlines(path) except Exception as e: return finalize("c3_edit", {"file": file_path}, f"Read error: {e}", "read error") results = [] any_normalized = False + any_applied = False for i, patch in enumerate(edit_list): old = patch.get("old_string", "") new = patch.get("new_string", "") @@ -190,6 +231,7 @@ def handle_edit(file_path: str, old_string: str, new_string: str, continue content = new_content + any_applied = True n = count if r_all else 1 if used_fallback: any_normalized = True @@ -202,22 +244,27 @@ def handle_edit(file_path: str, old_string: str, new_string: str, + (" [norm]" if used_fallback else "") + f" | {desc}") - try: - path.write_text(content, encoding="utf-8") - except Exception as e: - return finalize("c3_edit", {"file": file_path}, - f"Write error: {e}", "write error") + # Only touch the file when at least one patch actually changed it — + # avoids rewriting (and re-EOL-normalizing) an unchanged file and + # logging a phantom ledger entry when every patch missed. + if any_applied: + try: + _write_preserving_newlines(path, content, _newline) + except Exception as e: + return finalize("c3_edit", {"file": file_path}, + f"Write error: {e}", "write error") # Log batch to ledger as one entry (store each patch's old/new for diff view) - batch_detail = {"patches": [ - { - "old_string": p.get("old_string", "")[:_DETAIL_CAP], - "new_string": p.get("new_string", "")[:_DETAIL_CAP], - **({"summary": p["summary"]} if p.get("summary") else {}), - } - for p in edit_list if p.get("old_string") is not None - ]} - _log_to_ledger(rel, summary or f"Batch edit: {len(edit_list)} patches", tag_list, svc, detail=batch_detail) + if any_applied: + batch_detail = {"patches": [ + { + "old_string": p.get("old_string", "")[:_DETAIL_CAP], + "new_string": p.get("new_string", "")[:_DETAIL_CAP], + **({"summary": p["summary"]} if p.get("summary") else {}), + } + for p in edit_list if p.get("old_string") is not None + ]} + _log_to_ledger(rel, summary or f"Batch edit: {len(edit_list)} patches", tag_list, svc, detail=batch_detail) applied = sum(1 for r in results if "NOT FOUND" not in r and "AMBIGUOUS" not in r and "skipped" not in r) norm_tag = " [unicode-normalized]" if any_normalized else "" @@ -234,7 +281,7 @@ def handle_edit(file_path: str, old_string: str, new_string: str, with file_lock: try: - content = path.read_text(encoding="utf-8") + content, _newline = _read_preserving_newlines(path) except Exception as e: return finalize("c3_edit", {"file": file_path}, f"Read error: {e}", "read error") @@ -260,7 +307,7 @@ def handle_edit(file_path: str, old_string: str, new_string: str, occurrences = count if replace_all else 1 try: - path.write_text(new_content, encoding="utf-8") + _write_preserving_newlines(path, new_content, _newline) except Exception as e: return finalize("c3_edit", {"file": file_path}, f"Write error: {e}", "write error") diff --git a/cli/tools/memory.py b/cli/tools/memory.py index bd824de..34e8f88 100644 --- a/cli/tools/memory.py +++ b/cli/tools/memory.py @@ -5,6 +5,10 @@ def handle_memory(action: str, query: str, fact: str, category: str, top_k: int, svc, finalize, fact_id: str = "") -> str: if action == "add": + if not fact or not fact.strip(): + return finalize("c3_memory", {"action": action}, + "fact is required to add a memory (got empty/whitespace)", + "missing fact") sid = (svc.session_mgr.current_session or {}).get("id", "") res = svc.memory.remember(fact, category or "general", sid) return finalize("c3_memory", {"action": action}, diff --git a/cli/tools/read.py b/cli/tools/read.py index 51def5e..5934bcf 100644 --- a/cli/tools/read.py +++ b/cli/tools/read.py @@ -53,10 +53,20 @@ def _coerce_lines(val: Any): except (json.JSONDecodeError, ValueError): return None return parsed if isinstance(parsed, list) else None + # Comma-separated integers like "1,2,3" → list of line numbers. + if "," in val: + try: + nums = [int(p.strip()) for p in val.split(",") if p.strip()] + except ValueError: + return None + return nums or None try: return int(val) except ValueError: - if "-" in val: # "start-end" like "22-40" + # "start-end" like "22-40". Only treat as a range when both ends + # are non-negative ints — a leading "-5" is a (rejected) negative, + # not a range. partition on the FIRST "-" preserves that. + if "-" in val: a, _, b = val.partition("-") try: return [int(a.strip()), int(b.strip())] @@ -117,7 +127,19 @@ def _worker_finalize(name, args, resp, summ, **kw): # Resolve ranges ranges = [] - if lines: + + def _add_range(start: int, end: int) -> None: + # Swap reversed ranges (e.g. "40-22") and drop non-positive line + # numbers — lines are 1-based, so a negative/zero spec is invalid + # rather than a relative offset. + if start > end: + start, end = end, start + if end < 1: + return + start = max(start, 1) + ranges.append((start, end)) + + if lines is not None: if isinstance(lines, int): line_specs = [lines] elif isinstance(lines, (list, tuple)) and len(lines) == 2 and all(isinstance(x, int) for x in lines): @@ -129,11 +151,11 @@ def _worker_finalize(name, args, resp, summ, **kw): for spec in line_specs: if isinstance(spec, int): - ranges.append((spec, spec)) + _add_range(spec, spec) elif isinstance(spec, (list, tuple)) and len(spec) >= 2: - ranges.append((int(spec[0]), int(spec[1]))) + _add_range(int(spec[0]), int(spec[1])) elif isinstance(spec, (list, tuple)) and len(spec) == 1: - ranges.append((int(spec[0]), int(spec[0]))) + _add_range(int(spec[0]), int(spec[0])) # Ensure file_memory index is fresh. # When the watcher is running, it handles updates in the background — diff --git a/cli/tools/validate.py b/cli/tools/validate.py index f605eaa..e3499be 100644 --- a/cli/tools/validate.py +++ b/cli/tools/validate.py @@ -19,20 +19,50 @@ def _popen_no_window() -> dict: kw["creationflags"] = subprocess.CREATE_NO_WINDOW return kw + def _run_with_timeout(cmd: list[str], cwd: str | None = None): + """Popen + communicate(timeout); on timeout kill the whole tree. + + subprocess.run(timeout=...) only kills the direct child, orphaning + `node` spawned by `npx tsc`. Popen + taskkill /F /T reaps the tree. + Returns (stdout, stderr) — empty strings on timeout/error. + """ + proc = subprocess.Popen( + cmd, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + stdin=subprocess.DEVNULL, + text=True, encoding="utf-8", errors="replace", + cwd=cwd, + **_popen_no_window(), + ) + try: + out, err = proc.communicate(timeout=timeout) + return out or "", err or "" + except subprocess.TimeoutExpired: + if sys.platform == "win32": + subprocess.run( + ["taskkill", "/F", "/T", "/PID", str(proc.pid)], + capture_output=True, stdin=subprocess.DEVNULL, + **_popen_no_window(), + ) + else: + proc.kill() + try: + proc.wait(timeout=5) + except Exception: + pass + return "", "" + if ext == ".py": exe = shutil.which("pyright") if not exe: return [] try: - proc = await asyncio.to_thread( - subprocess.run, + proc_out, proc_err = await asyncio.to_thread( + _run_with_timeout, [exe, "--outputjson", str(full)], - capture_output=True, text=True, timeout=timeout, - stdin=subprocess.DEVNULL, - **_popen_no_window(), ) try: - data = __import__("json").loads(proc.stdout) + data = __import__("json").loads(proc_out) diags = data.get("generalDiagnostics", []) errors = [d for d in diags if d.get("severity") == "error"] warns = [d for d in diags if d.get("severity") == "warning"] @@ -48,11 +78,11 @@ def _popen_no_window() -> dict: warnings.append(f" L{ln}: {d.get('message', '')[:80]}") except Exception: # Fallback: count error lines from plain text output - lines = (proc.stdout + proc.stderr).splitlines() + lines = (proc_out + proc_err).splitlines() errs = [l for l in lines if " - error:" in l or " error " in l.lower()] if errs: warnings.append(f"pyright: {len(errs)} issue(s) — run pyright {full.name} for details") - except (subprocess.TimeoutExpired, Exception): + except Exception: pass elif ext in (".ts", ".tsx"): @@ -65,14 +95,12 @@ def _popen_no_window() -> dict: if not tsc_args: return [] try: - proc = await asyncio.to_thread( - subprocess.run, + proc_out, proc_err = await asyncio.to_thread( + _run_with_timeout, tsc_args, - capture_output=True, text=True, timeout=timeout, - stdin=subprocess.DEVNULL, cwd=str(full.parent), - **_popen_no_window(), + str(full.parent), ) - output = proc.stdout + proc.stderr + output = proc_out + proc_err # tsc output: "file.ts(10,5): error TS2304: Cannot find..." errs = re.findall(r"error TS\d+:", output) warns = re.findall(r"warning TS\d+:", output) @@ -86,7 +114,7 @@ def _popen_no_window() -> dict: for line in output.splitlines()[:3]: if "error TS" in line: warnings.append(f" {line.strip()[:100]}") - except (subprocess.TimeoutExpired, Exception): + except Exception: pass return warnings diff --git a/core/mcp_toml.py b/core/mcp_toml.py index b0aab5a..6faa455 100644 --- a/core/mcp_toml.py +++ b/core/mcp_toml.py @@ -14,13 +14,32 @@ from pathlib import Path +def _strip_toml_comment(raw: str) -> str: + """Strip a trailing ``#`` comment, but only when the ``#`` is outside a + quoted string. + + A naive ``raw.split("#", 1)[0]`` truncates values that legitimately contain + ``#`` inside quotes (e.g. a Windows path like ``"C:/tools/c#/run.exe"``). + """ + in_single = False + in_double = False + for i, ch in enumerate(raw): + if ch == "'" and not in_double: + in_single = not in_single + elif ch == '"' and not in_single: + in_double = not in_double + elif ch == "#" and not in_single and not in_double: + return raw[:i] + return raw + + def parse_toml_mcp_servers(content: str) -> dict: """Parse ``[mcp_servers.]`` sections from TOML content into a dict.""" servers: dict = {} current_server = None for raw in content.splitlines(): - line = raw.split("#", 1)[0].strip() + line = _strip_toml_comment(raw).strip() if not line: continue @@ -67,6 +86,11 @@ def upsert_toml_section(toml_path: Path, section: str, entries: dict) -> None: content = toml_path.read_text(encoding="utf-8") if toml_path.exists() else "" header = f"[{section}]" + # When skipping the target section, also skip any of its dotted child + # subtables (e.g. ``[mcp_servers.c3.env]`` under ``[mcp_servers.c3]``). + # Otherwise a child table is left orphaned beneath the freshly re-appended + # section, corrupting the file on re-run. + child_prefix = f"{header[:-1]}." # "[mcp_servers.c3]" -> "[mcp_servers.c3." lines = content.splitlines() new_lines = [] skip = False @@ -76,7 +100,10 @@ def upsert_toml_section(toml_path: Path, section: str, entries: dict) -> None: skip = True continue if skip and stripped.startswith("["): - skip = False + # A following section header ends the skip — unless it is a dotted + # child of the target section, which we also drop. + if not stripped.startswith(child_prefix): + skip = False if not skip: new_lines.append(line) diff --git a/core/web_security.py b/core/web_security.py index 542e294..c4eb567 100644 --- a/core/web_security.py +++ b/core/web_security.py @@ -99,6 +99,12 @@ def check_request(request, allowed: set[str]) -> tuple[bool, str]: host = _hostname(getattr(request, "host", "") or "") if host and host not in allowed: return False, f"host '{host}' is not allowlisted" + # A missing/empty Host on a state-changing request can't be vetted against + # the allowlist, so it slips past the DNS-rebinding defense. When the server + # is loopback-only (the default, hardened posture), require Host presence on + # mutating methods rather than silently allowing the unverifiable request. + if not host and request.method in _MUTATING_METHODS and allowed <= _LOOPBACK_HOSTS: + return False, "missing Host header on mutating request" # 2) Origin check — anti cross-origin CSRF. Browsers always send Origin on # state-changing requests, so a mismatch is a reliable CSRF signal. When diff --git a/oracle-guide/changelog.md b/oracle-guide/changelog.md index 328bb93..d4a8042 100644 --- a/oracle-guide/changelog.md +++ b/oracle-guide/changelog.md @@ -1,5 +1,23 @@ # Oracle Changelog +## v1.2.1 (2026-06-22) + +### Security (C3 v2.39.0) +- **`POST /api/config`** now requires the Bearer token and an allowlisted key set. It was + previously unauthenticated, allowing any local process to disable Discovery auth + (`api_require_auth=false`) or repoint `ollama_base_url`. +- **`GET /api/apikey`** returns a masked token unless a valid Bearer token is presented; it + previously leaked the raw token over HTTP. `generate`/`rotate` still reveal the new token + once. +- **Discovery `project_path` validation**: project paths are validated against discovered + projects before any read (previously any `.c3` project on the machine was readable by + path). +- **MCP transport auth** now reads live config, so dashboard auth toggles apply without a + restart; chat/config endpoints return JSON errors for malformed bodies; the activity + digest now flags truncated scans. + +--- + ## v1.2.0 (2026-06-14) ### Activity Reporting (C3 v2.38.0) diff --git a/oracle/mcp_oracle.py b/oracle/mcp_oracle.py index 9130608..c2d8c36 100644 --- a/oracle/mcp_oracle.py +++ b/oracle/mcp_oracle.py @@ -83,14 +83,27 @@ class _BearerAuthMiddleware: Implemented at the ASGI layer (not Starlette's ``BaseHTTPMiddleware``) so it never buffers responses — MCP streamable-HTTP / SSE responses stream through intact. Rejects requests whose ``Authorization`` header fails ``api_auth.verify``. + + Enforcement is decided per request from live config (``api_require_auth``), + so toggling auth in the dashboard takes effect without restarting the MCP + transport — matching the Flask Discovery guard. ``require_auth`` is only the + fallback default if config can't be read. """ def __init__(self, app, require_auth: bool = True): self.app = app self.require_auth = require_auth + def _enforce(self) -> bool: + """Read live ``api_require_auth`` (falls back to the build-time value).""" + try: + from oracle.config import load_config + return bool(load_config().get("api_require_auth", self.require_auth)) + except Exception: + return self.require_auth + async def __call__(self, scope, receive, send): - if scope.get("type") == "http" and self.require_auth: + if scope.get("type") == "http" and self._enforce(): headers = dict(scope.get("headers") or []) auth = headers.get(b"authorization", b"").decode("latin-1") if not api_auth.verify(api_auth.extract_bearer(auth)): @@ -142,8 +155,10 @@ def build_app(registry, version: str = "", require_auth: bool = True, path: str """Build the Starlette ASGI app for the MCP server (auth middleware attached).""" mcp = build_mcp(registry, version) app = mcp.http_app(path=path) - if require_auth: - app.add_middleware(_BearerAuthMiddleware) + # Always install the gate; it decides per request from live config whether to + # enforce (so enabling auth in the dashboard works without a restart). The + # build-time ``require_auth`` is passed as the fallback default only. + app.add_middleware(_BearerAuthMiddleware, require_auth=require_auth) # Host-header allowlist (defense-in-depth vs DNS rebinding). Added last so it # is the outermost middleware and runs before the bearer check. from core.web_security import allowed_hostnames diff --git a/oracle/oracle_server.py b/oracle/oracle_server.py index db66d35..5cfb3dd 100644 --- a/oracle/oracle_server.py +++ b/oracle/oracle_server.py @@ -193,10 +193,32 @@ def api_config_get(): return jsonify(cfg) +# Keys that POST /api/config may set. Derived from config DEFAULTS so the +# allowlist tracks the schema; anything outside this set is rejected. Notably +# excludes nothing sensitive by accident — but the gate below still requires the +# Bearer token, so even allowlisted keys can't be flipped by an unauthenticated +# local process (e.g. POST {"api_require_auth": false} to strip Discovery auth). +from oracle.config import DEFAULTS as _CONFIG_DEFAULTS # noqa: E402 + +_CONFIG_SETTABLE_KEYS = frozenset(_CONFIG_DEFAULTS.keys()) + + @app.route("/api/config", methods=["POST"]) def api_config_set(): global _cfg + # Require the Bearer token: this endpoint can disable Discovery auth or + # repoint ollama_base_url, so it must never be reachable unauthenticated. + token = extract_bearer(request.headers.get("Authorization")) + if not verify_api_key(token): + return jsonify({"error": "unauthorized"}), 401 body = request.get_json(silent=True) or {} + if not isinstance(body, dict): + return jsonify({"error": "body must be a JSON object"}), 400 + # Reject unknown keys rather than blindly cfg.update(body) — an attacker + # could otherwise smuggle arbitrary keys into the persisted config. + unknown = sorted(k for k in body if k not in _CONFIG_SETTABLE_KEYS) + if unknown: + return jsonify({"error": "unknown config keys", "keys": unknown}), 400 cfg = load_config() cfg.update(body) save_config(cfg) @@ -515,7 +537,7 @@ def api_chat(): """Streaming chat with Oracle via SSE.""" if not _chat_engine: return jsonify({"error": "not initialized"}), 500 - data = request.get_json() or {} + data = request.get_json(silent=True) or {} message = data.get("message", "").strip() conv_id = data.get("conversation_id") or None if not message: @@ -549,7 +571,7 @@ def api_chat_conversations_list(): def api_chat_conversations_create(): if not _chat_store: return jsonify({"error": "not initialized"}), 500 - data = request.get_json() or {} + data = request.get_json(silent=True) or {} title = data.get("title") conv_id = _chat_store.create_conversation(title) return jsonify({"id": conv_id}), 201 @@ -575,7 +597,7 @@ def api_chat_conversation_delete(conv_id): def api_chat_conversation_title(conv_id): if not _chat_store: return jsonify({"error": "not initialized"}), 500 - data = request.get_json() or {} + data = request.get_json(silent=True) or {} title = data.get("title", "").strip() if not title: return jsonify({"error": "No title provided"}), 400 @@ -596,7 +618,7 @@ def api_chat_command(): """Execute a slash command.""" if not _chat_engine: return jsonify({"error": "not initialized"}), 500 - data = request.get_json() or {} + data = request.get_json(silent=True) or {} conv_id = data.get("conversation_id") command = data.get("command", "").strip() if not command: @@ -719,8 +741,15 @@ def api_discovery_mcp_info(): # ── Discovery API key management (local dashboard — unauthenticated, loopback) ── -def _apikey_status() -> dict: - """Status payload for the Discovery API token + connection info.""" +def _apikey_status(reveal: bool = False) -> dict: + """Status payload for the Discovery API token + connection info. + + ``reveal`` controls whether the unmasked token is included. It is False for + plain ``GET /api/apikey`` status reads (the raw key must never be returned + over HTTP unauthenticated) and only True when the caller either presents a + valid Bearer token or just created the key via an explicit generate/rotate + POST. When False, ``key`` is empty and the snippet carries a placeholder. + """ key = api_auth.peek() host = _cfg.get("bind_host", "127.0.0.1") port = int(_cfg.get("mcp_port", 3332)) @@ -729,18 +758,19 @@ def _apikey_status() -> dict: masked = "" if key: masked = (key[:4] + "…" + key[-4:]) if len(key) > 12 else "••••" + snippet_token = key if (key and reveal) else "" snippet = { "mcpServers": { "c3-oracle": { "type": "http", "url": url, - "headers": {"Authorization": f"Bearer {key or ''}"}, + "headers": {"Authorization": f"Bearer {snippet_token}"}, } } } return { "exists": bool(key), - "key": key or "", + "key": (key or "") if reveal else "", "masked": masked, "require_auth": bool(_cfg.get("api_require_auth", True)), "api_enabled": bool(_cfg.get("api_enabled", True)), @@ -754,29 +784,39 @@ def _apikey_status() -> dict: @app.route("/api/apikey", methods=["GET"]) def api_apikey_get(): - """Return the Discovery API token (if any) + connection info.""" + """Return Discovery API token status + connection info. + + The unmasked token is only included when the caller presents a valid Bearer + token; otherwise only the masked form is returned (never the raw key over + HTTP unauthenticated). + """ try: - return jsonify(_apikey_status()) + reveal = verify_api_key(extract_bearer(request.headers.get("Authorization"))) + return jsonify(_apikey_status(reveal=reveal)) except Exception as e: return jsonify({"error": str(e)}), 500 @app.route("/api/apikey/generate", methods=["POST"]) def api_apikey_generate(): - """Create a token if none exists; returns the current status.""" + """Create a token if none exists; returns the current status. + + Reveals the key in the response: this is an explicit local creation action, + so the dashboard can show/copy the just-created token once. + """ try: api_auth.get_or_create_key() - return jsonify(_apikey_status()) + return jsonify(_apikey_status(reveal=True)) except Exception as e: return jsonify({"error": str(e)}), 500 @app.route("/api/apikey/rotate", methods=["POST"]) def api_apikey_rotate(): - """Replace the token with a fresh one.""" + """Replace the token with a fresh one (revealed once for copy).""" try: api_auth.rotate() - return jsonify(_apikey_status()) + return jsonify(_apikey_status(reveal=True)) except Exception as e: return jsonify({"error": str(e)}), 500 diff --git a/oracle/services/activity_reporter.py b/oracle/services/activity_reporter.py index 8faddf7..19f78bf 100644 --- a/oracle/services/activity_reporter.py +++ b/oracle/services/activity_reporter.py @@ -23,6 +23,15 @@ _MIN_TS = "0000-01-01T00:00:00" _MAX_TS = "9999-12-31T23:59:59" +# Per-source scan caps. When a source returns exactly its cap, the day may have +# more rows than we scanned, so the digest surfaces ``truncated: true`` rather +# than silently undercounting a very busy day. (Paging is overkill — a flag is +# enough for a digest.) +_CAP_ACTIVITY = 10000 +_CAP_SESSIONS = 100 +_CAP_SESSION_STATS = 500 +_CAP_EDITS = 10000 + _NARRATE_SYSTEM = ( "You are C3's activity reporter. Given a JSON digest of a developer's work " "across their projects for a time window, write a concise, friendly summary " @@ -67,6 +76,9 @@ def report(self, date: str = "", since: str = "", until: str = "", "window": window, "totals": self._aggregate_totals(proj_reports), "projects": proj_reports, + # True if any project hit a per-source scan cap (counts may be + # undercounted for that project). See _CAP_* constants. + "truncated": any(pr.get("truncated") for pr in proj_reports), "narrative": None, } if narrate: @@ -131,6 +143,7 @@ def _report_project(self, proj: dict, lo: str, hi: str) -> dict: "cost_usd": 0.0, "first_activity": None, "last_activity": None, + "truncated": False, } # Guard: only read projects that already have a .c3 dir — never create # one as a side effect (ActivityLog/SessionManager mkdir on init). @@ -142,7 +155,10 @@ def _report_project(self, proj: dict, lo: str, hi: str) -> dict: # Activity log → per-type event counts. try: counts: dict[str, int] = {} - for e in ActivityLog(path).get_recent(limit=10000, since=lo, until=hi): + rows = list(ActivityLog(path).get_recent(limit=_CAP_ACTIVITY, since=lo, until=hi)) + if len(rows) >= _CAP_ACTIVITY: + base["truncated"] = True + for e in rows: etype = e.get("type", "unknown") counts[etype] = counts.get(etype, 0) + 1 if e.get("timestamp"): @@ -156,7 +172,10 @@ def _report_project(self, proj: dict, lo: str, hi: str) -> dict: try: sm = SessionManager(path) # Sessions started within the window. - for s in sm.list_sessions(100): + sess_rows = list(sm.list_sessions(_CAP_SESSIONS)) + if len(sess_rows) >= _CAP_SESSIONS: + base["truncated"] = True + for s in sess_rows: started = s.get("started", "") if started and lo <= started <= hi: base["sessions"].append({ @@ -169,7 +188,10 @@ def _report_project(self, proj: dict, lo: str, hi: str) -> dict: }) timestamps.append(started) # Token / cost from hook-captured session stats. - for st in sm.get_session_stats(500): + stat_rows = list(sm.get_session_stats(_CAP_SESSION_STATS)) + if len(stat_rows) >= _CAP_SESSION_STATS: + base["truncated"] = True + for st in stat_rows: ts = st.get("ts", "") if ts and lo <= ts <= hi: base["tokens"]["input"] += int(st.get("input_tokens", 0) or 0) @@ -180,7 +202,10 @@ def _report_project(self, proj: dict, lo: str, hi: str) -> dict: # Edit ledger → edits vs git mutations (get_history filters >= lo only). try: - for en in EditLedger(path).get_history(since=lo, limit=10000): + edit_rows = list(EditLedger(path).get_history(since=lo, limit=_CAP_EDITS)) + if len(edit_rows) >= _CAP_EDITS: + base["truncated"] = True + for en in edit_rows: ts = en.get("timestamp", "") if ts and ts > hi: continue diff --git a/oracle/services/c3_bridge.py b/oracle/services/c3_bridge.py index c33246c..eaf7c38 100644 --- a/oracle/services/c3_bridge.py +++ b/oracle/services/c3_bridge.py @@ -29,6 +29,29 @@ def _noop_facts(*_a, **_kw) -> str: return "" +def validate_project_path(scanner, project_path: str) -> str: + """Resolve ``project_path`` and confirm it is a discovered C3 project. + + Returns the resolved absolute path. Raises ``ValueError`` if the path is + empty or is not a member of ``scanner.discover()`` — preventing any caller + (chat tools, Discovery API) from reading an arbitrary ``.c3`` project on the + machine simply by supplying its path. + """ + if not project_path: + raise ValueError("project_path is required") + resolved = str(Path(project_path).resolve()) + try: + discovered = scanner.discover() if scanner else [] + except Exception: + discovered = [] + known = {str(Path(p.get("path", "")).resolve()) for p in discovered if p.get("path")} + if resolved not in known: + raise ValueError( + f"Unknown project: {project_path}. Not a registered C3 project." + ) + return resolved + + class C3Bridge: """Bridge between Oracle and C3 tool handlers with per-project runtime cache.""" @@ -43,8 +66,12 @@ def __init__(self, scanner): # ── Runtime cache ────────────────────────────────────────────── def get_runtime(self, project_path: str) -> C3Runtime: - """Return cached runtime or build a new one. LRU eviction at MAX_CACHED.""" - project_path = str(Path(project_path).resolve()) + """Return cached runtime or build a new one. LRU eviction at MAX_CACHED. + + ``project_path`` is validated against discovered projects first, so a + runtime can never be built for an arbitrary path on the machine. + """ + project_path = validate_project_path(self.scanner, project_path) with self._lock: if project_path in self._runtimes: self._access_order.remove(project_path) diff --git a/oracle/services/chat_engine.py b/oracle/services/chat_engine.py index 6cd379c..ed2645e 100644 --- a/oracle/services/chat_engine.py +++ b/oracle/services/chat_engine.py @@ -16,6 +16,7 @@ _agent_tls = threading.local() from oracle.config import load_config +from oracle.services.c3_bridge import validate_project_path from oracle.services.chat_store import ChatStore from oracle.services.cross_memory import CrossMemory from oracle.services.health_checker import HealthChecker @@ -921,6 +922,7 @@ def _tool_list_projects(self) -> dict: def _tool_query_memory( self, project_path: str, query: str = "", category: str = "", limit: int = 10 ) -> dict: + project_path = validate_project_path(self.scanner, project_path) facts = self.reader.read_facts(project_path) if category: facts = [f for f in facts if f.get("category", "") == category] @@ -976,9 +978,11 @@ def _tool_search_facts(self, query: str, limit: int = 20) -> dict: return {"query": query, "total_matches": len(all_matches), "results": top} def _tool_project_health(self, project_path: str) -> dict: + project_path = validate_project_path(self.scanner, project_path) return self.health_checker.check(project_path) def _tool_analyze_project(self, project_path: str) -> dict: + project_path = validate_project_path(self.scanner, project_path) return self.insight_engine.analyze_project(project_path) def _tool_cross_insights(self, project_path: str = "") -> dict: @@ -1004,11 +1008,13 @@ def _tool_cross_insights(self, project_path: str = "") -> dict: def _tool_suggest_action( self, project_path: str, action: str, fact_ids: list, reason: str ) -> dict: + project_path = validate_project_path(self.scanner, project_path) data = {"fact_ids": fact_ids, "reason": reason} suggestion = self.writer.suggest(project_path, action, data) return {"suggestion_id": suggestion.get("id"), "status": "pending", "type": action} def _tool_read_graph(self, project_path: str) -> dict: + project_path = validate_project_path(self.scanner, project_path) return self.reader.get_graph_stats(project_path) def _tool_activity_report(self, date: str = "", since: str = "", until: str = "", diff --git a/pyproject.toml b/pyproject.toml index bb09d15..61fdffe 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "code-context-control" -version = "2.38.1" +version = "2.39.0" description = "Local code-intelligence layer for AI coding tools (Claude Code, Codex, Gemini, Copilot). Retrieve less, read less, edit safer." readme = "README.md" requires-python = ">=3.10" diff --git a/services/claude_md.py b/services/claude_md.py index 7e6caeb..1abbe32 100644 --- a/services/claude_md.py +++ b/services/claude_md.py @@ -81,8 +81,11 @@ # First line of every legacy (marker-less) C3 instruction doc. Used to detect # and replace pre-marker C3 content instead of leaving it duplicated above the -# new block. Both the compact and nano workflows start with this heading. -C3_LEGACY_FIRST_LINE = "## C3 Tools" +# new block. Both the compact and nano workflows start with this exact heading, +# so the full string (not just "## C3 Tools") is required to avoid mistaking a +# genuine user file that merely opens with a "## C3 Tools" heading for a +# replaceable legacy doc. +C3_LEGACY_FIRST_LINE = "## C3 Tools — MANDATORY" def wrap_c3_block(content: str) -> str: @@ -98,25 +101,37 @@ def merge_c3_block(existing: str, new_block: str) -> str: 1. If the C3 markers are already present, replace only the marked region and preserve everything the user wrote before and after it. - 2. Legacy (marker-less) C3 docs are recognised by their leading - ``## C3 Tools`` heading and replaced wholesale, while any trailing - ``# User Notes`` section (the pre-marker convention) is preserved. + 2. Legacy (marker-less) C3 docs are recognised by their full leading + ``## C3 Tools — MANDATORY`` heading (and the absence of markers) and + replaced wholesale, while any trailing ``# User Notes`` section (the + pre-marker convention) is preserved. 3. A genuine, user-authored file with neither markers nor the legacy signature is never overwritten — the C3 block is appended below it. """ new_block = new_block.strip() # 1. Markers present → surgical in-place replacement. + # + # Use the FIRST BEGIN and the LAST END so a region containing duplicated + # or nested markers collapses to a single clean block. Guard against + # out-of-order markers (an END that precedes the BEGIN): in that corrupt + # case the slice would overlap, so fall through to the append/rewrite + # paths below instead of producing garbage. if C3_BLOCK_BEGIN in existing and C3_BLOCK_END in existing: start = existing.index(C3_BLOCK_BEGIN) - end = existing.index(C3_BLOCK_END) + len(C3_BLOCK_END) - before = existing[:start].rstrip() - after = existing[end:].lstrip() - parts = [p for p in (before, new_block, after) if p] - return "\n\n".join(parts) + "\n" + end = existing.rindex(C3_BLOCK_END) + len(C3_BLOCK_END) + if end > start: + before = existing[:start].rstrip() + after = existing[end:].lstrip() + parts = [p for p in (before, new_block, after) if p] + return "\n\n".join(parts) + "\n" # 2. Legacy marker-less C3 doc → replace head, keep trailing user notes. - if existing.lstrip().startswith(C3_LEGACY_FIRST_LINE): + # Require BOTH the full legacy signature AND the absence of the managed + # markers, so a marker-bearing file (handled above, or one with corrupt + # out-of-order markers) is never wholesale-replaced here. + has_markers = C3_BLOCK_BEGIN in existing or C3_BLOCK_END in existing + if not has_markers and existing.lstrip().startswith(C3_LEGACY_FIRST_LINE): tail = "" if "# User Notes" in existing: tail = existing[existing.index("# User Notes"):].strip() diff --git a/services/compressor.py b/services/compressor.py index 340c033..63e5d9d 100644 --- a/services/compressor.py +++ b/services/compressor.py @@ -377,7 +377,11 @@ def _render_node(node_list, depth=0): decl = decl[:-1].strip() if stype == "class": - extracted.append(f"\n{prefix}class {name}:") + # Use the language-appropriate declaration (preserves + # extends/implements/generics/visibility for JS/TS/Java/ + # Go/Rust). Fall back to Python-style only when empty. + class_decl = decl if decl else f"class {name}:" + extracted.append(f"\n{prefix}{class_decl}") else: extracted.append(f"{prefix}{decl}") diff --git a/services/context_snapshot.py b/services/context_snapshot.py index 3506618..ee433b2 100644 --- a/services/context_snapshot.py +++ b/services/context_snapshot.py @@ -5,6 +5,8 @@ provides compact briefings to reinstate context after /clear. """ import json +import os +import time from datetime import datetime, timezone from pathlib import Path @@ -142,8 +144,22 @@ def capture(self, session_mgr, memory_store, } path = self.data_dir / f"snap_{snapshot['snapshot_id']}.json" - with open(path, 'w', encoding='utf-8') as f: - json.dump(snapshot, f, indent=2) + # Atomic write: a partial snapshot (crash mid-dump) is the newest file + # and would make _load_snapshot("latest") raise. temp + os.replace + # guarantees the snapshot file is either absent or complete. + tmp = path.with_suffix(f".json.tmp-{os.getpid()}-{int(time.time() * 1000)}") + try: + with open(tmp, 'w', encoding='utf-8') as f: + json.dump(snapshot, f, indent=2) + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) + finally: + try: + if tmp.exists(): + tmp.unlink() + except Exception: + pass token_count = count_tokens(json.dumps(snapshot)) return { @@ -260,20 +276,34 @@ def search(self, query: str, top_k: int = 5) -> list: return results def _load_snapshot(self, snapshot_id: str) -> dict: - """Load a snapshot by ID or 'latest'.""" + """Load a snapshot by ID or 'latest'. + + For 'latest', skip past any corrupt/partial files to the next-newest + good snapshot instead of raising. + """ if snapshot_id == "latest": files = sorted(self.data_dir.glob("snap_*.json"), reverse=True) if not files: return {"error": "No snapshots found"} - path = files[0] - else: - path = self.data_dir / f"snap_{snapshot_id}.json" - + last_error = None + for path in files: + try: + with open(path, encoding='utf-8') as f: + return json.load(f) + except Exception as exc: # corrupt/partial — try next-newest + last_error = exc + continue + return {"error": f"No readable snapshots found: {last_error}"} + + path = self.data_dir / f"snap_{snapshot_id}.json" if not path.exists(): return {"error": f"Snapshot not found: {snapshot_id}"} - with open(path, encoding='utf-8') as f: - return json.load(f) + try: + with open(path, encoding='utf-8') as f: + return json.load(f) + except Exception as exc: + return {"error": f"Snapshot is corrupt: {snapshot_id}: {exc}"} def _full_briefing(self, snap: dict) -> str: """Level 0: Full briefing with all details.""" diff --git a/services/conversation_store.py b/services/conversation_store.py index e0a12b7..0446dc4 100644 --- a/services/conversation_store.py +++ b/services/conversation_store.py @@ -14,7 +14,10 @@ import gzip import json import math +import os import re +import sys +import threading import time from collections import Counter from datetime import datetime @@ -40,7 +43,9 @@ def __init__(self, project_path: str): self.store_dir = self.project_path / ".c3" / "conversations" self.store_dir.mkdir(parents=True, exist_ok=True) self._sessions_file = self.store_dir / "sessions.json" - self._sessions: list = [] # in-memory cache, cleared on write + # None = not loaded yet; [] is a legitimate "loaded, empty" state. + self._sessions: list | None = None + self._sessions_lock = threading.Lock() self._search_index = TextIndex() self._search_meta: dict[str, dict] = {} self._search_dirty = True @@ -162,33 +167,35 @@ def add_turn(self, session_id: str, role: str, text: str, with open(session_file, "a", encoding="utf-8") as f: f.write(json.dumps(turn, ensure_ascii=False) + "\n") - # Update session index - sessions = self._load_sessions() - existing = next((s for s in sessions if s["session_id"] == session_id), None) - if existing: - existing["turns"] = existing.get("turns", 0) + 1 - existing["ended"] = ts - existing["user_tokens"] = existing.get("user_tokens", 0) + (tokens if role == "user" else 0) - existing["assistant_tokens"] = existing.get("assistant_tokens", 0) + (tokens if role == "assistant" else 0) - if source and existing.get("source") in (None, "", "manual"): - existing["source"] = self._normalize_source(source) - if role == "user" and existing.get("turns", 0) == 1: - existing["title"] = text[:100].replace("\n", " ") - else: - sessions.append({ - "session_id": session_id, - "title": (text[:100].replace("\n", " ") if role == "user" else session_id[:24]), - "source": self._normalize_source(source), - "source_file": None, - "source_mtime": 0, - "started": ts, - "ended": ts, - "turns": 1, - "user_tokens": tokens if role == "user" else 0, - "assistant_tokens": tokens if role == "assistant" else 0, - "compressed": False, - }) - self._save_sessions(sessions) + # Update session index — read-modify-write must be atomic so concurrent + # add_turn() calls don't lose count/token increments. + with self._sessions_lock: + sessions = self._load_sessions() + existing = next((s for s in sessions if s["session_id"] == session_id), None) + if existing: + existing["turns"] = existing.get("turns", 0) + 1 + existing["ended"] = ts + existing["user_tokens"] = existing.get("user_tokens", 0) + (tokens if role == "user" else 0) + existing["assistant_tokens"] = existing.get("assistant_tokens", 0) + (tokens if role == "assistant" else 0) + if source and existing.get("source") in (None, "", "manual"): + existing["source"] = self._normalize_source(source) + if role == "user" and existing.get("turns", 0) == 1: + existing["title"] = text[:100].replace("\n", " ") + else: + sessions.append({ + "session_id": session_id, + "title": (text[:100].replace("\n", " ") if role == "user" else session_id[:24]), + "source": self._normalize_source(source), + "source_file": None, + "source_mtime": 0, + "started": ts, + "ended": ts, + "turns": 1, + "user_tokens": tokens if role == "user" else 0, + "assistant_tokens": tokens if role == "assistant" else 0, + "compressed": False, + }) + self._save_sessions(sessions) self._search_dirty = True return turn @@ -736,32 +743,60 @@ def _upsert_session(self, meta: dict): self._save_sessions(sessions) def _load_sessions(self) -> list: - if self._sessions: + # None = not loaded yet; an empty list is a valid loaded state and must + # NOT trigger a re-read (which previously re-parsed the file every call). + if self._sessions is not None: return self._sessions if self._sessions_file.exists(): try: with open(self._sessions_file, encoding="utf-8") as f: loaded = json.load(f) - changed = False - for s in loaded: - src = s.get("source") - if not src: - src = "claude" if s.get("source_file") else "manual" - s["source"] = src - changed = True - norm = self._normalize_source(src) - if norm != src: - s["source"] = norm - changed = True - self._sessions = loaded - if changed: - self._save_sessions(self._sessions) - return self._sessions - except Exception: - pass + except Exception as exc: + # Corrupt/partial file: back it up and surface loudly. Do NOT + # silently reset to [] — that would let the next save persist an + # empty catalog and wipe the session history permanently. + self._backup_corrupt_sessions() + if self._sessions is not None: + # Keep whatever was already loaded in this process. + return self._sessions + raise RuntimeError( + f"sessions.json is corrupt and was backed up: {exc}" + ) from exc + changed = False + for s in loaded: + src = s.get("source") + if not src: + src = "claude" if s.get("source_file") else "manual" + s["source"] = src + changed = True + norm = self._normalize_source(src) + if norm != src: + s["source"] = norm + changed = True + self._sessions = loaded + if changed: + self._save_sessions(self._sessions) + return self._sessions self._sessions = [] return self._sessions + def _backup_corrupt_sessions(self): + """Move a corrupt sessions.json aside so it is never overwritten.""" + try: + n = 0 + while True: + bak = self._sessions_file.with_suffix(f".json.corrupt-{n}") + if not bak.exists(): + break + n += 1 + os.replace(self._sessions_file, bak) + print( + f"[conversation_store] corrupt sessions.json backed up to {bak.name}", + file=sys.stderr, + ) + except Exception: + pass + def _ensure_search_index(self): if not self._search_dirty: return @@ -800,8 +835,24 @@ def _ensure_search_index(self): def _save_sessions(self, sessions: list): self._sessions = sessions - with open(self._sessions_file, "w", encoding="utf-8") as f: - json.dump(sessions, f, ensure_ascii=False, indent=2) + # Atomic write: serialize to a temp file in the same dir, then + # os.replace() (atomic on both Windows and POSIX). This prevents a + # crash mid-write from leaving a truncated/partial sessions.json. + tmp = self._sessions_file.with_suffix( + f".json.tmp-{os.getpid()}-{int(time.time() * 1000)}" + ) + try: + with open(tmp, "w", encoding="utf-8") as f: + json.dump(sessions, f, ensure_ascii=False, indent=2) + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, self._sessions_file) + finally: + try: + if tmp.exists(): + tmp.unlink() + except Exception: + pass self._search_dirty = True def _chunk_text(self, text: str) -> list[str]: diff --git a/services/edit_ledger.py b/services/edit_ledger.py index e0f4962..fa6323f 100644 --- a/services/edit_ledger.py +++ b/services/edit_ledger.py @@ -11,6 +11,7 @@ import subprocess import sys import threading +import uuid from collections import Counter from datetime import datetime, timezone from pathlib import Path @@ -90,7 +91,7 @@ def log_edit(self, file: str, change_type: str, summary: str, self._version_cache[rel] = new_v entry = { - "id": f"edit_{now.strftime('%Y%m%d_%H%M%S')}_{self._seq_counter:03d}", + "id": f"edit_{now.strftime('%Y%m%d_%H%M%S')}_{self._seq_counter:03d}_{uuid.uuid4().hex[:4]}", "timestamp": now.isoformat(), "session_id": session_id or "", "file": rel, @@ -104,8 +105,9 @@ def log_edit(self, file: str, change_type: str, summary: str, } if detail: entry["detail"] = detail - with open(self.ledger_file, "a", encoding="utf-8") as f: - f.write(json.dumps(entry) + "\n") + with self._write_lock: + with open(self.ledger_file, "a", encoding="utf-8") as f: + f.write(json.dumps(entry) + "\n") if self._total_count is not None: self._total_count += 1 return entry @@ -159,33 +161,44 @@ def get_version(self, file: str) -> str: return f"v{v}" if v > 0 else "v0" def tag_edit(self, edit_id: str, tag: str) -> bool: - """Add a tag to an existing edit. Rewrites the entry in-place.""" + """Add a tag to an existing edit by appending a patch entry. + + Append-only: rather than rewriting the whole file (which races with + concurrent appends from the hook process and enrich/validate), this + writes a ``{"target_id": id, "tags_add": [tag]}`` patch that + ``_load_merged`` applies. Returns True if the target edit exists. + """ if not self.ledger_file.exists(): return False - lines = self.ledger_file.read_text(encoding="utf-8").splitlines() + # Confirm the target base entry exists before appending a patch. found = False - new_lines = [] - for line in lines: - if not line.strip(): - new_lines.append(line) - continue + try: + for line in self.ledger_file.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + try: + entry = json.loads(line) + except (json.JSONDecodeError, ValueError): + continue + if "target_id" not in entry and entry.get("id") == edit_id: + found = True + break + except Exception: + return False + if not found: + return False + patch = { + "target_id": edit_id, + "tags_add": [tag], + "tagged_at": datetime.now(timezone.utc).isoformat(), + } + with self._write_lock: try: - entry = json.loads(line) - except json.JSONDecodeError: - new_lines.append(line) - continue - if entry.get("id") == edit_id: - tags = entry.get("tags", []) - if tag not in tags: - tags.append(tag) - entry["tags"] = tags - found = True - new_lines.append(json.dumps(entry)) - else: - new_lines.append(line) - if found: - self.ledger_file.write_text("\n".join(new_lines) + "\n", encoding="utf-8") - return found + with open(self.ledger_file, "a", encoding="utf-8") as f: + f.write(json.dumps(patch) + "\n") + except Exception: + return False + return True # ── Private helpers ─────────────────────────────────────────────── @@ -323,9 +336,11 @@ def _load_merged(self, file_filter: str = None, since_filter: str = None, except Exception: return [] - # Apply patches — each patch carries git enrichment and/or validation data + # Apply patches — each patch carries git enrichment, validation, and/or tags + orphaned: list = [] for target_id, patch_list in patches.items(): if target_id not in base: + orphaned.append(target_id) continue for patch in patch_list: if "git" in patch: @@ -336,6 +351,22 @@ def _load_merged(self, file_filter: str = None, since_filter: str = None, base[target_id]["valid"] = patch["valid"] if patch.get("errors"): base[target_id]["lint_errors"] = patch["errors"] + if patch.get("tags_add"): + tags = base[target_id].get("tags") or [] + for t in patch["tags_add"]: + if t not in tags: + tags.append(t) + base[target_id]["tags"] = tags + if orphaned: + # Best-effort diagnostics — never let logging break the read path. + try: + print( + f"[edit_ledger] {len(orphaned)} orphaned patch target_id(s) " + f"with no base entry: {orphaned[:10]}", + file=sys.stderr, + ) + except Exception: + pass norm_file = file_filter.replace("\\", "/") if file_filter else None results = [] diff --git a/services/file_memory.py b/services/file_memory.py index ebc73cd..ea935a8 100644 --- a/services/file_memory.py +++ b/services/file_memory.py @@ -9,6 +9,7 @@ import hashlib import json import re +import threading import time from pathlib import Path from typing import Optional @@ -44,7 +45,16 @@ def __init__(self, project_path: str): self._diag_path = self.store_dir / "_diagnostics.jsonl" self._map_cache = {} self._search_index = TextIndex() - self._rebuild_search_index() + # Building the search index reads every tracked record off disk — for a + # large project that is tens of thousands of files (~2x that many reads + # via list_tracked + get), far too heavy for the constructor, which sits + # on the MCP handshake path. Defer it: built lazily on first search(), + # mirroring ConversationStore._ensure_search_index. + self._search_dirty = True + # Guards the lazy index: the first search() build can race a background + # update()'s add_or_update on the shared TextIndex. Reentrant so + # search() may hold it across _ensure_search_index() + the query. + self._search_lock = threading.RLock() def get(self, rel_path: str) -> Optional[dict]: """Load a file's memory record, or None if not tracked.""" @@ -93,7 +103,9 @@ def update(self, rel_path: str, ai_summary: str = None) -> Optional[dict]: existing["summary"] = ai_summary existing["updated_at"] = time.strftime("%Y-%m-%dT%H:%M:%S") self._save(rel_path, existing) - self._search_index.add_or_update(rel_path, self._search_doc(existing)) + with self._search_lock: + if not self._search_dirty: + self._search_index.add_or_update(rel_path, self._search_doc(existing)) self._cache_map(rel_path, existing) return existing # If we are here, we are forcing a fresh extraction @@ -115,7 +127,9 @@ def update(self, rel_path: str, ai_summary: str = None) -> Optional[dict]: self._save(rel_path, record) self._cache_map(rel_path, record) - self._search_index.add_or_update(rel_path, self._search_doc(record)) + with self._search_lock: + if not self._search_dirty: + self._search_index.add_or_update(rel_path, self._search_doc(record)) return record def get_map(self, rel_path: str) -> Optional[str]: @@ -262,8 +276,11 @@ def list_tracked(self) -> list: return [p for p in tracked if p] def search(self, query: str, top_k: int = 5) -> list[dict]: + with self._search_lock: + self._ensure_search_index() + ranked = self._search_index.search(query, top_k=top_k) results = [] - for rel_path, score in self._search_index.search(query, top_k=top_k): + for rel_path, score in ranked: record = self.get(rel_path) if not record: continue @@ -366,6 +383,18 @@ def _search_doc(self, record: dict) -> str: fields.append(child.get("type", "")) return " ".join(str(field) for field in fields if field) + def _ensure_search_index(self): + """Build the search index on first use (deferred off __init__). + + The build reads every tracked record from disk, so it must never run on + the constructor / MCP handshake path. Mirrors ConversationStore. + """ + with self._search_lock: + if not self._search_dirty: + return + self._rebuild_search_index() + self._search_dirty = False + def _rebuild_search_index(self): docs = {} for rel_path in self.list_tracked(): @@ -519,12 +548,53 @@ def _find_python_block_end(self, lines: list, start: int) -> int: return len(lines) def _find_brace_block_end(self, lines: list, start: int) -> int: - """Find end of a brace-delimited block.""" + """Find end of a brace-delimited block. + + Counts braces only in *code*, skipping any ``{`` / ``}`` that appear + inside string/char/template literals or line/block comments. Without + this, a brace inside a string (e.g. ``log("}")``) before the real close + would prematurely zero the depth and truncate the block's line range. + """ depth = 0 found_open = False + # Cross-line state: which delimiter we're inside, and whether we're in a + # block comment. Strings (", ', `) honor backslash escapes; block + # comments and backtick (template/raw) strings span lines. + in_string = None # the active quote char, or None + in_block_comment = False for i in range(start, len(lines)): line = lines[i] - for ch in line: + j = 0 + n = len(line) + while j < n: + ch = line[j] + if in_block_comment: + if ch == '*' and j + 1 < n and line[j + 1] == '/': + in_block_comment = False + j += 2 + continue + j += 1 + continue + if in_string is not None: + if ch == '\\' and in_string != '`': + # Escaped char inside a normal string/char literal. + j += 2 + continue + if ch == in_string: + in_string = None + j += 1 + continue + # Not in a string or block comment — look for openers. + if ch == '/' and j + 1 < n and line[j + 1] == '/': + break # rest of the line is a line comment + if ch == '/' and j + 1 < n and line[j + 1] == '*': + in_block_comment = True + j += 2 + continue + if ch in ('"', "'", '`'): + in_string = ch + j += 1 + continue if ch == '{': depth += 1 found_open = True @@ -532,6 +602,7 @@ def _find_brace_block_end(self, lines: list, start: int) -> int: depth -= 1 if found_open and depth == 0: return i + 1 # 1-indexed + j += 1 return len(lines) def _normalize_type(self, kind: str) -> str: diff --git a/services/parser.py b/services/parser.py index 9099b87..ba11fb8 100644 --- a/services/parser.py +++ b/services/parser.py @@ -235,6 +235,18 @@ def _extract_docstring_js(node, lines: List[str]) -> Optional[str]: prev = prev.prev_sibling return None +class _SyntheticParent: + """Lightweight stand-in node whose only role is to expose a single child. + + Used so an exported declaration node can be re-fed through _walk_js_ts's + child-iteration loop and recorded as a section (rather than descended into). + """ + __slots__ = ("children",) + + def __init__(self, child): + self.children = [child] + + def _walk_js_ts(node, lines: List[str], sections: List[Dict[str, Any]], parent_section=None): for child in node.children: if child.type in ('class_declaration', 'abstract_class_declaration', 'interface_declaration', 'type_alias_declaration', 'enum_declaration'): @@ -365,10 +377,12 @@ def _walk_js_ts(node, lines: List[str], sections: List[Dict[str, Any]], parent_s sections.append(section) elif child.type == 'export_statement': - # Drill down + # Process the exported declaration node itself (not its children). + # Recursing into decl.children would descend INTO the declaration and + # skip recording the exported class/function/const as a section. decl = next((c for c in child.children if c.type != 'export' and c.type != 'default'), None) if decl: - _walk_js_ts(decl, lines, sections, parent_section) + _walk_js_ts(_SyntheticParent(decl), lines, sections, parent_section) def _walk_html(node, lines: List[str], sections: List[Dict[str, Any]]): """Extract headings and elements with IDs from HTML AST.""" @@ -710,13 +724,25 @@ def _subproc_check(cmd: List[str], content: str, suffix: str, checker: str, time if _sys.platform == "win32": kwargs["creationflags"] = _subprocess.CREATE_NO_WINDOW - r = _subprocess.run(cmd + [tmp], stdin=_subprocess.DEVNULL, capture_output=True, text=True, timeout=timeout, **kwargs) + # Popen + _kill_proc_tree on timeout: subprocess.run's timeout only kills + # the direct child, leaving any grandchild processes (e.g. Rscript -> R) + # running. encoding/errors avoid UnicodeDecodeError on non-UTF-8 output. + proc = _subprocess.Popen( + cmd + [tmp], stdin=_subprocess.DEVNULL, + stdout=_subprocess.PIPE, stderr=_subprocess.PIPE, + text=True, encoding="utf-8", errors="replace", **kwargs, + ) + try: + out, err = proc.communicate(timeout=timeout) + except _subprocess.TimeoutExpired: + _kill_proc_tree(proc) + return _result("checker_timeout", checker, detail=f"{cmd[0]} exceeded {timeout}s.") return { "status": "ok", "checker": checker, - "returncode": r.returncode, - "stdout": r.stdout[:65536], - "stderr": r.stderr[:65536], + "returncode": proc.returncode, + "stdout": (out or "")[:65536], + "stderr": (err or "")[:65536], } except FileNotFoundError: return _result("checker_unavailable", checker, detail=f"{cmd[0]} is not installed or not on PATH.") diff --git a/tests/test_activity_reporter.py b/tests/test_activity_reporter.py index fcfd5e3..55f4e71 100644 --- a/tests/test_activity_reporter.py +++ b/tests/test_activity_reporter.py @@ -125,6 +125,21 @@ def test_window_label_marks_today(self): d = self.reporter.report(date=today) self.assertIn("today", d["window"]["label"]) + def test_not_truncated_under_cap(self): + d = self.reporter.report(date=self.DAY) + self.assertFalse(d["truncated"]) + self.assertFalse(d["projects"][0]["truncated"]) + + def test_truncated_flag_when_activity_cap_hit(self): + # Drop the activity-log cap below the row count so the scan caps out. + from unittest import mock + + from oracle.services import activity_reporter as ar + with mock.patch.object(ar, "_CAP_ACTIVITY", 3): + d = self.reporter.report(date=self.DAY) + self.assertTrue(d["truncated"]) + self.assertTrue(d["projects"][0]["truncated"]) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_claude_md_merge.py b/tests/test_claude_md_merge.py index 42996f7..50ae86a 100644 --- a/tests/test_claude_md_merge.py +++ b/tests/test_claude_md_merge.py @@ -72,6 +72,45 @@ def test_genuine_user_file_is_appended_not_destroyed(self): # User content stays above the appended C3 block. self.assertLess(merged.index("Our House Rules"), merged.index(C3_BLOCK_BEGIN)) + def test_out_of_order_markers_do_not_corrupt(self): + # END appears BEFORE BEGIN (corrupt/hand-mangled file). The surgical + # slice would overlap, so we must NOT do the in-place replace; instead + # fall through and append a single clean block. + corrupt = ( + "# My header\nkeep me\n\n" + + C3_BLOCK_END + "\nstray\n" + C3_BLOCK_BEGIN + "\nold body\n" + ) + merged = merge_c3_block(corrupt, wrap_c3_block("NEW C3 BODY")) + self.assertIn("# My header", merged) + self.assertIn("keep me", merged) + self.assertIn("NEW C3 BODY", merged) + # A valid, well-ordered block now exists. + self.assertLess(merged.index(C3_BLOCK_BEGIN), merged.rindex(C3_BLOCK_END)) + + def test_duplicate_blocks_collapse_to_one(self): + # Two marked regions (e.g. from an old buggy run) collapse to a single + # block: first BEGIN .. last END is replaced wholesale. + existing = ( + wrap_c3_block("FIRST OLD") + "\n\nmiddle user text\n\n" + + wrap_c3_block("SECOND OLD") + ) + merged = merge_c3_block(existing, wrap_c3_block("NEW C3 BODY")) + self.assertEqual(merged.count(C3_BLOCK_BEGIN), 1) + self.assertEqual(merged.count(C3_BLOCK_END), 1) + self.assertIn("NEW C3 BODY", merged) + self.assertNotIn("FIRST OLD", merged) + self.assertNotIn("SECOND OLD", merged) + + def test_user_file_opening_with_short_c3_tools_heading_preserved(self): + # A genuine user file that merely opens with "## C3 Tools" (NOT the full + # legacy "## C3 Tools — MANDATORY" signature) must be preserved, not + # wholesale-replaced as a legacy doc. + user_file = "## C3 Tools I Like\nMy own notes about tooling.\n" + merged = merge_c3_block(user_file, wrap_c3_block(C3_CONTENT)) + self.assertIn("## C3 Tools I Like", merged) + self.assertIn("My own notes about tooling.", merged) + self.assertIn(C3_BLOCK_BEGIN, merged) + class TestWriteC3InstructionDoc(unittest.TestCase): def setUp(self): diff --git a/tests/test_edit_ledger_hook.py b/tests/test_edit_ledger_hook.py new file mode 100644 index 0000000..0accde8 --- /dev/null +++ b/tests/test_edit_ledger_hook.py @@ -0,0 +1,88 @@ +"""Edit-ledger hook regressions. + +Covers two correctness fixes in cli/hook_edit_ledger.py + cli/_hook_utils.py: + #3 NotebookEdit path (`notebook_path`) is extracted, so notebook edits get logged. + #4 Generated edit ids carry a random suffix to avoid same-second collisions + between the hook process and the server (services/edit_ledger.py). +""" +import json +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO_ROOT)) + +from cli._hook_utils import get_tool_input_path # noqa: E402 + + +def _run_ledger_hook(project: Path, tool_name: str, tool_input: dict) -> str: + """Spawn the real ledger hook against a temp project; return ledger text.""" + payload = json.dumps({"tool_name": tool_name, "tool_input": tool_input}) + subprocess.run( + [sys.executable, str(REPO_ROOT / "cli" / "hook_edit_ledger.py")], + input=payload, capture_output=True, text=True, + encoding="utf-8", cwd=str(project), timeout=20, + ) + ledger = project / ".c3" / "edit_ledger.jsonl" + return ledger.read_text(encoding="utf-8") if ledger.exists() else "" + + +class TestGetToolInputPath(unittest.TestCase): + def test_notebook_path_extracted(self): + # Fix #3: NotebookEdit uses notebook_path, not file_path. + path = get_tool_input_path({"tool_input": {"notebook_path": "nb.ipynb"}}) + self.assertEqual(path, "nb.ipynb") + + def test_file_path_still_wins(self): + path = get_tool_input_path( + {"tool_input": {"file_path": "a.py", "notebook_path": "b.ipynb"}} + ) + self.assertEqual(path, "a.py") + + +class TestEditLedgerHook(unittest.TestCase): + def setUp(self): + self.tmp = Path(tempfile.mkdtemp()) + (self.tmp / ".c3").mkdir() + + def tearDown(self): + shutil.rmtree(self.tmp, ignore_errors=True) + + def test_notebook_edit_is_logged(self): + # Fix #3: a notebook edit must produce a ledger entry. + (self.tmp / "nb.ipynb").write_text("{}", encoding="utf-8") + text = _run_ledger_hook( + self.tmp, "NotebookEdit", + {"notebook_path": str(self.tmp / "nb.ipynb")}, + ) + self.assertTrue(text.strip(), "NotebookEdit produced no ledger entry") + entry = json.loads(text.strip().splitlines()[-1]) + self.assertEqual(entry["file"], "nb.ipynb") + + def test_edit_id_has_random_suffix(self): + # Fix #4: ids look like edit___<4hex> and are unique even when + # two entries land in the same second. + (self.tmp / "foo.py").write_text("x = 1\n", encoding="utf-8") + ids = [] + for _ in range(3): + text = _run_ledger_hook( + self.tmp, "Edit", + {"file_path": str(self.tmp / "foo.py"), + "old_string": "x", "new_string": "y"}, + ) + ids.append(json.loads(text.strip().splitlines()[-1])["id"]) + for eid in ids: + parts = eid.split("_") + # edit, YYYYMMDD, HHMMSS, seq, hex4 + self.assertEqual(len(parts), 5, f"unexpected id shape: {eid}") + self.assertEqual(len(parts[-1]), 4, f"missing 4-hex suffix: {eid}") + int(parts[-1], 16) # suffix is valid hex + self.assertEqual(len(set(ids)), len(ids), "edit ids collided") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_edit_normalization.py b/tests/test_edit_normalization.py index d2e0eff..458240f 100644 --- a/tests/test_edit_normalization.py +++ b/tests/test_edit_normalization.py @@ -179,5 +179,126 @@ def test_batch_mode_with_fallback(self): "a = 'ONE'\nb = \"TWO\"\n") +class TestNewlinePreservation(unittest.TestCase): + """Regression: on Windows, read_text()+write_text() round-trips line + endings through os.linesep, rewriting an entire LF-only file to CRLF + after a one-line edit. The fix detects + preserves the original EOL.""" + + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.root = Path(self.tmp.name) + self.svc = _make_svc(self.root) + + def tearDown(self): + self.tmp.cleanup() + + def _write_bytes(self, rel: str, data: bytes) -> Path: + p = self.root / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_bytes(data) + return p + + def test_single_edit_preserves_lf(self): + self._write_bytes("lf.txt", b"a\nb\nc\n") + handle_edit( + "lf.txt", "b", "B", + summary="", tags="", replace_all=False, + svc=self.svc, finalize=_finalize, + ) + self.assertEqual((self.root / "lf.txt").read_bytes(), b"a\nB\nc\n") + + def test_single_edit_preserves_crlf(self): + self._write_bytes("crlf.txt", b"a\r\nb\r\nc\r\n") + handle_edit( + "crlf.txt", "b", "B", + summary="", tags="", replace_all=False, + svc=self.svc, finalize=_finalize, + ) + self.assertEqual((self.root / "crlf.txt").read_bytes(), b"a\r\nB\r\nc\r\n") + + def test_batch_edit_preserves_lf(self): + self._write_bytes("lf2.txt", b"a\nb\nc\n") + handle_edit( + "lf2.txt", "", "", + summary="", tags="", replace_all=False, + svc=self.svc, finalize=_finalize, + edits='[{"old_string":"a","new_string":"A"},' + '{"old_string":"c","new_string":"C"}]', + ) + self.assertEqual((self.root / "lf2.txt").read_bytes(), b"A\nb\nC\n") + + def test_batch_edit_preserves_crlf(self): + self._write_bytes("crlf2.txt", b"a\r\nb\r\nc\r\n") + handle_edit( + "crlf2.txt", "", "", + summary="", tags="", replace_all=False, + svc=self.svc, finalize=_finalize, + edits='[{"old_string":"a","new_string":"A"}]', + ) + self.assertEqual((self.root / "crlf2.txt").read_bytes(), b"A\r\nb\r\nc\r\n") + + +class TestBatchNoOp(unittest.TestCase): + """Regression: batch mode wrote the file + logged a ledger entry even + when zero patches applied (all NOT FOUND/AMBIGUOUS). Fix: only write + + log when at least one patch actually modified the content.""" + + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.root = Path(self.tmp.name) + self.svc = _make_svc(self.root) + # Real-ish ledger spy so we can assert it is NOT called. + self.svc.edit_ledger = MagicMock() + + def tearDown(self): + self.tmp.cleanup() + + def _write_bytes(self, rel: str, data: bytes) -> Path: + p = self.root / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_bytes(data) + return p + + def test_no_op_batch_does_not_rewrite_or_log(self): + self._write_bytes("x.txt", b"a\nb\nc\n") + before_mtime = (self.root / "x.txt").stat().st_mtime_ns + resp = handle_edit( + "x.txt", "", "", + summary="", tags="", replace_all=False, + svc=self.svc, finalize=_finalize, + edits='[{"old_string":"nope","new_string":"X"}]', + ) + self.assertIn("0/1 patches applied", resp) + # File untouched (bytes + mtime unchanged). + self.assertEqual((self.root / "x.txt").read_bytes(), b"a\nb\nc\n") + self.assertEqual((self.root / "x.txt").stat().st_mtime_ns, before_mtime) + # No ledger entry recorded for a no-op batch. + self.svc.edit_ledger.log_edit.assert_not_called() + + def test_partial_batch_writes_and_logs(self): + self._write_bytes("y.txt", b"a\nb\nc\n") + resp = handle_edit( + "y.txt", "", "", + summary="", tags="", replace_all=False, + svc=self.svc, finalize=_finalize, + edits='[{"old_string":"a","new_string":"A"},' + '{"old_string":"nope","new_string":"X"}]', + ) + self.assertIn("1/2 patches applied", resp) + self.assertEqual((self.root / "y.txt").read_bytes(), b"A\nb\nc\n") + + def test_non_dict_element_rejected(self): + self._write_bytes("z.txt", b"a\nb\nc\n") + resp = handle_edit( + "z.txt", "", "", + summary="", tags="", replace_all=False, + svc=self.svc, finalize=_finalize, + edits='["not-a-dict"]', + ) + self.assertIn("non-object element", resp) + # File untouched. + self.assertEqual((self.root / "z.txt").read_bytes(), b"a\nb\nc\n") + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_enforcement_flip.py b/tests/test_enforcement_flip.py index 3ed2304..4c06c44 100644 --- a/tests/test_enforcement_flip.py +++ b/tests/test_enforcement_flip.py @@ -11,17 +11,35 @@ import sys import tempfile import unittest +from datetime import datetime, timezone from pathlib import Path REPO_ROOT = Path(__file__).resolve().parents[1] -def _run_hook(tool_name: str, tool_input: dict) -> dict: - """Spawn the hook in a throwaway project and parse its stdout JSON.""" +def _run_hook(tool_name: str, tool_input: dict, signal_tool: str | None = None) -> dict: + """Spawn the hook in a throwaway project and parse its stdout JSON. + + If signal_tool is given, seed .c3/last_c3_call.json with a fresh signal + written by that c3 tool (mirrors hook_c3_signal.py output). + """ tmp = Path(tempfile.mkdtemp()) try: (tmp / ".c3").mkdir() (tmp / ".c3" / "activity_log.jsonl").write_text("", encoding="utf-8") + if signal_tool is not None: + read_unlock = signal_tool in { + "c3_search", "c3_compress", "c3_filter", + "c3_read", "c3_impact", "c3_validate", + } + (tmp / ".c3" / "last_c3_call.json").write_text( + json.dumps({ + "timestamp": datetime.now(timezone.utc).isoformat(), + "tool": signal_tool, + "read_unlocked": read_unlock, + }), + encoding="utf-8", + ) shutil.copy(REPO_ROOT / "cli" / "hook_pretool_enforce.py", tmp / "hook.py") shutil.copy(REPO_ROOT / "cli" / "_hook_utils.py", tmp / "_hook_utils.py") @@ -87,6 +105,53 @@ def test_unknown_tool_passes_through(self): r = _run_hook("SomeRandomTool", {}) self.assertEqual(r, {}, "Unenforced tools must produce no output") + # ── Fix #1: signal bypass regression ──────────────────────────────────── + def test_read_signal_does_not_unlock_native_edit(self): + """A read-class c3 signal (c3_status) must NOT unlock a native Edit.""" + r = _run_hook( + "Edit", + {"file_path": "foo.py", "old_string": "a", "new_string": "b"}, + signal_tool="c3_status", + ) + hso = r.get("hookSpecificOutput") or {} + self.assertEqual( + hso.get("permissionDecision"), "deny", + f"Edit must stay blocked despite a non-write c3 signal. Got {r}", + ) + + def test_read_signal_does_not_unlock_native_write(self): + r = _run_hook( + "Write", {"file_path": "foo.py", "content": "x"}, + signal_tool="c3_search", + ) + hso = r.get("hookSpecificOutput") or {} + self.assertEqual( + hso.get("permissionDecision"), "deny", + f"Write must stay blocked despite a read-class c3 signal. Got {r}", + ) + + def test_write_signal_unlocks_native_edit(self): + """A write-class c3 signal (c3_edit) unlocks a native Edit.""" + r = _run_hook( + "Edit", + {"file_path": "foo.py", "old_string": "a", "new_string": "b"}, + signal_tool="c3_edit", + ) + self.assertNotIn( + "hookSpecificOutput", r, + f"Edit must be allowed after a c3_edit signal. Got {r}", + ) + + def test_agent_signal_unlocks_native_multiedit(self): + r = _run_hook( + "MultiEdit", {"file_path": "foo.py", "edits": []}, + signal_tool="c3_agent", + ) + self.assertNotIn( + "hookSpecificOutput", r, + f"MultiEdit must be allowed after a c3_agent signal. Got {r}", + ) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_mcp_toml.py b/tests/test_mcp_toml.py index 7a2d096..8d2fab4 100644 --- a/tests/test_mcp_toml.py +++ b/tests/test_mcp_toml.py @@ -40,6 +40,19 @@ def test_ignores_non_mcp_sections(self): toml = '[other]\nx = 1\n[mcp_servers.a]\ncommand = "c"\n' self.assertEqual(set(parse_toml_mcp_servers(toml)), {"a"}) + def test_keeps_hash_inside_quoted_value(self): + # A "#" inside a quoted value (e.g. a Windows path "C:/tools/c#/run.exe") + # must NOT be treated as a comment and truncated. + toml = '[mcp_servers.c3]\ncommand = "C:/tools/c#/run.exe"\n' + self.assertEqual( + parse_toml_mcp_servers(toml)["c3"]["command"], "C:/tools/c#/run.exe" + ) + + def test_strips_real_trailing_comment(self): + # A genuine comment outside quotes is still stripped. + toml = '[mcp_servers.c3]\ncommand = "python" # the interpreter\n' + self.assertEqual(parse_toml_mcp_servers(toml)["c3"]["command"], "python") + class TestEscape(unittest.TestCase): def test_backslash_to_slash(self): @@ -73,6 +86,32 @@ def test_upsert_preserves_other_sections(self): self.assertIn("[keep]", text) self.assertIn("[mcp_servers.c3]", text) + def test_upsert_drops_orphaned_child_subtable(self): + # A child subtable like [mcp_servers.c3.env] under [mcp_servers.c3] must + # be removed along with its parent on re-upsert, not left orphaned + # beneath the freshly re-appended section (which would corrupt config). + f = self._tmp() + f.write_text( + "[mcp_servers.c3]\n" + 'command = "old"\n' + "[mcp_servers.c3.env]\n" + 'FOO = "bar"\n' + "[keep]\n" + "x = 1\n", + encoding="utf-8", + ) + upsert_toml_section(f, "mcp_servers.c3", {"command": "new"}) + text = f.read_text(encoding="utf-8") + # Exactly one c3 section header; no orphaned child table remains. + self.assertEqual(text.count("[mcp_servers.c3]"), 1) + self.assertNotIn("[mcp_servers.c3.env]", text) + # Unrelated user section is preserved. + self.assertIn("[keep]", text) + # Re-parse is clean: c3 has the new command and no stray "c3.env" server. + parsed = parse_toml_mcp_servers(text) + self.assertEqual(parsed["c3"]["command"], "new") + self.assertNotIn("c3.env", parsed) + def test_remove_returns_false_when_absent(self): self.assertFalse(remove_toml_section(self._tmp(), "mcp_servers.none")) diff --git a/tests/test_oracle_security_fixes.py b/tests/test_oracle_security_fixes.py new file mode 100644 index 0000000..b5149b0 --- /dev/null +++ b/tests/test_oracle_security_fixes.py @@ -0,0 +1,159 @@ +"""Security-fix tests for the Oracle server (PyPI release hardening). + +Covers: + #1 POST /api/config requires the Bearer token + rejects unknown keys. + #2 GET /api/apikey masks the raw token unless a valid Bearer token is given. + #3 project_path tool args are validated against discovered projects. + +Uses Flask's test client with a stub registry/executor and an in-memory config +so no Ollama/keyring/real config file is touched. The API key is supplied via +the ``C3_ORACLE_API_KEY`` env override (read by api_auth at verify time). +""" +from __future__ import annotations + +import os +import sys +import unittest +from pathlib import Path +from unittest import mock + +REPO_ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(REPO_ROOT)) + +# Deterministic key via env override — read by api_auth at verify time. +os.environ["C3_ORACLE_API_KEY"] = "test-secret-key" + +import oracle.oracle_server as srv # noqa: E402 +from oracle.services.tool_registry import TIER_ACTION, ToolRegistry # noqa: E402 + + +class _StubExecutor: + def execute(self, name, args=None): + return {"dispatched": name, "args": args or {}} + + +# ── #1 + #2: config / apikey HTTP gates ────────────────────────────── + + +class TestConfigAuthGate(unittest.TestCase): + """POST /api/config must require the Bearer token and allowlist keys.""" + + @classmethod + def setUpClass(cls): + # Re-assert the env override here (not just at module import): another + # oracle test module pops it in its tearDownClass, and class run order is + # not guaranteed, so set it idempotently per class. + os.environ["C3_ORACLE_API_KEY"] = "test-secret-key" + srv._cfg = { + "api_enabled": True, + "api_require_auth": True, + "api_max_tier": "action", + "bind_host": "127.0.0.1", + "mcp_port": 3332, + "mcp_enabled": True, + "ollama_base_url": "https://ollama.com", + "model": "gemma4:31b-cloud", + } + srv._tool_registry = ToolRegistry(_StubExecutor(), max_tier=TIER_ACTION) + srv._bridge = None # skip bridge re-verify branch + srv.app.config["TESTING"] = True + cls.client = srv.app.test_client() + cls.auth = {"Authorization": "Bearer test-secret-key"} + + def setUp(self): + # Never write the real config; serve a copy of the in-memory cfg. + self._saved = {} + self._lc = mock.patch.object(srv, "load_config", lambda: dict(srv._cfg)) + self._sc = mock.patch.object(srv, "save_config", self._saved.update) + self._lc.start() + self._sc.start() + self.addCleanup(self._lc.stop) + self.addCleanup(self._sc.stop) + + def test_post_config_unauthenticated_rejected(self): + # The original vuln: strip Discovery auth with no credentials. + r = self.client.post("/api/config", json={"api_require_auth": False}) + self.assertEqual(r.status_code, 401) + self.assertFalse(self._saved) # nothing persisted + + def test_post_config_bad_token_rejected(self): + r = self.client.post("/api/config", json={"model": "x"}, + headers={"Authorization": "Bearer nope"}) + self.assertEqual(r.status_code, 401) + + def test_post_config_unknown_key_rejected(self): + r = self.client.post("/api/config", json={"evil_key": 1, "model": "ok"}, + headers=self.auth) + self.assertEqual(r.status_code, 400) + self.assertIn("evil_key", r.get_json().get("keys", [])) + self.assertFalse(self._saved) # rejected wholesale, nothing persisted + + def test_post_config_known_key_accepted(self): + r = self.client.post("/api/config", json={"model": "new-model"}, + headers=self.auth) + self.assertEqual(r.status_code, 200) + self.assertTrue(r.get_json()["saved"]) + self.assertEqual(self._saved.get("model"), "new-model") + + +class TestApikeyMasking(unittest.TestCase): + """GET /api/apikey must not leak the raw token without a Bearer token.""" + + @classmethod + def setUpClass(cls): + os.environ["C3_ORACLE_API_KEY"] = "test-secret-key" # see note above + srv._cfg = { + "api_enabled": True, "api_require_auth": True, + "bind_host": "127.0.0.1", "mcp_port": 3332, "mcp_enabled": True, + } + srv.app.config["TESTING"] = True + cls.client = srv.app.test_client() + cls.auth = {"Authorization": "Bearer test-secret-key"} + + def test_get_apikey_unauthenticated_masks_key(self): + body = self.client.get("/api/apikey").get_json() + self.assertTrue(body["exists"]) + self.assertEqual(body["key"], "") # raw key never returned + self.assertTrue(body["masked"]) # masked form still available for status + snippet_auth = body["snippet"]["mcpServers"]["c3-oracle"]["headers"]["Authorization"] + self.assertEqual(snippet_auth, "Bearer ") # placeholder, not raw + + def test_get_apikey_authenticated_reveals_key(self): + body = self.client.get("/api/apikey", headers=self.auth).get_json() + self.assertEqual(body["key"], "test-secret-key") + snippet_auth = body["snippet"]["mcpServers"]["c3-oracle"]["headers"]["Authorization"] + self.assertEqual(snippet_auth, "Bearer test-secret-key") + + +# ── #3: project_path membership validation ─────────────────────────── + + +class _StubScanner: + def __init__(self, projects): + self._projects = projects + + def discover(self): + return list(self._projects) + + +class TestProjectPathValidation(unittest.TestCase): + def test_validate_rejects_unknown_path(self): + from oracle.services.c3_bridge import validate_project_path + scanner = _StubScanner([{"path": str(REPO_ROOT)}]) + with self.assertRaises(ValueError): + validate_project_path(scanner, str(REPO_ROOT.parent)) + + def test_validate_accepts_known_path(self): + from oracle.services.c3_bridge import validate_project_path + scanner = _StubScanner([{"path": str(REPO_ROOT)}]) + resolved = validate_project_path(scanner, str(REPO_ROOT)) + self.assertEqual(resolved, str(REPO_ROOT.resolve())) + + def test_validate_rejects_empty_path(self): + from oracle.services.c3_bridge import validate_project_path + with self.assertRaises(ValueError): + validate_project_path(_StubScanner([]), "") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_service_durability.py b/tests/test_service_durability.py new file mode 100644 index 0000000..cad5904 --- /dev/null +++ b/tests/test_service_durability.py @@ -0,0 +1,186 @@ +"""Durability/concurrency regressions for the .c3 service layer. + +Covers the server-side fixes (the hook side lives in test_edit_ledger_hook.py): + EditLedger: append-only tag_edit, locked log_edit append, collision- + resistant ids, tags_add patch merge, orphan-patch tolerance. + ConversationStore: atomic sessions.json write, no-wipe on corrupt load, + empty-list cache sentinel (no re-read). + FileMemoryStore: brace-block end ignores braces in strings/comments. + ContextSnapshot: _load_snapshot('latest') skips a corrupt newest file. +""" +import json +import sys +import tempfile +import unittest +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO_ROOT)) + +from services.context_snapshot import ContextSnapshot # noqa: E402 +from services.conversation_store import ConversationStore # noqa: E402 +from services.edit_ledger import EditLedger # noqa: E402 +from services.file_memory import FileMemoryStore # noqa: E402 + + +class TestEditLedger(unittest.TestCase): + def setUp(self): + self.tmp = Path(tempfile.mkdtemp()) + self.ledger = EditLedger(str(self.tmp)) + + def _lines(self): + return [ + json.loads(ln) + for ln in self.ledger.ledger_file.read_text(encoding="utf-8").splitlines() + if ln.strip() + ] + + def test_log_edit_id_has_random_suffix(self): + # Fix #3: ids carry a 4-hex suffix so server/hook can't collide. + e1 = self.ledger.log_edit("a.py", "edit", "one", include_git=False) + e2 = self.ledger.log_edit("a.py", "edit", "two", include_git=False) + for eid in (e1["id"], e2["id"]): + parts = eid.split("_") + self.assertEqual(len(parts), 5, f"unexpected id shape: {eid}") + self.assertEqual(len(parts[-1]), 4) + int(parts[-1], 16) + self.assertNotEqual(e1["id"], e2["id"]) + + def test_tag_edit_appends_patch_not_rewrite(self): + # Fix #1: tag_edit must APPEND a patch, never rewrite existing lines. + e = self.ledger.log_edit("a.py", "edit", "x", include_git=False) + before = self.ledger.ledger_file.read_text(encoding="utf-8") + ok = self.ledger.tag_edit(e["id"], "reviewed") + self.assertTrue(ok) + after = self.ledger.ledger_file.read_text(encoding="utf-8") + # Original content is a prefix of the new content (append-only). + self.assertTrue(after.startswith(before), "tag_edit rewrote the file") + lines = self._lines() + patch = lines[-1] + self.assertEqual(patch["target_id"], e["id"]) + self.assertEqual(patch["tags_add"], ["reviewed"]) + # And _load_merged applies the tag to the base entry. + merged = self.ledger.get_history(file="a.py") + self.assertIn("reviewed", merged[-1]["tags"]) + + def test_tag_edit_unknown_id_returns_false(self): + self.ledger.log_edit("a.py", "edit", "x", include_git=False) + self.assertFalse(self.ledger.tag_edit("edit_nope_000_zzzz", "t")) + + def test_load_merged_tolerates_orphan_patch(self): + # Fix #4: orphan patch (no base) must not crash _load_merged. + e = self.ledger.log_edit("a.py", "edit", "x", include_git=False) + with open(self.ledger.ledger_file, "a", encoding="utf-8") as f: + f.write(json.dumps({"target_id": "edit_ghost_000_aaaa", + "tags_add": ["t"]}) + "\n") + merged = self.ledger.get_history() + self.assertEqual(len(merged), 1) + self.assertEqual(merged[0]["id"], e["id"]) + + +class TestConversationStore(unittest.TestCase): + def setUp(self): + self.tmp = Path(tempfile.mkdtemp()) + + def _store(self): + return ConversationStore(str(self.tmp)) + + def test_add_turn_atomic_and_indexed(self): + store = self._store() + store.add_turn("s1", "user", "hello world", source="manual") + store.add_turn("s1", "assistant", "hi there", source="manual") + stats = store.get_stats() + self.assertEqual(stats["sessions"], 1) + self.assertEqual(stats["turns"], 2) + # sessions.json is valid JSON (atomic write left no partial file). + sf = self.tmp / ".c3" / "conversations" / "sessions.json" + json.loads(sf.read_text(encoding="utf-8")) + # No leftover temp files. + leftovers = list((self.tmp / ".c3" / "conversations").glob("sessions.json.tmp-*")) + self.assertEqual(leftovers, []) + + def test_corrupt_sessions_not_wiped(self): + # Fix #5: a corrupt sessions.json must be backed up, not silently + # reset to [] (which would let the next save wipe history). + store = self._store() + store.add_turn("s1", "user", "hello", source="manual") + sf = self.tmp / ".c3" / "conversations" / "sessions.json" + sf.write_text("{ this is not json", encoding="utf-8") + fresh = self._store() # forces a fresh load from the corrupt file + with self.assertRaises(Exception): + fresh._load_sessions() + backups = list((self.tmp / ".c3" / "conversations").glob("sessions.json.corrupt-*")) + self.assertTrue(backups, "corrupt sessions.json was not backed up") + + def test_empty_sessions_cached_no_reread(self): + # Fix #6: an empty (but loaded) catalog must not be re-read every call. + store = self._store() + self.assertEqual(store._load_sessions(), []) + self.assertIsNotNone(store._sessions) # sentinel flipped from None + # Simulate a writer creating a file AFTER first load; cache should hold. + sf = self.tmp / ".c3" / "conversations" / "sessions.json" + sf.write_text("[{\"session_id\": \"x\"}]", encoding="utf-8") + self.assertEqual(store._load_sessions(), []) # served from cache + + +class TestFileMemoryBraceBlock(unittest.TestCase): + def setUp(self): + self.tmp = Path(tempfile.mkdtemp()) + self.store = FileMemoryStore(str(self.tmp)) + + def test_brace_in_string_not_counted(self): + # Fix #9: a '}' inside a string before the real close must be ignored. + lines = [ + "function f() {", + ' log("}");', + " return 1;", + "}", + ] + end = self.store._find_brace_block_end(lines, 0) + self.assertEqual(end, 4) # the real closing brace is line 4 (1-indexed) + + def test_brace_in_line_comment_not_counted(self): + lines = [ + "function f() {", + " // closing } here is a comment", + " return 1;", + "}", + ] + self.assertEqual(self.store._find_brace_block_end(lines, 0), 4) + + def test_brace_in_block_comment_not_counted(self): + lines = [ + "function f() {", + " /* } still", + " open } */", + " return 1;", + "}", + ] + self.assertEqual(self.store._find_brace_block_end(lines, 0), 5) + + +class TestContextSnapshotCorrupt(unittest.TestCase): + def setUp(self): + self.tmp = Path(tempfile.mkdtemp()) + self.snap = ContextSnapshot(str(self.tmp)) + + def test_latest_skips_corrupt_newest(self): + # Fix #10: a corrupt newest snapshot must not break 'latest'. + d = self.snap.data_dir + (d / "snap_20260101_000000.json").write_text( + json.dumps({"snapshot_id": "20260101_000000"}), encoding="utf-8") + # A lexicographically-newer corrupt file. + (d / "snap_20260102_000000.json").write_text("{ broken", encoding="utf-8") + loaded = self.snap._load_snapshot("latest") + self.assertNotIn("error", loaded) + self.assertEqual(loaded["snapshot_id"], "20260101_000000") + + def test_corrupt_specific_returns_error(self): + d = self.snap.data_dir + (d / "snap_bad.json").write_text("{ broken", encoding="utf-8") + loaded = self.snap._load_snapshot("bad") + self.assertIn("error", loaded) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_web_security.py b/tests/test_web_security.py index 6e0f0aa..cf3d810 100644 --- a/tests/test_web_security.py +++ b/tests/test_web_security.py @@ -102,5 +102,36 @@ def test_preflight_from_evil_origin_gets_no_cors(self): self.assertIsNone(r.headers.get("Access-Control-Allow-Origin")) +class _FakeRequest: + """Minimal stand-in exposing only what check_request reads, so we can + simulate a fully absent Host header (Flask's test client always injects one).""" + + def __init__(self, method="POST", host="", headers=None): + self.method = method + self.host = host + self.headers = headers or {} + + +class TestMissingHostGuard(unittest.TestCase): + def setUp(self): + self.allowed = ws.allowed_hostnames() # loopback-only (hardened default) + + def test_missing_host_mutating_request_blocked(self): + ok, reason = ws.check_request(_FakeRequest(method="POST", host=""), self.allowed) + self.assertFalse(ok) + self.assertIn("Host", reason) + + def test_missing_host_get_still_allowed(self): + ok, _ = ws.check_request(_FakeRequest(method="GET", host=""), self.allowed) + self.assertTrue(ok) + + def test_missing_host_mutating_allowed_when_not_loopback_only(self): + # An intentional non-loopback deployment widens the allowlist; a missing + # Host there is not auto-rejected (the loopback-only hardening lifts). + allowed = ws.allowed_hostnames(bind_host="myhost.internal") + ok, _ = ws.check_request(_FakeRequest(method="POST", host=""), allowed) + self.assertTrue(ok) + + if __name__ == "__main__": unittest.main()