Phase 5 — Background-engine hardening (rides 0.19.0)

[rennf93]

Phase 5 — Background-engine hardening (rides 0.19.0)

Fifth phase of the 2026-07-06 full-codebase-scan fix program. Twenty verified background-engine findings + an e2e smoke module, all under the existing ## [0.19.0] CHANGELOG section. Stacks on Phase 4 (fix/scan-git-workflow, PR #318) — base is fix/scan-git-workflow; merge #318 first.

Security

• H24 (1bb82827) — release_executor.wait_for_ci keeps polling through the window on a non-success conclusion; only conclusion == "success" returns True and loop exhaustion returns False. A failed first attempt while a re-run was still in_progress no longer blocks a real release — a re-run that flips the same commit green still publishes.
• H25 (96214624, bc19cbb5) — sweep_orphan_release_locks runs at Orchestrator.start and deletes any roboco:release_proposal:* Redis key whose UUID owner isn't in the in-flight approve registry. A restart mid-execute orphaned the mutex (TTL 3000s, no heartbeat) and a CEO retry got already_in_progress for up to 50 min; after a restart the registry is empty so every surviving key is an orphan. Best-effort (Redis outage logs a warning, never crashes). bc19cbb5 is a controller cleanup that dropped a # type: ignore the per-task reviewer had waved through (uses cast("asyncio.Task[None]", ...) under TYPE_CHECKING).

Fixed

• M1 (59525b20) — tiktok_client._refresh commits the rotated tokens in an independent session before returning; a caller lock-loss rollback no longer discards them. TikTok already invalidated the old refresh_token, so the pre-fix discard was a permanent credential lockout.
• M2 (18ec8a5a) — x_engine.open_feature_spotlight_exploration re-arms past a stale exploration: past 2*interval with no live HoM spawn, the stale one is cancelled and a fresh one originates (an exploration left PENDING with no live spawn no longer gates the engine silent).
• M3 + M5 (41494d4a) — x_post_service.reject() + video_post_service.reject() raise on an already-COMPLETED draft (no CANCELLED-on-a-live-post lie); X edited_body write deferred into the single-flight lock — the CEO's edit is applied to the re-read row after the in-lock COMPLETED check, so a concurrent approve that already posted can't have the edit land on the just-posted task.
• M4 (296d224e) — list_completed_video_tasks bounded .order_by(created_at.desc()).limit(video_render_scan_limit) (default 200) backed by ix_tasks_source_status_created (migration 066), instead of re-reading every completed video task ever each cycle.
• M6 + M7 (b30ec223) — X _mark_seen runs after the meaningful + project checks (a low-engagement mention is no longer permanently marked seen; one that later goes viral is still draftable); X mentions persist the since_id cursor in Redis so a burst >50 between ticks isn't silently dropped.
• M8 + M9 + L9 (cc2ef3ac) — release gate scoped to the read-clone HEAD (head_sha passed to get_latest_ci_conclusion so a stale older-commit CI run can't read "green" while HEAD's CI is in_progress; a missing HEAD falls to unknown → no proposal); _run_git 30s timeout bubbles a clear error instead of hanging the thread; _commits_since splits with maxsplit=2 so a \x1f embedded in a commit subject/body no longer garbles the CHANGELOG line.
• M10 + L35 (557e432b) — dep_update dedupe by (git_url, dep_update_command): a monorepo with two distinct commands gets one open task per command, not one for the repo blocking the second ecosystem's drift. run_cycle fetches open tasks once and _eligible checks membership in-memory (TaskTable.project is lazy="joined", so the project row rides along in one SELECT — no N+1).
• M11 (f9fd1676, 460e8798) — engine-loop liveness watchdog: each of the eight background engine loops (self_heal, ci_watch, dep_update, release_manager, x_feature_spotlight, video_render, x_mentions, roadmap_engine) records a monotonic heartbeat at start + after a successful cycle; _check_loop_liveness (called from _check_health) logs a warning when now - last_success > 2 * interval. A silently-died cycle task stops recording → after 2*interval the health loop logs "engine loop stalled". 460e8798 is a controller scope-widening commit: the brief listed six loops; the mentions-poll + roadmap-engine loops are also engine loops that could stall silently, so the watchdog now covers all eight.
• M21 (e2c8be48) — _run_video_render_cycle commits after each _render_video_task instead of one trailing commit. A raise mid-cycle no longer rolls back prior renders (each is durable before the next is attempted), and the next cycle no longer re-renders + re-originates a second held video_post draft — the committed render_status="rendered" is the idempotency key the next scan skips.
• M22 (ed3401a8) — _detect_stuck_tasks skips any task whose source is CEO-held (release_manager, x_post/x_reply, video_post, PR-review, self-heal pre-approve) before the age/issues check, reusing the existing _is_held_ceo_source helper. A held draft sitting PENDING is no longer auto-blocked, wedging the held-artifact flow.
• L6 (d949b31a) — video_renderer_client._save writes to {path}.mp4.tmp then Path.replace(path) (atomic POSIX rename(2)). A re-render can no longer clobber the MP4 the panel is streaming mid-read — a reader sees either the old or the new file, never a half-written one.
• L11 (035e05ba) — self_heal._fingerprint documented as by-design stable per-signal (NOT per-run): the dedup's intent is one open fix task per regression signal, so a persistently-red CI signal doesn't open a second task every cycle. A documenting comment records the acceptance so a future reader/agent doesn't "fix" it into a per-run hash and reintroduce the dedupe collision. (Controller amended the commit subject — the implementer's original "current-state by design" was inverted; the code is a stable content hash, explicitly NOT current-state.)
• L34 (96214624) — the release-approve heartbeat shares one redis.from_url client per approve (~40 per-tick pools collapse to one), opened in approve() and closed in its finally.
• L36 (082fbcf3) — MultiProjectCITelemetrySource.fetch gathers per-project samples concurrently with asyncio.gather so one slow GitHub 429 doesn't stall the whole sweep; Nones filtered.

e2e smoke

• tests/e2e_smoke/test_background_engines.py (6a078478) — 4 cross-layer scenarios: H24 (wait_for_ci polls through a non-success to a later success), H25 (orphan release-mutex sweep against real Redis db 15 — orphan deleted, in-flight preserved), M1 (TikTok rotated refresh-token survives a caller lock-loss rollback via the independent-session commit), M21 (_run_video_render_cycle commits per-task — A's render + video_post draft durable despite B's raise). M11 skipped (a non-flaky liveness harness needs a controllable clock + running loop tick the in-process stack can't model; unit-covered by tests/unit/runtime/test_orchestrator_liveness.py). Each stub drives the real service method + real DB/Redis, mocking only the external legs the harness can't model (GitHub CI, TikTok egress, video-renderer sidecar). Deviations documented in the module docstring.

Migration

• 066_tasks_source_status_idx (296d224e) — ix_tasks_source_status_created on tasks(source, status, created_at), backing the M4 bounded render-loop scan. down_revision="065_rag_index_failures". No ENUM. Revision id 27 chars (fits the repo's alembic_version VARCHAR(32)). Round-trip verified (upgrade head → downgrade -1 → upgrade head).

Verification

• uv run ruff format + ruff check — clean on all Phase 5 changed files.
• uv run mypy roboco/ — clean on changed files.
• Unit suites green: test_release_executor.py (H24), test_release_proposal_orphan_sweep.py (H25+L34, 6 passed), test_tiktok_client.py (M1), test_x_engine.py (M2+M6+M7), test_x_post_service.py + test_video_post_service.py (M3+M5), test_video_engine.py (M4), test_release_readiness.py + test_release_manager_engine.py (M8+M9+L9), test_dep_update_engine.py (M10+L35), test_multi_ci_telemetry.py (L36, 4 passed), test_self_heal_engine.py (L11, 10 passed), test_orchestrator_liveness.py (M11, 9 passed — 8 loops + watchdog), test_video_render_loop.py (M21, 9 passed), test_orchestrator_stuck_tasks.py (M22, 2 passed), test_video_renderer_client.py (L6, 10 passed).
• make e2e-smoke — every Phase 5 scenario passes in isolation (4 passed). Full-suite session-scoped workspace contamination across e2e files is a pre-existing harness limitation, documented + out of scope.
• Per-task reviews: Tasks 1-16 each Approved, spec ✅ (sonnet reviewers). Whole-branch opus review: Clean — ship it (all 20 findings correct at the root-cause site; zero constraint violations; migration 066 + CHANGELOG + e2e + test hygiene PASS; all cross-task interactions compose). Two Minor test-strength notes fixed in 64d81fe2: corrected the e2e docstring's M11 unit-test filename (test_loop_liveness_watchdog.py → test_orchestrator_liveness.py) + strengthened the M11 failed-cycle test to spy on _record_loop_heartbeat and assert it fires exactly once (start heartbeat only, post-success skipped on a raised cycle). No production code changed in the fix wave.

Model note

Implementers ran on sonnet; per-task reviews on sonnet; whole-branch review on opus. kimi-k2.7-code:cloud is not routable in this harness (flagged to CEO; accepted).

🤖 Generated with Claude Code

[github-actions]

Thanks for opening your first pull request on RoboCo!

Quick checklist before review (most of these are enforced by CI, but worth a glance):

• make quality — ruff format check, ruff check, mypy, pytest (≥80% coverage), and the rest of the gate
• Panel changes pass pnpm lint and pnpm exec tsc --noEmit (run from panel/)
• No # noqa / # type: ignore shortcuts; pre-existing violations in touched files are fixed
• Added an entry under ## [Unreleased] in CHANGELOG.md
• Signed the CLA (the bot will prompt you on this PR)
• Signed your commits — master requires verified signatures (SSH signing setup)
• Updated any affected docs under docs/

See CONTRIBUTING.md for the full workflow and the Code of Conduct for the community standards we follow.

Welcome aboard — a maintainer will review shortly.