Skip to content

feat: supervise portable custom box services - #25

Merged
bmdavis419 merged 7 commits into
mainfrom
agent/arbitrary-services
Jul 23, 2026
Merged

feat: supervise portable custom box services#25
bmdavis419 merged 7 commits into
mainfrom
agent/arbitrary-services

Conversation

@bmdavis419

@bmdavis419 bmdavis419 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add portable executable service drop-ins under ~/.config/hermes-box/services.d/
  • supervise each foreground service independently with restart backoff, safe PID identity checks, and distinct redacted/rotated service-<name> logs
  • integrate custom services with hb up, reconciliation, pause/resume, status, doctor guidance, backup/import/upgrade persistence, and container shutdown
  • allow exact custom-service log queries through tx9 logs --source service-<name>
  • document the service contract and add focused lifecycle, concurrency, signal, reload, security, and logging regressions

Important behavior

  • definitions must be direct executable regular files with safe names and foreground processes
  • symlinks, unsafe names, directories, and non-executable files are ignored and reported
  • custom-service failures do not block steady-state Executor/Hermes reconciliation
  • interactive hb up / hb resume still return failure when a configured service cannot be reconciled
  • hb pause synchronously stops services before backup state is checkpointed

Validation

  • make syntax
  • make lint
  • make test
  • go vet -buildvcs=false ./...
  • go test -buildvcs=false ./...
  • git diff --check

No explicit build command was run, per repository instructions.

Closes #24


Open in Devin Review

Note

Add supervision of portable custom box services via tx9-services

  • Introduces guest/tx9-services, a new OS-level supervisor for user-provided service drop-ins defined as executables in services.d/. Services are restarted automatically, logged under service-<name> sources, and managed with durable state and safe process identity checks.
  • hb up and hb resume now reconcile custom services before starting Executor and the gateway, failing if reconciliation fails. hb pause stops services, and hb reconcile stops services while quiesced but continues core reconciliation on service failure.
  • Adds hb services and hb services-reload commands that delegate to the tx9-services helper.
  • tx9-logs capture gains a parent-death SIGKILL shim (via re-exec) and precise startup failure reporting (process_start_failed events for exec/setup errors). Log source filtering now accepts service-<name> patterns.
  • Provisioning installs tx9-services at $OPT/bin/tx9-services with mode 0755, and the build context marks it executable.
  • Risk: hb up/hb resume will fail if service reconciliation fails, which is a new failure mode for those commands.

Macroscope summarized 3e7089b.

Greptile Summary

This PR adds supervision for portable custom services inside a box. The main changes are:

  • Executable service drop-ins with independent restart and process tracking.
  • Service lifecycle integration with hb, reconciliation, and container shutdown.
  • Redacted service logs with exact service-<name> filtering.
  • Provisioning, documentation, and focused lifecycle tests.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the updated code.
  • The lifecycle and process-safety fixes include focused tests for startup failure, shutdown, stale process identity, and orphan prevention.

Important Files Changed

Filename Overview
guest/tx9-services Adds service discovery, reconciliation, process identity checks, restart supervision, stopping, and status output.
guest/tx9-logs Adds parent-death handling, startup failure reporting, and custom-service log sources.
guest/hb Integrates custom services with startup, pause, resume, reconciliation, status, and doctor commands.
guest/hb-workload Stops custom services during quiescence and workload termination.
provision/provision.sh Installs the new service supervisor as an executable guest tool.

Reviews (6): Last reviewed commit: "fix: make service log regression asserti..." | Re-trigger Greptile

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Changes

Adds portable custom service drop-ins with supervised reconciliation, restart and quiesce behavior, hb services commands, service-specific log sources, packaging updates, documentation, and regression coverage.

Portable custom services

Layer / File(s) Summary
Service supervisor and lifecycle state
guest/tx9-services, tests/regressions-services.sh
Supervises executable service definitions, tracks process identity and fingerprints, handles restarts and stopping, reports status, and validates lifecycle behavior.
Guest lifecycle integration
guest/hb, guest/hb-workload, tests/regressions-hb-workload.sh
Integrates custom services with startup, pause, resume, reconciliation, shutdown, status, doctor, and signal handling.
Service log sources and contracts
guest/tx9-logs, internal/cli/*, README.md, docs/tx9-cli-design.md
Supports service-<name> log sources and documents service definitions, supervision, lifecycle behavior, and log access.
Distribution and validation wiring
internal/assets/*, provision/provision.sh, tests/static.sh, Makefile, docker/entrypoint.sh
Packages tx9-services as an executable, extends static and asset checks, includes regression scripts in shell validation, and updates an entrypoint comment.

Possibly related PRs

  • davis7dotsh/tx9#13: Both changes extend the CLI dispatch and logs command surface.
  • davis7dotsh/tx9#21: Both changes modify Hermes lifecycle reconciliation in guest/hb and guest/hb-workload.
  • davis7dotsh/tx9#22: Both changes extend the tx9-logs capture and source-query pipeline.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR implements support for arbitrary services starting under Hermes, which directly addresses issue #24.
Out of Scope Changes check ✅ Passed The changes stay focused on portable custom service supervision, related CLI/logging updates, packaging, and tests.
Title check ✅ Passed The title clearly summarizes the main change: supervising portable custom services.
Description check ✅ Passed The description matches the changeset and accurately describes the new service supervision and logging behavior.

Comment @coderabbitai help to get the list of available commands.

macroscopeapp[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

macroscopeapp[bot]

This comment was marked as resolved.

macroscopeapp[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

macroscopeapp[bot]

This comment was marked as resolved.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (4)
tests/regressions-logs.sh (2)

300-311: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Inline zombie-aware polling loop duplicates wait_for_process_exit.

This loop reimplements the same "kill -0 / zombie-state" polling logic just added as wait_for_process_exit() in tests/regressions-services.sh. See consolidated comment.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/regressions-logs.sh` around lines 300 - 311, Replace the duplicated
kill-0 and zombie-state polling loop with the existing wait_for_process_exit()
helper from regressions-services.sh, passing pdeath_child_pid and preserving the
subsequent failure check and error message.

291-293: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Track pdeath_child_pid for harness cleanup.

pdeath_capture_pid is added to pids (line 290) but pdeath_child_pid — the actual foreground process under test — is not. If the survival check at lines 307-311 trips (i.e., the fix being validated is broken), the test exits without this process being reaped. See consolidated comment.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/regressions-logs.sh` around lines 291 - 293, Update the PID setup
around pdeath_capture_pid and pdeath_child_pid so the child PID read from
pdeath_pid_file is also appended to the pids cleanup list. Preserve the existing
kill -0 survival check and ensure the foreground process is tracked before the
test can exit on failure.
tests/regressions-services.sh (2)

294-345: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Track old_pdeath_service/new_pdeath_service PIDs for cleanup.

old_pdeath_capture/new_pdeath_capture are added to pids (lines 325, 334) but old_pdeath_service and new_pdeath_service — the actual guarded script processes — never are. If an assertion between lines 335-339 fails (i.e., exactly the scenario this test guards against: the new pair not replacing the old one), the test exits without these processes being reaped by the harness's cleanup trap, leaking a python3 -c 'signal.pause()' process. See consolidated comment.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/regressions-services.sh` around lines 294 - 345, Add both
old_pdeath_service and new_pdeath_service to the pids cleanup list immediately
after each PID is captured, alongside the corresponding capture PID. Keep the
existing assertions and process lifecycle checks unchanged so the guarded
service processes are reaped if any later assertion fails.

47-57: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate polling logic vs. regressions-logs.sh.

This helper duplicates the zombie-aware exit-polling loop added inline in tests/regressions-logs.sh (lines 300-311) for the same class of "did the process die" check. Worth sharing via tests/lib.sh — see consolidated comment.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/regressions-services.sh` around lines 47 - 57, Move the shared
zombie-aware process-exit polling logic from wait_for_process_exit in
regressions-services.sh and the inline loop in regressions-logs.sh into a common
helper in tests/lib.sh. Update both callers to reuse that helper while
preserving the existing timeout, polling, and return-status behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/regressions-logs.sh`:
- Around line 300-311: Replace the duplicated kill-0 and zombie-state polling
loop with the existing wait_for_process_exit() helper from
regressions-services.sh, passing pdeath_child_pid and preserving the subsequent
failure check and error message.
- Around line 291-293: Update the PID setup around pdeath_capture_pid and
pdeath_child_pid so the child PID read from pdeath_pid_file is also appended to
the pids cleanup list. Preserve the existing kill -0 survival check and ensure
the foreground process is tracked before the test can exit on failure.

In `@tests/regressions-services.sh`:
- Around line 294-345: Add both old_pdeath_service and new_pdeath_service to the
pids cleanup list immediately after each PID is captured, alongside the
corresponding capture PID. Keep the existing assertions and process lifecycle
checks unchanged so the guarded service processes are reaped if any later
assertion fails.
- Around line 47-57: Move the shared zombie-aware process-exit polling logic
from wait_for_process_exit in regressions-services.sh and the inline loop in
regressions-logs.sh into a common helper in tests/lib.sh. Update both callers to
reuse that helper while preserving the existing timeout, polling, and
return-status behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 83b14d1e-6b8e-4073-b045-7a53f75843dd

📥 Commits

Reviewing files that changed from the base of the PR and between bdec5f5 and c53b208.

📒 Files selected for processing (3)
  • guest/tx9-logs
  • tests/regressions-logs.sh
  • tests/regressions-services.sh

bmdavis419 and others added 2 commits July 22, 2026 14:13
The all-source query asserted an early healthy-log event survived an
unfiltered --tail 100 while the intentionally broken service kept
crash-looping and appending records, so the check raced the flood.
Filter with --grep before the tail so eviction of unrelated events
cannot fail the assertion.

Both service log assertions also relied on jq -e select() over a
multi-record JSONL stream, whose exit status tracks the final record
rather than whether any record matched. Slurp and assert any() instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bmdavis419
bmdavis419 merged commit a6e4f07 into main Jul 23, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ablity for arbitrary services to be started

1 participant