Skip to content

test: isolate user state across platforms - #1215

Open
steipete wants to merge 2 commits into
mainfrom
codex/test-isolation
Open

test: isolate user state across platforms#1215
steipete wants to merge 2 commits into
mainfrom
codex/test-isolation

Conversation

@steipete

@steipete steipete commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add one shared testutil.IsolateUserDirs helper that redirects state, config, home, AppData, and local AppData into a private per-test tree on every supported OS
  • migrate the affected CLI and provider tests, with package-wide safety nets for CLI, DigitalOcean, and Hyper-V tests that could otherwise reach persistent user state
  • extend the existing Windows CI job to run the isolation helper and affected provider suites natively

The helper asserts that os.UserConfigDir() remains inside the test root and creates every synthetic user directory with private permissions. Package-wide isolation preserves the original Go build and module cache locations so nested CLI builds do not redownload dependencies.

Thanks to @paulcam206 for the detailed reports and cross-platform reproduction work.

Reproduction and proof

On unmodified main, running one reported CLI test with a controlled fallback home passed but left this persistent lock behind:

$ env -u XDG_STATE_HOME -u XDG_CONFIG_HOME HOME="$repro_root" go test -count=1 -run '^TestAutoRouteStaticLeaseIgnoresNonStaticIDs$' -v ./internal/cli
=== RUN   TestAutoRouteStaticLeaseIgnoresNonStaticIDs
--- PASS: TestAutoRouteStaticLeaseIgnoresNonStaticIDs (0.00s)
PASS
$ find "$repro_root" -path '*/crabbox/*'
/tmp/crabbox-state-repro.Z2zIbT/Library/Application Support/crabbox/state/claim-locks/cbx_abc123.json.lock

The same controlled-fallback check on this branch leaves no Crabbox path behind:

$ env -u XDG_STATE_HOME -u XDG_CONFIG_HOME HOME="$repro_root" go test -count=1 -run '^TestAutoRouteStaticLeaseIgnoresNonStaticIDs$' -v ./internal/cli
=== RUN   TestAutoRouteStaticLeaseIgnoresNonStaticIDs
--- PASS: TestAutoRouteStaticLeaseIgnoresNonStaticIDs (0.00s)
PASS
ok  github.com/openclaw/crabbox/internal/cli  0.474s
$ find "$repro_root" -path '*/crabbox/*'
$ echo "PASS: no Crabbox files were written beneath the fallback home"
PASS: no Crabbox files were written beneath the fallback home

Verification

$ gofmt -w $(git ls-files '*.go')
$ go vet ./...
$ go test -race -p 1 ./...
ok  github.com/openclaw/crabbox/internal/cli  210.560s
ok  github.com/openclaw/crabbox/internal/providers/digitalocean  2.757s
ok  github.com/openclaw/crabbox/internal/providers/hyperv  1.716s
ok  github.com/openclaw/crabbox/internal/testutil  (cached)

The full local parallel race command was also attempted, but this machine had only 2.9 GiB free and parallel linking stopped with no space left on device; the identical suite passed with -p 1. The hosted Go job passed the exact go test -race ./... command at https://github.com/openclaw/crabbox/actions/runs/30673261639/job/91295228438.

Native Windows execution passed in the expanded existing Windows CI job at https://github.com/openclaw/crabbox/actions/runs/30673261639/job/91295228445. That job exercised the shared helper plus AWS, Daytona, DigitalOcean, Hyper-V, Namespace, Scaleway, Vast, and Vultr before rerunning the existing Pond Mesh cancellation proof. Locally, the Windows implementation was checked against Go's documented os.UserConfigDir behavior: %AppData% is redirected via APPDATA, while USERPROFILE and LOCALAPPDATA cover the other standard-library user-directory resolvers.

Closes #1212
Closes #1213

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P1 Urgent regression or broken agent/channel workflow affecting real users now. labels Jul 31, 2026
@clawsweeper

clawsweeper Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 31, 2026, 7:54 PM ET / 23:54 UTC.

ClawSweeper review

What this changes

The branch adds a shared Go test helper that redirects user state and config directories into private temporary paths, applies it to affected CLI/provider tests, and expands native Windows CI coverage.

Merge readiness

⚠️ Ready for maintainer review - 1 item remains

Keep this PR open for maintainer merge review. Current main still resolves test state through XDG_STATE_HOME or os.UserConfigDir, while this branch installs cross-platform test isolation for those paths and supplies before/after terminal proof plus successful native Windows CI evidence; no patch defect was found.

Priority: P1
Reviewed head: 80c870efb573f2b411a1c2fe2f6ce0d1e4d30a9c

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) Strong real-world proof and a focused, source-aligned patch make this a normal mergeable-quality reliability fix.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (terminal): The PR body shows a controlled before/after terminal reproduction with no Crabbox path left under the fallback home and links a successful native Windows CI run for the expanded suite.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): The PR body shows a controlled before/after terminal reproduction with no Crabbox path left under the fallback home and links a successful native Windows CI run for the expanded suite.
Evidence reviewed 5 items Current-main exposure: Current main uses XDG_STATE_HOME when set, otherwise os.UserConfigDir, for claim state; config lookup also uses os.UserConfigDir. This is the platform-dependent path the PR isolates.
Shared repair boundary: The branch's helper sets HOME, USERPROFILE, XDG config/state, APPDATA, and LOCALAPPDATA under a temporary root, then guards that os.UserConfigDir stays within that root.
Focused Windows coverage: The workflow adds a native Windows command covering the helper and the AWS, Daytona, DigitalOcean, Hyper-V, Namespace, Scaleway, Vast, and Vultr suites, while retaining the existing Pond Mesh cancellation check.
Findings None None.
Security None None.

How this fits together

Crabbox Go tests exercise CLI and provider code that reads user configuration, state, and platform home directories. The new test utility redirects those inputs before tests create claims, credentials, or provider artifacts, so test execution no longer reaches persistent developer state.

flowchart LR
  A[Go CLI and provider tests] --> B[User-directory resolver]
  B --> C[Shared test isolation helper]
  C --> D[Private temporary home and state]
  D --> E[Claim and config test files]
  C --> F[Package-wide test setup]
  F --> G[Native Windows CI]
  E --> H[Hermetic test results]
Loading

Before merge

  • Complete next step (P2) - No mechanical repair is needed; a maintainer should review and merge the already-proven test-isolation change.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Changed test surface 25 files affected; 2 test-utility files added The implementation centralizes platform isolation while migrating only tests that reach user state or config.
Windows coverage 1 existing Windows job expanded The job now exercises the helper and eight affected provider packages natively before its existing cancellation proof.

Technical review

Best possible solution:

Merge the shared isolation helper and its targeted test/Windows-CI coverage so all affected packages keep user-state artifacts inside test-owned temporary directories.

Do we have a high-confidence way to reproduce the issue?

Yes—source inspection confirms current main falls back to OS user configuration directories, and the PR body provides a controlled before/after CLI reproduction plus native Windows CI evidence.

Is this the best way to solve the issue?

Yes—the shared helper covers the actual resolver inputs across supported platforms and package-wide setup protects tests that do not install per-test isolation themselves.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f793138a161e.

Labels

Label justifications:

  • P1: The defect can make normal developer test runs mutate persistent user state and produce cross-run failures.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body shows a controlled before/after terminal reproduction with no Crabbox path left under the fallback home and links a successful native Windows CI run for the expanded suite.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body shows a controlled before/after terminal reproduction with no Crabbox path left under the fallback home and links a successful native Windows CI run for the expanded suite.

Evidence

What I checked:

  • Current-main exposure: Current main uses XDG_STATE_HOME when set, otherwise os.UserConfigDir, for claim state; config lookup also uses os.UserConfigDir. This is the platform-dependent path the PR isolates. (internal/cli/claim.go:1756, f793138a161e)
  • Shared repair boundary: The branch's helper sets HOME, USERPROFILE, XDG config/state, APPDATA, and LOCALAPPDATA under a temporary root, then guards that os.UserConfigDir stays within that root. (internal/testutil/userdirs.go:22, 80c870efb573)
  • Focused Windows coverage: The workflow adds a native Windows command covering the helper and the AWS, Daytona, DigitalOcean, Hyper-V, Namespace, Scaleway, Vast, and Vultr suites, while retaining the existing Pond Mesh cancellation check. (.github/workflows/ci.yml:69, 80c870efb573)
  • Current-main provenance: Blame traces the current DigitalOcean helper, Hyper-V test setup, Windows workflow, and user-directory implementations to the current feature-history commit by Peter Steinberger; the branch is not contained by the checked-out main branch. (internal/providers/digitalocean/backend_test.go:185, 4775e3bbc61d)
  • Patch hygiene: The 25-file patch has no whitespace errors, and the supplied PR evidence reports a successful full race suite and successful expanded native Windows job. (.github/workflows/ci.yml:84, 80c870efb573)

Likely related people:

  • steipete: Current blame for the state/config paths, provider test helpers, and Windows workflow traces to Peter Steinberger, who also authored the two isolation commits. (role: current area contributor and repair author; confidence: high; commits: 4775e3bbc61d, 3e1c4828eacd, 80c870efb573; files: internal/cli/claim.go, internal/cli/config.go, internal/providers/digitalocean/backend_test.go)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (1 earlier review cycle)
  • reviewed 2026-07-31T23:38:47.498Z sha 80c870e :: needs maintainer review before merge. :: none

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P1 Urgent regression or broken agent/channel workflow affecting real users now. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

1 participant