[RAPTOR-18983][RAPTOR-18974] refactor(workload): move state under .datarobot and port up internals

[wojtekwdr]

RATIONALE

A project that uses the workload commands ends up with two DataRobot entries at its root: the .datarobot.yaml manifest and a separate .wapi/ state directory. Two sibling dot-entries for one product reads like two tools, and it is the first thing a user sees when they open the project. All CLI-owned state moves under .datarobot/, which this CLI already uses for .datarobot/cli/state.yaml.

The same change also ports the internal/workload primitives that dr workload up will build on. No command surface changes there, so nothing user-visible ships from that half.

Closes RAPTOR-18983 and RAPTOR-18974.

CHANGES

State directory

• .wapi/ becomes .datarobot/wapi/, holding the same config.json, manifest.json, history.log, .checkouts/ and self-ignoring .gitignore.
• wapi.Dir() is the single resolver. It falls back to a legacy root .wapi/ when one is present, so a project linked by an older CLI stays readable. The fallback is a stat rather than cached state, so a migration that cannot complete degrades to reading the old location instead of failing the command.
• wapi.EnsureMigrated() relocates a legacy directory on the next state-touching command. It is silent and cannot fail; both outcomes are logged at debug level. --dry-run and --diff skip it, so a preview never moves anything on disk.
• systemExcludes gains .datarobot/wapi, not a bare .datarobot. Entries match as prefixes, so a bare one would also stop syncing .datarobot/cli/state.yaml. The legacy .wapi stays listed so an un-migrated project does not upload its own state.
• Hand-built paths in sync/rollback.go and sync/synclock.go now go through the package, so there is one definition of where state lives.
• Stale rollback recovery sweeps both locations, so backups from a sync that crashed before the directory moved are still applied.
• Help text and error strings across cmd/artifact/code and cmd/artifact/build follow. Read errors name the real file via wapi.ConfigPath() instead of a hard-coded path.
• .wapiignore stays at the project root: it is authored and committed by the user, the same convention as .gitignore.

Workload internals

• WaitForWorkload with IsTerminalWorkloadStatus and IsWorkloadErrorStatus.
• New internal/workload/execenv.go: ResolveExecutionEnvironment maps a name or id to both execution-environment ids for generated-mode builds.
• Artifact spec primitives: Container.Name, Container.EnvironmentVars, the EnvironmentVar type including its credential-backed source shape, and the PrimaryContainerName and PrimaryEnvironmentVars helpers.
• PatchPrimaryContainer is deliberately not ported: artifacts are immutable now, every change mints a new version, and that removed every caller.

Along the way

• fsutil.DirExists and FileExists handled only ErrNotExist, leaving info nil on any other stat error and dereferencing it. A project root containing a file named .datarobot produced a stack trace; both now return false for every stat error.
• The primary-container traversal existed in four copies in artifact.go and had drifted: PrimaryContainerName required a non-empty name, so an unnamed primary made it report a sidecar's name while the other three read the primary. All four now share one primaryContainer() helper.

PR Automation

Comment-Commands: Trigger CI by commenting on the PR:

• /trigger-smoke-test or /trigger-test-smoke - Run smoke tests
• /trigger-install-test or /trigger-test-install - Run installation tests

Labels: Apply labels to trigger workflows:

• run-smoke-tests or go - Run smoke tests on demand (only works for non-forked PRs)

Important

For Forked PRs: The run-smoke-tests label won't work. A required Smoke Tests check will block merge until a maintainer acts:

• A maintainer uses /approve-smoke-tests to run smoke tests (results will set the check)
• A maintainer uses /skip-smoke-tests to bypass the check without running tests

Please comment requesting a maintainer review if you need smoke tests to run.

[datarobot-pr-review-router]

🎫 Jira:

• RAPTOR-18983 — 2b. dr workload state: keep .wapi inside .datarobot instead of a second root directory
• RAPTOR-18974 — 3. dr workload internals: execution-environment resolution, workload wait, artifact container primitives

[c-h-russell-walker]

FYI we already use .datarobot in Codespaces so you may want to ensure this works well with that.

Please see:
https://github.com/datarobot/notebooks/tree/main#usage-of-datarobot-in-codespaces

[Copilot]

Pull request overview

This PR consolidates workload/artifact code-sync state under a single project root directory (.datarobot/) by moving the legacy .wapi/ state into .datarobot/wapi/ with a backwards-compatible resolver + silent migration. It also ports/extends internal workload primitives needed for dr workload up, and includes a small fsutil correctness fix and test hardening.

Changes:

• Move sync state from .wapi/ to .datarobot/wapi/ with legacy fallback resolution and an on-demand migration (EnsureMigrated), updating sync/rollback/locking and user-facing strings accordingly.
• Add/port workload internals (WaitForWorkload, execution-environment resolver, artifact primary-container helpers, environment var shapes).
• Fix fsutil.DirExists/FileExists to treat all os.Stat errors as “does not exist” (preventing nil deref), and harden config tests against process-global viper leakage.

Reviewed changes

Copilot reviewed 61 out of 61 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
internal/workload/workload.go	Add workload terminal/error status helpers and WaitForWorkload polling utility.
internal/workload/workload_test.go	Add unit tests for workload status helpers and WaitForWorkload.
internal/workload/wapi/paths.go	Centralize state dir resolution (.datarobot/wapi with legacy .wapi fallback) and export helpers.
internal/workload/wapi/paths_test.go	Test new resolver behavior (current vs legacy preference, unlinked default).
internal/workload/wapi/migrate.go	Introduce silent, non-failing legacy-to-current state migration helper.
internal/workload/wapi/migrate_test.go	Test migration behavior (move, both-present, unmovable fallback).
internal/workload/wapi/manifest.go	Update docs/comments to refer to the new state directory location.
internal/workload/wapi/manifest_test.go	Update tests to read/write manifest via Dir()-resolved paths.
internal/workload/wapi/init.go	Update initialization to create the new state dir layout and detect already-linked projects in both locations.
internal/workload/wapi/init_test.go	Update init tests for the new state dir layout.
internal/workload/wapi/history.go	Update docs/comments for history log path semantics.
internal/workload/wapi/history_test.go	Update history tests to use Dir()-resolved paths.
internal/workload/wapi/helpers_test.go	Update helper that seeds a state dir for tests to use Dir().
internal/workload/wapi/errors.go	Update error messages and comments to be state-dir-generic.
internal/workload/wapi/doc.go	Update package docs to reference .datarobot/wapi.
internal/workload/wapi/config.go	Update docs/comments to refer to new state dir location.
internal/workload/wapi/config_test.go	Update config tests to use Dir()-resolved paths.
internal/workload/sync/synclock.go	Use wapi.Dir() for sync lock placement under the state directory.
internal/workload/sync/rollback.go	Use wapi.RollbackDir() and recover stale rollback trees from both current and legacy locations.
internal/workload/sync/rollback_test.go	Update rollback tests for new rollback dir helpers and legacy recovery sweep.
internal/workload/sync/phase6_state.go	Update comment to reflect history log location generically.
internal/workload/sync/phase1_gather.go	Improve read error paths to use wapi.ConfigPath()/wapi.Dir() rather than hard-coded .wapi paths.
internal/workload/sync/phase0_preflight.go	Run migration before recovery/locking (skipped for diff/preview modes).
internal/workload/sync/path_safety_test.go	Update assertions to use rollback dir helper instead of hard-coded legacy path.
internal/workload/sync/limits.go	Update comment to reflect new rollback tree location.
internal/workload/sync/doc.go	Update package docs to be location-agnostic.
internal/workload/resolveart.go	Update artifact id resolution hints/errors to be linked-project/state-dir aware.
internal/workload/resolveart_test.go	Add coverage for legacy state-dir reads and update assertions for new messaging/paths.
internal/workload/ignore/matcher.go	Expand system excludes to include .datarobot/wapi while keeping legacy .wapi.
internal/workload/ignore/matcher_test.go	Extend system-exclude coverage for new state dir and ensure .datarobot/cli is not excluded.
internal/workload/ignore/doc.go	Update docs to reflect new system exclude semantics.
internal/workload/execenv.go	Add ResolveExecutionEnvironment helper for name-or-id resolution with paging and host validation.
internal/workload/execenv_test.go	Add tests for EE resolution by name/id, paging, missing versions, not-found, and cross-host next rejection.
internal/workload/artifact.go	Add container name/env var parsing and unify “primary container” traversal via shared helper + new Primary* helpers.
internal/workload/artifact_test.go	Add tests for primary-container helpers and env var marshal/unmarshal shapes.
internal/validate/validator.go	Update comment example paths for ID validation to match new state-dir layout.
internal/fsutil/fsutil.go	Fix stat error handling to return false on any error (avoid nil deref).
internal/fsutil/fsutil_test.go	Add regression tests for ENOTDIR/EACCES cases to prevent panics.
internal/config/perms_test.go	Ensure viper state is reset after tests to avoid leakage across shuffles.
internal/config/config_test.go	Reset viper between suite tests and tighten config file creation/assertions.
docs/development/workload-wapi-validation.md	Update docs to refer to workload “state directory” under .datarobot/wapi.
docs/development/README.md	Update dev docs index entry to “Workload state directory validation”.
docs/commands/README.md	Update command docs to reference .datarobot/wapi state directory.
docs/commands/artifact.md	Update artifact docs for new state dir path + legacy migration note.
cmd/artifact/code/versions/cmd.go	Resolve project dir to abs path, run migration, and improve config read error path reporting.
cmd/artifact/code/versions/cmd_test.go	Update test comment for new state dir wording.
cmd/artifact/code/init/display.go	Update “already linked” messaging to point at the resolved state dir.
cmd/artifact/code/init/display_test.go	Update assertions for new “already linked” messaging and dynamic state dir path.
cmd/artifact/code/init/cmd.go	Update help text to .datarobot/wapi and run migration before link checks.
cmd/artifact/code/init/cmd_test.go	Update init tests for new state dir paths and wording.
cmd/artifact/code/codesync/cmd_test.go	Update test comments for “state directory” terminology.
cmd/artifact/code/cmd.go	Update top-level artifact code help text for new state dir location.
cmd/artifact/code/checkout/cmd.go	Update help text for new state dir location and migrate before running.
cmd/artifact/code/checkout/cmd_test.go	Update output assertions and history path reads for new state dir layout.
cmd/artifact/code/checkout/checkout.go	Update config read error to use wapi.ConfigPath().
cmd/artifact/build/logs/cmd.go	Update help text for new linked config path.
cmd/artifact/build/internal/buildargs/buildargs.go	Make help text state-dir-agnostic (“linked project” instead of .wapi).
cmd/artifact/build/internal/buildargs/buildargs_test.go	Update “not linked” assertions to match new wording.
cmd/artifact/build/get/cmd.go	Update help text and example wording for linked artifact id source.
cmd/artifact/build/create/cmd.go	Update help text for new linked config path.
cmd/artifact/build/cmd.go	Update help text for new linked config path.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[adamalpi]

Solid PR overall — migration edge cases are well covered, the lock-inode analysis holds under rename, path normalization makes the new two-segment exclude safe on Windows, and all touched packages pass go test -race at the PR head. A few findings below; the two Medium naming/UX ones are worth deciding before merge since this release is the one cheap migration window.

[c-h-russell-walker]

FYI we already use .datarobot in Codespaces so you may want to ensure this works well with that.

Please see: https://github.com/datarobot/notebooks/tree/main#usage-of-datarobot-in-codespaces

Remembered I wanted to over-communicate that besides checking the functionality in Codespaces that it's also important to note:

... hidden directory in Codespaces ...

So if you put things in .datarobot it won't be see in the Codespaces (left sidepanel) filesystems UI (or the API response that powers it) - it'll show in the terminal of course but not in API/UI - it's excluded in code:

We purposefully don't return this directory in the API results that the UI uses and therefore this directory does not show in the filesystem left sidepanel. This directory name is a constant DR_NOTEBOOKS_FILESYSTEM_HIDDEN_DIRECTORY seen in /common/collections/filesystems.py

https://github.com/datarobot/notebooks/blob/main/common/collections/filesystems.py#L91

[ajalon1]

🤖 Two minor security observations from a review pass (low severity / defense-in-depth): (1) case-sensitive system-exclude matching in the ignore matcher, and (2) an unbounded pagination loop in ResolveExecutionEnvironment. Repros and suggested fixes are inline.

[wojtekwdr]

/approve-smoke-tests

[github-actions]

🔐 Fork PR smoke tests triggered by @wojtekwdr

⚠️ Security Notice: This will run tests with access to repository secrets.

What happens next:

1. Security scans will run automatically (Trivy, gosec)
2. If security scans pass, smoke tests will run
3. Results will be posted as PR comments

⚠️ Important: Review the PR code carefully before approving!

[github-actions]

🔐 Fork smoke tests started by maintainer

⏳ Security scans passed. Running smoke tests...

Commit: 1ccfdb8ae0ff3e52ee63ce1a045a2ac5563e7495
View run

[github-actions]

✅ All smoke tests passed! (Fork PR)

✅ Security Scan: success
✅ Linux: success
✅ Windows: success

View run details