Skip to content

feat(driver): implement sandbox RPCs (Create/Get/List/Stop/Delete/Validate + Watch) - #14

Merged
morphis merged 17 commits into
canonical:mainfrom
kadinsayani:feat/lxd-driver-rpcs
Aug 13, 2026
Merged

feat(driver): implement sandbox RPCs (Create/Get/List/Stop/Delete/Validate + Watch)#14
morphis merged 17 commits into
canonical:mainfrom
kadinsayani:feat/lxd-driver-rpcs

Conversation

@kadinsayani

@kadinsayani kadinsayani commented Jul 14, 2026

Copy link
Copy Markdown
Member

Summary

Wires up the ComputeDriver gRPC service against LXD: every RPC beyond GetCapabilities was previously Unimplemented.

  • ValidateSandboxCreate — required-field validation, label-key format checks (shared with create_sandbox's own validation so the two can't drift), and GPU count sanity (rejects 0, None already means "one GPU, default assignment")
  • GetSandbox / ListSandboxes — read from LXD instance config; resolvable by sandbox_name or sandbox_id (LXD has no by-id lookup, so sandbox_id-only requests fall back to scanning for a config match)
  • CreateSandbox — create stopped, push token file, start; rolls back (deletes) the instance if the token push fails, so a failed create doesn't leave an orphaned container; always uses the configured default image in v1 (template.image is accepted but not yet consulted)
  • StopSandbox — graceful stop; idempotent on not-running (handles both LXD's synchronous 400 and its asynchronous OperationFailed shape via a shared is_already_stopped helper)
  • DeleteSandbox — force-stop + delete; idempotent on 404 at every step (including a concurrent-deletion race on the force-stop step itself); broadcasts a WatchSandboxes Deleted event on success
  • WatchSandboxes — deletion broadcast channel so the gateway evicts a deleted sandbox immediately rather than waiting on its 60s reconcile loop; a lagged receiver gets a data_loss stream error (rather than silently missing events) so it knows to reconnect and re-list
  • Fail-fast startup check: exits with a clear error if the configured default image alias isn't present in LXD, instead of surfacing an opaque 404 on the first create_sandbox call
  • README: adds a v1-alpha status line, a Quickstart (the real sequence verified end-to-end this cycle), and Security/Known-limitations sections

End-to-end verification

Ran a full smoke test against a real OpenShell gateway + this driver (plaintext gateway, --bind-address 0.0.0.0, LXD on lxdbr0):

  • Manual CLI lifecycle: sandbox create reached Ready with a fully-populated policy (filesystem_policy, landlock, process) — exercises token delivery, container DHCP bring-up, and supervisor policy fetch in one pass. sandbox exec returned a working shell as the sandbox user. sandbox list immediately after sandbox delete did not show the deleted sandbox, confirming WatchSandboxes's broadcast path (not just the ~60s reconcile fallback).
  • The real e2e Rust test suite from OpenShell's e2e/rust/tests/ (smoke, sandbox_lifecycle ×2, sandbox_labels) — 4/4 passing, run unmodified against this driver.

Test plan

  • Unit tests (cargo test -p openshell-driver-lxd --lib) — 7 passing, including new ValidateSandboxCreate coverage for label-key and GPU-count validation.
  • Integration tests (tests/sandbox_lifecycle.rs) — full create → get → list → stop → delete lifecycle + sandbox_id-only resolution + 404/invalid-argument safety, against real LXD + the openshell-sandbox image — 5/5 passing.
  • Manual e2e smoke test against a real gateway (see above).
  • cargo build --workspace, clippy -D warnings, fmt --check, shellcheck all clean.

@kadinsayani
kadinsayani force-pushed the feat/lxd-driver-rpcs branch 2 times, most recently from e2d17e8 to 6fff66d Compare July 15, 2026 16:51
@kadinsayani
kadinsayani force-pushed the feat/lxd-driver-rpcs branch 12 times, most recently from 3ebd972 to 42cfd38 Compare July 18, 2026 01:31
@kadinsayani
kadinsayani requested a review from Copilot July 20, 2026 16:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements the remaining ComputeDriver sandbox lifecycle RPCs for openshell-driver-lxd by wiring them to LXD instance operations/config, adds deletion-driven WatchSandboxes events, and updates the sandbox image build + CI to support end-to-end integration testing against a real LXD daemon.

Changes:

  • Implement ValidateSandboxCreate, Create/Get/List/Stop/DeleteSandbox, and WatchSandboxes in the gRPC service + core driver logic (including LXD config/device/profile mapping).
  • Add real-LXD integration tests covering the full sandbox lifecycle via the in-process gRPC service.
  • Update image/build/CI scripts (rockcraft-based openshell-sandbox build, deterministic lxc publish compression, and leaner apt installs).

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
scripts/setup-lxd-test-env.sh Adjusts LXD test env image-alias existence check before staging the test image.
scripts/openshell-container-init.sh Simplifies /sandbox ownership setup inside the container init wrapper.
scripts/build-sandbox-image.sh Makes published image compression deterministic (--compression none).
rockcraft.yaml Builds openshell-sandbox from source via the Rust plugin and pins the version/commit.
crates/openshell-driver-lxd/tests/sandbox_lifecycle.rs Adds full sandbox lifecycle integration tests (create/get/list/stop/delete + error cases).
crates/openshell-driver-lxd/tests/get_capabilities.rs Updates test wiring for the new LxdComputeDriver::new(config, lxd) constructor.
crates/openshell-driver-lxd/src/mapping.rs Adds mapping helpers to translate between proto sandbox shapes and LXD instance config/devices/profiles.
crates/openshell-driver-lxd/src/main.rs Adds LXD endpoint selection (Unix socket vs HTTPS) and constructs LxdClient for the driver.
crates/openshell-driver-lxd/src/lib.rs Exposes the new internal mapping module.
crates/openshell-driver-lxd/src/grpc.rs Implements sandbox RPC handlers and adds a broadcast-backed WatchSandboxes deletion stream.
crates/openshell-driver-lxd/src/driver.rs Implements the core sandbox lifecycle against LXD (create/push token/start, get/list, stop, delete, id lookup).
crates/openshell-driver-lxd/src/config.rs Adds --gateway-grpc-port used to construct OPENSHELL_ENDPOINT for sandboxes.
crates/lxd-client/src/operations.rs Logs malformed websocket event frames at WARN instead of silently skipping.
Cargo.toml Enables tokio-stream’s sync feature for BroadcastStream.
Cargo.lock Updates lockfile for dependency feature changes.
.github/workflows/tests.yaml Uses apt-get install --no-install-recommends in test workflows.
.github/workflows/static-checks.yaml Uses apt-get install --no-install-recommends in static-check workflows.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/openshell-driver-lxd/src/driver.rs
Comment thread scripts/setup-lxd-test-env.sh Outdated
Comment thread crates/openshell-driver-lxd/src/grpc.rs
@kadinsayani
kadinsayani force-pushed the feat/lxd-driver-rpcs branch 2 times, most recently from 59c3c27 to 49dc694 Compare July 20, 2026 22:10
@kadinsayani
kadinsayani force-pushed the feat/lxd-driver-rpcs branch 2 times, most recently from 772e522 to 97a00cb Compare July 21, 2026 18:39
Copilot AI review requested due to automatic review settings July 21, 2026 20:41
@kadinsayani
kadinsayani force-pushed the feat/lxd-driver-rpcs branch from 97a00cb to d980919 Compare July 21, 2026 20:41

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 2 comments.

Comment thread crates/openshell-driver-lxd/src/driver.rs
Comment thread crates/openshell-driver-lxd/src/driver.rs Outdated
Copilot AI review requested due to automatic review settings July 21, 2026 20:52
@kadinsayani
kadinsayani force-pushed the feat/lxd-driver-rpcs branch from d980919 to 18253fb Compare July 21, 2026 20:52
Copilot AI review requested due to automatic review settings July 22, 2026 16:59
@kadinsayani
kadinsayani force-pushed the feat/lxd-driver-rpcs branch from b2ea999 to 5f06c9a Compare July 22, 2026 16:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

crates/openshell-driver-lxd/src/grpc.rs:171

  • BroadcastStreamRecvError also has a Closed variant; the current match is non-exhaustive and won’t compile (or will require a wildcard arm). Handle Closed explicitly and map it to a gRPC status (e.g. Unavailable).
            Err(BroadcastStreamRecvError::Lagged(n)) => Err(Status::data_loss(format!(
                "WatchSandboxes receiver lagged and missed {n} event(s); reconnect and re-list to resync"
            ))),
        });

Comment thread crates/openshell-driver-lxd/src/main.rs
Comment thread crates/openshell-driver-lxd/src/driver.rs
Comment thread crates/openshell-driver-lxd/tests/get_capabilities.rs Outdated
@kadinsayani
kadinsayani force-pushed the feat/lxd-driver-rpcs branch from 5f06c9a to a801ab9 Compare July 22, 2026 19:22
Copilot AI review requested due to automatic review settings July 22, 2026 19:22
@kadinsayani
kadinsayani marked this pull request as ready for review July 22, 2026 19:22
@kadinsayani
kadinsayani requested a review from a team July 22, 2026 19:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 3 comments.

Comment thread crates/openshell-driver-lxd/tests/get_capabilities.rs Outdated
Comment thread crates/openshell-driver-lxd/tests/sandbox_lifecycle.rs Outdated
Comment thread crates/openshell-driver-lxd/src/driver.rs Outdated
main.rs and the get_capabilities test built LxdComputeDriver with only
a Config, never constructing the LxdClient that the --lxd-url/socket
flags were added for. Build an LxdEndpoint from those flags (HTTPS+mTLS
when --lxd-url is set, a Unix socket otherwise) and pass the resulting
LxdClient through; update the test call sites to match.

LxdComputeDriver now holds an lxd: LxdClient field.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
…ndboxes

Implement the three read-only RPCs:

- validate_sandbox_create: checks that name, id, spec, and template are
  present, that every spec.template.labels key passes the same
  validation build_create_config already enforces at actual creation
  time (mapping::is_valid_label_key, exposed pub(crate) so both call
  sites share one rule instead of drifting apart), and that
  resource_requirements.gpu.count is at least 1 when set (0 is
  rejected; omitting count already means "one GPU, default
  assignment"). This lets a caller catch a request that would fail at
  create_sandbox before actually provisioning anything.
- get_sandbox: fetches the instance by name; a 404 maps to NOT_FOUND
- list_sandboxes: lists instances, filtered to those tagged with
  user.openshell.sandbox_id

Also add a resolve_name helper in grpc.rs that returns sandbox_name,
or an InvalidArgument error if it's empty. It takes sandbox_id too
(matching the proto request shape) but doesn't use it yet — all
current RPCs are name-addressed only.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
stop_sandbox stops the instance, treating "already stopped" as
success so repeated calls are safe.

delete_sandbox force-stops the instance first (LXD refuses to delete
a running instance), then deletes it; a 404 is treated as
already-deleted so callers can retry safely. On success it returns
the instance's user.openshell.sandbox_id, so a future WatchSandboxes
implementation can notify the gateway when a sandbox is removed.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
Tests the full create -> get -> list -> stop -> delete path through
the gRPC service, driven in-process against a real LXD daemon
(matching the pattern in tests/get_capabilities.rs).

- create_get_list_stop_delete_lifecycle: verifies the happy path end
  to end, including that list_sandboxes returns the new sandbox and
  that delete_sandbox is idempotent (a second call returns
  deleted=false)
- get_sandbox_unknown_name_returns_not_found: verifies the 404 to
  NOT_FOUND gRPC status mapping

Requires a running LXD with a default storage pool, an lxdbr0 network,
and an openshell-sandbox image alias.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
Add a server-streaming WatchSandboxes RPC that notifies the gateway
when a sandbox is deleted, so it can immediately evict the sandbox
from its store instead of waiting for the next reconcile cycle
(~60s TTL).

Uses a tokio broadcast channel (capacity 64) held inside
ComputeDriverService. Every successful DeleteSandbox that returns a
non-empty sandbox_id publishes to the channel; WatchSandboxes
subscribers receive WatchSandboxesDeletedEvent frames. A lagged
receiver (missed events because it fell behind the channel's
capacity) gets a data-loss stream error instead of having those
events silently dropped, so the gateway knows to reconnect and re-list
rather than trusting a store that may have missed a deletion.

Adds tokio-stream's "sync" feature to the workspace for
BroadcastStream.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
create_sandbox: if push_file_into_instance fails after the instance
has been created (stopped, not yet started), delete it before
returning the error. Without this the caller gets an error but a
stopped LXD container is left behind with no owner.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
create_sandbox: always use config.default_image for v1 - template.image
is accepted by ValidateSandboxCreate but not consulted to select sandbox
content per the v1 spec. Log a debug message when a non-empty
template.image is ignored so operators can see why their image choice
had no effect.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
stop_sandbox: the previous guard only checked for a synchronous 400
"not running" response, but ignored the possibility that
wait_operation itself could fail with the same underlying cause.

delete_sandbox: the previous guard was a blanket catch on any 400,
silently discarding real stop errors that happened to also return
400.

LXD's stop can also fail asynchronously - the request is accepted
(202) and the operation fails once polled, surfacing as
LxdError::OperationFailed rather than a synchronous 400. This happens
whenever the instance transitions to stopped between the request and
LXD processing it (e.g. a delete racing a prior stop). Add an
is_already_stopped helper that checks both the synchronous and
asynchronous error shapes, and use it at both failure points in both
methods.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
The instance can disappear between the initial get_instance check and
the force-stop call under concurrent deletion, and DeleteSandbox is
documented as idempotent on not-found — a 404 here shouldn't surface
as an error.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
…ly skipping

A JSON deserialization error on a single event frame already skips to
the next frame rather than reconnecting. Add a tracing::warn! so the
occurrence is visible in logs without aborting the wait.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
…ox/DeleteSandbox

resolve_name only ever looked at sandbox_name; sandbox_id was accepted
but silently ignored (bound as _sandbox_id). GetSandboxRequest,
StopSandboxRequest, and DeleteSandboxRequest all carry both fields
precisely so a caller can address a sandbox by either — a caller that
only has the gateway-assigned ID would get a confusing "sandbox_name
is required" error instead of a successful lookup.

Add LxdComputeDriver::find_name_by_sandbox_id, which scans
driver-managed instances for a user.openshell.sandbox_id config match
(LXD has no by-id lookup, only by-name). resolve_name now falls back
to it when sandbox_name is empty, and returns NOT_FOUND when neither
field resolves to an instance.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
If config.default_image isn't imported into LXD, the driver previously
started fine and only failed on the first create_sandbox call, as an
opaque LXD 404 wrapped in a gRPC error — hard to diagnose remotely,
especially once this runs as a background service rather than
something started interactively at a terminal.

Add LxdClient::image_alias_exists (GET /1.0/images/aliases/<alias>,
treating a 404 as Ok(false)) and check it once at startup, after
constructing LxdClient and before starting the gRPC server. Exit with
a clear error naming the missing alias and pointing at `make
sandbox-image` if it's absent. If the check itself errors (e.g. LXD
isn't reachable yet), log a warning and continue rather than blocking
startup — that failure mode is already surfaced clearly wherever it's
next hit.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
@kadinsayani
kadinsayani force-pushed the feat/lxd-driver-rpcs branch from a801ab9 to 76f651a Compare July 22, 2026 21:05
@kadinsayani
kadinsayani requested a review from morphis July 27, 2026 16:16
Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
Syncs proto/compute_driver.proto with upstream NVIDIA/OpenShell main,
which brings two changes:

  - DriverSandbox gains `string workspace = 6`, used by drivers to build
    collision-safe resource names and labels in shared-namespace mode.
  - DriverSandboxSpec.sandbox_token is now annotated with the
    `openshell.options.v1.secret` field option, so the contract imports
    options.proto.

Vendor proto/options.proto alongside it: protoc must resolve the import
on its include path or codegen fails outright. Teach the sync-proto
target to sync both files rather than compute_driver.proto alone, so the
next sync doesn't break the same way.

Persist the workspace on the LXD instance as user.openshell.workspace
and read it back in instance_to_driver_sandbox, mirroring how namespace
is already handled, so Get/List/Watch report the value the gateway sent
instead of silently dropping it. Gateways that don't yet populate the
field round-trip an empty string, which is the correct no-op.

Proto and driver changes land together so every commit builds.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>
Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>

@morphis morphis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morphis
morphis merged commit e591558 into canonical:main Aug 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants