Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions container-runner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ There are two working local paths:
2. Run the built Unity server behind `container-runner`, create a Rivet actor locally,
and connect a FishNet client through the local Rivet guard URL.

A production image for **Rivet Compute** (the Cloud Run serverless model) is provided
see [Rivet Compute](#rivet-compute) below.
A production image for **Rivet Compute** (the serverless model) is provided. See
[Rivet Compute](#rivet-compute) below.

## Project Layout

Expand Down Expand Up @@ -207,7 +207,7 @@ Knobs: `LOAD_COUNT` (default 25), `LOAD_CONCURRENCY` (default 64).

**Running the full 1000:** 1000 local instances is ~2000 processes (a Rust runner + Node
child each) and needs a beefy host plus a raised `ulimit -n`. For a true 1000-container run,
point `load-test.mjs` at **Rivet Cloud** instead — Cloud Run scales the containers, no local
point `load-test.mjs` at **Rivet Cloud** instead, which scales the containers with no local
limit. Set the engine env and let a single pool auto-scale:

```bash
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ const body = {
// Seconds the engine holds the /start request before draining to a fresh one.
request_lifespan: 900,
drain_grace_period: 30,
// 1:1 actor<->container mapping (Cloud Run concurrency=1 model).
// 1:1 actor<->container mapping (serverless concurrency=1 model).
slots_per_runner: 1,
max_runners: 1,
max_concurrent_actors: 1,
Expand Down
2 changes: 1 addition & 1 deletion container-runner/examples/e2e-test/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Local end-to-end: self-hosted Rivet engine + the game container (container-runner
# wrapping the Node test server). All x86_64/amd64 to mirror Cloud Run.
# wrapping the Node test server). All x86_64/amd64 to mirror the serverless platform.
#
# Flow:
# create actor (POST /actors) -> engine POSTs /api/rivet/start to game:8080
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# server), then drive a WebSocket ping-pong through the guard to every one.
#
# Each container-runner instance is one "container": its own front-door port, its own child
# port, and its own serverless runner pool (load-<i>). This mirrors the Cloud Run 1:1
# port, and its own serverless runner pool (load-<i>). This mirrors the serverless 1:1
# actor<->container model locally, so `LOAD_COUNT` instances == that many containers.
#
# engine (:7420 guard, :7421 api)
Expand Down
4 changes: 2 additions & 2 deletions container-runner/examples/test-server/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# Build from the RIVET REPO ROOT so the workspace and examples are in context.
# Arch-agnostic: builds for the host/requested platform.
# Local (native, e.g. arm64 Mac): docker build -f container-runner/examples/test-server/Dockerfile -t game:latest .
# Cloud Run (x86_64): docker build --platform linux/amd64 -f container-runner/examples/test-server/Dockerfile -t game:amd64 .
# Serverless (x86_64): docker build --platform linux/amd64 -f container-runner/examples/test-server/Dockerfile -t game:amd64 .
#
# Production pattern (per spec) — instead of building from source, curl a released
# binary into an existing image:
Expand All @@ -33,7 +33,7 @@ COPY --from=builder /usr/local/bin/rivet-container-runner /usr/local/bin/rivet-c
COPY container-runner/examples/test-server/ /app/test-server/
RUN cd /app/test-server && npm install --omit=dev

# Cloud Run sets $PORT (the serverless front door). The child listens on CHILD_PORT.
# The serverless platform sets $PORT (the front door). The child listens on CHILD_PORT.
ENV PORT=8080 \
CHILD_PORT=7770 \
RIVET_ACTOR_NAME=game
Expand Down
2 changes: 1 addition & 1 deletion container-runner/examples/test-server/server.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
//
// This is the child process that container-runner (rivet-container-runner) spawns inside
// the container. It stands in for a real Unity FishNet dedicated server while we validate
// the Rivet -> Cloud Run -> container pipeline.
// the Rivet -> serverless -> container pipeline.
//
// Normal behavior:
// - Binds HTTP+WebSocket on $PORT (default 7770) on 0.0.0.0.
Expand Down
95 changes: 35 additions & 60 deletions container-runner/src/actor.rs
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
//! The `GameServer` actor: wraps one child game-server process per actor.
//!
//! Lifecycle: `on_start` reserves a port and spawns the child, waiting for
//! readiness (so the actor is never reported ready before the child listens),
//! `run` is a watchdog that reports unexpected child exits,
//! `on_fetch`/`on_websocket` proxy tunneled traffic to the child's port, and
//! `on_destroy` stops the child while the instance stays warm for the next
//! placement.
//! `on_start` reserves a port and spawns the child (waiting for readiness), `run`
//! watchdogs unexpected child exits, `on_fetch`/`on_websocket` proxy tunneled
//! traffic to the child, and `on_sleep`/`on_destroy` stop it.

use std::sync::{Arc, LazyLock};

Expand All @@ -14,16 +11,15 @@
use rivetkit::{Actor, ActorKeySegment, Ctx, Request, Response, WebSocket, action};
use tokio::sync::Mutex as TokioMutex;

use crate::child::{ChildProcess, SpawnSpec, log_prefix};

Check warning on line 14 in container-runner/src/actor.rs

View workflow job for this annotation

GitHub Actions / Rustfmt

Diff in /home/runner/work/actors/actors/container-runner/src/actor.rs
use crate::input::ActorInput;
use crate::{
children, drain_grace, effective_stop_grace, exit_token, release_child_port,
request_exit, reserve_child_port, runner_config,
};

/// Live actor contexts on this instance, keyed by actor id. Lets the process
/// shutdown path report actors as crashed when the platform reclaims the
/// container out from under them.
/// Live actor contexts keyed by actor id, so the shutdown path can report actors
/// as crashed when the platform reclaims the container.
static ACTOR_CTXS: LazyLock<scc::HashMap<String, Ctx<GameServer>>> =
LazyLock::new(scc::HashMap::new);

Expand All @@ -32,14 +28,12 @@
}

impl GameServer {
/// Shared teardown for sleep and destroy: for a game server the two are
/// materially the same event, because the in-memory match state lives in
/// the child and cannot outlive the container. The launch spec is the
/// persisted actor state, so a later wake respawns an equivalent child.
/// Shared teardown for sleep and destroy. For a game server they are the same
/// event: match state lives in the child and cannot outlive the container, and
/// a later wake respawns an equivalent child from the persisted launch spec.
async fn stop_child(&self, actor_id: &str, reason: &str) {
// Remove from the registry FIRST so the watchdog treats the exit as
// deliberate, then stop. `stop` is idempotent if the process shutdown
// sweep already stopped this child.
// Remove from the registry first so the watchdog treats the exit as
// deliberate. `stop` is idempotent if the shutdown sweep already ran.
children().remove_async(actor_id).await;
ACTOR_CTXS.remove_async(actor_id).await;
let child = self.child.lock().await.take();
Expand All @@ -48,11 +42,9 @@
release_child_port(child.child_port).await;
}

// Exit the whole process once the last child on this instance stops. The
// runner is PID 1, so `request_exit` cancels `EXIT`, which wakes `main`
// to run the graceful envoy close and then return, stopping the container
// so the platform reaps it. Guarded on an empty registry so a multi-actor
// instance does not tear down siblings still hosting a child.
// Exit the process once the last child stops: `request_exit` cancels
// `EXIT`, waking `main` to close the envoy and return (the runner is
// PID 1). Guarded on an empty registry so siblings survive.
if children().is_empty() {
request_exit(actor_id, reason);
} else {
Expand All @@ -64,12 +56,9 @@
}
}

/// Engine pause path (sleep, lost, going-away). Give the child up to
/// `DRAIN_GRACE` to finish its in-flight work and exit on its own before we
/// force a stop; the child is not signalled during the window. A natural
/// child exit ends the wait immediately, and a platform SIGTERM (which
/// cancels the exit token) cuts it short so the reclaim's SIGTERM→SIGKILL
/// budget is honored.
/// Engine pause (sleep, lost, going-away): let the child finish and exit on its
/// own for up to `DRAIN_GRACE` before forcing a stop. A child exit or a platform
/// SIGTERM (which cancels the exit token) ends the wait early.
async fn drain_then_stop_child(&self, actor_id: &str, reason: &str) {
let child = self.child.lock().await.clone();
if let Some(child) = child {
Expand Down Expand Up @@ -118,20 +107,17 @@
let actor_id = ctx.actor_id().to_string();
let key = actor_key_string(&ctx);

// Surface the resource monitor's status here, tagged with the actor id, so
// it is visible in actor-scoped log views. The monitor's own enable/disable
// logs are process-level and have no actor id, so they are filtered out of
// those views.
// Tagged with the actor id so it shows in actor-scoped log views; the
// monitor's own process-level enable/disable logs are filtered out there.
tracing::info!(
actor_id = %actor_id,
resource_monitor_enabled = crate::monitor::enabled(),
resource_monitor_source = crate::monitor::sampling_source(),
"resource monitor status"
);

// An engine retry for an actor that is already running here must be an
// idempotent no-op: rejecting it would make the engine tear down a
// healthy actor.
// An engine retry for an already-running actor must be an idempotent no-op;
// rejecting it would make the engine tear down a healthy actor.
if let Some(existing) = children().read_async(&actor_id, |_, c| c.clone()).await {
if !existing.has_exited() {
println!(
Expand Down Expand Up @@ -172,9 +158,8 @@
key: key.clone(),
};

// Version line, tagged with the actor id so it is visible in actor-scoped
// logs. `git_sha` is omitted entirely when unknown rather than logged as
// "unknown".
// Tagged with the actor id for actor-scoped logs; `git_sha` is omitted when
// unknown rather than logged as "unknown".
match crate::git_sha() {
Some(git_sha) => tracing::info!(
actor_id = %actor_id,
Expand All @@ -200,17 +185,13 @@
Ok(child) => Arc::new(child),
Err(err) => {
release_child_port(child_port).await;
// A failed start is this actor's alone and does not take the
// instance down. The container stays warm and ready for the next
// placement, and stays alive long enough for the log agent to
// drain the failure logs before the platform reaps it.
// A failed start is this actor's alone; it does not take down others.
return Err(err);
}
};

// The global registry lets the process shutdown path stop children
// even when actor hooks never run, and arbitrates the deliberate-stop
// vs unexpected-exit race for the watchdog in `run`.
// The global registry lets the shutdown path stop children when hooks never
// run, and arbitrates the deliberate-stop vs unexpected-exit race in `run`.
if children()
.insert_async(actor_id.clone(), child.clone())
.await
Expand All @@ -221,19 +202,16 @@
release_child_port(child_port).await;
anyhow::bail!("a child for actor {actor_id} is already registered");
}
// Register only now that startup has succeeded. Registering earlier would
// leak an entry for any generation whose start failed, since a failed
// start never runs on_destroy/on_sleep to remove it.
// Register only after startup succeeds; a failed start never runs a stop
// hook to remove the entry, so registering earlier would leak it.
register_ctx(&actor_id, &ctx).await;
*self.child.lock().await = Some(child);
Ok(())
}

/// Watchdog: waits for the child to exit. Deliberate stops remove the
/// child from the global registry first, so winning the `remove` race
/// means the exit was unexpected and the actor must be torn down. A clean
/// exit (code 0) destroys the actor; any other exit returns an error so
/// the framework reports an errored stop and the engine records the crash.
/// Watchdog for the child exiting. Deliberate stops remove it from the registry
/// first, so winning the `remove` race means the exit was unexpected: a clean
/// exit destroys the actor, any other reports an errored stop (a crash).
async fn run(self: Arc<Self>, ctx: Ctx<Self>) -> Result<()> {
let Some(child) = self.child.lock().await.clone() else {
anyhow::bail!("run: child process was never spawned");
Expand Down Expand Up @@ -294,10 +272,8 @@
crate::proxy::ws_proxy(child_port, path, ws).await
}

/// Engine-initiated sleep. `no_sleep` suppresses idle sleep, but the
/// engine can still sleep an actor (dashboard, crash policy, eviction
/// ahead of instance retirement); leaving the child running would orphan
/// it on an instance the engine considers vacated.
/// Engine-initiated sleep. `no_sleep` blocks only idle sleep; the engine can

Check warning on line 275 in container-runner/src/actor.rs

View workflow job for this annotation

GitHub Actions / Rustfmt

Diff in /home/runner/work/actors/actors/container-runner/src/actor.rs
/// still sleep an actor (dashboard, crash policy, eviction), so we stop the child.
async fn on_sleep(self: Arc<Self>, ctx: Ctx<Self>) -> Result<()> {
self.drain_then_stop_child(ctx.actor_id(), "actor sleeping").await;
Ok(())
Expand All @@ -318,10 +294,9 @@
.await;
}

/// Report every live actor on this instance as crashed. Called when the
/// platform reclaims the container (an unexpected SIGTERM) so the reclaim
/// surfaces as a crash on the engine instead of a silent reallocation. Runs
/// while the envoy is still connected so the crash reaches the engine.
/// Report every live actor as crashed. Called when the platform reclaims the
/// container (unexpected SIGTERM), while the envoy is still connected, so the
/// reclaim surfaces as a crash on the engine instead of a silent reallocation.
pub async fn crash_all_actors(message: &str) {
let mut ctxs = Vec::new();
ACTOR_CTXS
Expand Down
23 changes: 9 additions & 14 deletions container-runner/src/child.rs
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
//! Child game-server process management: spawn, log piping, readiness, SIGTERM stop.
//!
//! Ownership model: a dedicated "reaper" task exclusively owns the `tokio::process::Child`
//! and awaits its exit, publishing the result on a `watch` channel. `stop()` and readiness
//! checks signal/observe via the pid and the watch channel, so they never contend for the
//! child handle (which would deadlock against the reaper's long-lived `wait()`).
//! A dedicated reaper task owns the `tokio::process::Child` and publishes its exit on a
//! `watch` channel; `stop()` and readiness checks signal/observe via the pid and channel,
//! so they never contend for the child handle (which would deadlock the reaper's `wait()`).

use std::collections::HashMap;
use std::net::Ipv4Addr;
Expand Down Expand Up @@ -72,20 +71,17 @@ impl ChildProcess {

let prefix = log_prefix(&actor_id, key.as_deref());

// Guarantee the child port is free BEFORE spawning. Otherwise a stale child from a
// prior start (in a reused container instance) still holding the port would make
// `wait_until_ready` below false-positive: it connects to the OLD listener and
// reports the NEW child "ready" even though the new child failed to bind
// (`Address already in use`) and is dead. Refuse the start with a clear diagnostic
// instead — this container hosts exactly one game server on a fixed port.
// Refuse to spawn if the port is already taken. A stale child from a prior start
// still holding it would make `wait_until_ready` false-positive on the OLD listener
// while the new child dies with `Address already in use`.
if TcpStream::connect((Ipv4Addr::LOCALHOST, child_port))
.await
.is_ok()
{
anyhow::bail!(
"child port {child_port} is already in use before spawning `{program}`: a \
previous game server is still running in this container. container-runner \
hosts one actor per container configure the serverless runner with \
hosts one actor per container; configure the serverless runner with \
max_concurrent_actors=1 and platform request concurrency=1."
);
}
Expand Down Expand Up @@ -143,9 +139,8 @@ impl ChildProcess {
exited_rx,
};

// If the child crashes before opening its port, or never opens it, make sure we
// don't leave it running: kill it before surfacing the start failure. (The reaper
// task owns the tokio Child, so dropping `this` alone would NOT kill a hung child.)
// Kill the child before surfacing a readiness failure; dropping `this` alone would
// not (the reaper owns the tokio Child), leaving a hung child running.
if let Err(err) = this.wait_until_ready(readiness_timeout).await {
this.stop(Duration::from_secs(2)).await;
return Err(err);
Expand Down
17 changes: 7 additions & 10 deletions container-runner/src/input.rs
Original file line number Diff line number Diff line change
@@ -1,19 +1,16 @@
//! The actor input payload describing how to launch the child game server.
//!
//! Everything the game server needs to launch (command, args, env, port) is
//! carried in the actor's create-time `input` payload, CBOR-encoded per the
//! RivetKit convention. All fields are optional; anything omitted falls back
//! to the CLI-provided template (`rivet-container-runner -- <command...>`).
//! The decoded input is also the actor's persisted state so a woken actor
//! restores the same launch spec without re-decoding input.
//! The command, args, env, and port are carried in the actor's create-time `input`
//! (CBOR per RivetKit); anything omitted falls back to the CLI template
//! (`rivet-container-runner -- <command...>`). This is also the actor's persisted
//! state, so a woken actor restores the same launch spec.

use serde::{Deserialize, Serialize};
use std::collections::HashMap;

/// Shape of the actor `input` payload. Unknown fields are ignored rather than
/// rejected: this type is also the persisted actor state, and a strict decode
/// would break waking actors after a rollback to a binary that predates a
/// newly added field.
/// Shape of the actor `input` payload. Unknown fields are ignored, not rejected:
/// this is also the persisted state, and a strict decode would break waking actors
/// after a rollback to a binary predating a new field.
#[derive(Debug, Default, Serialize, Deserialize)]
pub struct ActorInput {
/// Overrides the CLI command template entirely (program + fixed args).
Expand Down
Loading
Loading