feat: skip unchanged checkpoints via in-guest eBPF inspector

[void-main]

Implements the proposal in #2580.

Adapts the Crab paper's "skip unchanged checkpoints" idea (arXiv 2604.28138) to e2b's Firecracker-microVM model. When the SDK calls POST /sandboxes/{id}/snapshots with the new opt-in skipIfUnchanged: true, the orchestrator consults an in-guest eBPF inspector inside envd and short-circuits to the prior snapshot's IDs (zero pause cost) when nothing recovery-relevant has changed. Default behavior is unchanged.

Why microVMs need a different approach than the paper

Crab's prototype uses runc + CRIU + ZFS containers with eBPF on the host kernel. Firecracker microVMs trap guest syscalls to the guest kernel, so host eBPF cannot see guest VFS, file paths, or PIDs. The paper itself cites e2b/Firecracker only as an expensive baseline in Table 1, not as a supported target.

This PR therefore:

• Runs the eBPF inspector inside the guest (in envd).
• Drops everything else from the paper that depends on host visibility (partial-component checkpoints, host-scoped scheduler, fast-forward replay, HTTPS MITM).
• Keeps the part that pays for itself: the skip-vs-full decision.

Architecture

Three slices, all gated behind the inspector-skip-unchanged LaunchDarkly flag and a coordinated envd version bump (0.5.15 → 0.5.16):

Inspector (in-guest, in envd)

• New Connect-RPC InspectorService with QueryChanges, ResetEpoch, Status.
• Filesystem net-change tracker. eBPF program attached to 16 raw syscall tracepoints (openat w/ write flags, unlinkat, renameat2, write*, truncate*, mkdirat, linkat, symlinkat, fallocate, fchmodat, fchownat, openat2). Counter is incremented only when bpf_get_current_cgroup_id() matches a userspace-populated allowlist of envd's user-spawned cgroups.
• Process / memory net-change tracker. Pure /proc reader: cgroup.procs membership delta + /proc/PID/pagemap soft-dirty bits on survivors, with /proc/PID/clear_refs reset at each epoch. Excludes envd's own PID.
• Graceful degradation. If the BPF object can't load or soft-dirty isn't supported, QueryChanges reports degraded=true and the orchestrator falls through to a full pause. Correctness preserved at every step.
• Build is gated by the inspector_bpf build tag so the envd binary still compiles cleanly on hosts without BPF tooling. Pre-compiled BPF objects are embedded via bpf2go; the envd binary itself stays pure-Go.

Orchestrator

• SandboxCheckpointRequest.skip_if_unchanged and SandboxCheckpointResponse.{unchanged, published_build_id}.
• Server.Checkpoint short-circuit branch before MarkStopping: opens a Connect-RPC client to the in-guest inspector (750ms cap), and on !degraded && !filesystem_changed && !processes_changed returns Unchanged: true, PublishedBuildId: <prior> immediately.
• New per-server lastPublishedSnapshot map (in-memory, best-effort; restart forces the next call to fall through).
• After successful full-path checkpoints, async Inspector.ResetEpoch so subsequent queries measure changes relative to the new baseline.
• New OrchestratorInspectorCheckpointDecisions counter sliced by decision={skipped,fallthrough,full} for skip-rate dashboards.

API

• POST /sandboxes/{sandboxID}/snapshots accepts skipIfUnchanged: bool and returns unchanged: bool on SnapshotInfo.
• New sqlc query GetTemplateIDByBuildID looks up the env_id for the prior build when the orchestrator short-circuits, so the speculative new build can be marked failed and the prior template returned to the caller.

Failure modes (all preserve correctness)

• Feature flag off → falls through to a full checkpoint.
• envd version < 0.5.16 → falls through.
• Inspector RPC error/timeout → falls through.
• Inspector reports degraded=true → falls through.
• No prior snapshot recorded for the sandbox → falls through.
• Orchestrator restart → in-memory map cleared → next call falls through.

Rollout

1. Feature flag default-off. Internal benchmarks on a read_file-heavy workload to measure actual skip rate. Crab reports 87% on Claude-code/Terminal-Bench; ours will differ because Firecracker pause cost and per-snapshot overhead are different.
2. Default-on for opted-in templates after we observe meaningful skip rates with no recovery regressions.
3. (Separate RFC, not in this PR) Partial checkpoints — memory-only/rootfs-only artifacts paired into a versioned manifest. That's significantly bigger surgery in Snapshot/upload paths; deferred.

Verification

• go vet clean across envd, orchestrator, api, shared, db, integration tests, with and without -tags inspector_bpf.
• envd unit tests: scaffold contract tests + a real-kernel soft-dirty test (skipped if CONFIG_MEM_SOFT_DIRTY is missing).
• orchestrator: lastSnapshotMap test.
• Integration tests in tests/integration/.../snapshot_skip_unchanged_test.go:
  • Default omitted field preserves historical behavior.
  • First call against a fresh sandbox cannot short-circuit (orchestrator-logic property; holds even with the inspector loaded).
  • After a real file write between two skip-eligible calls, the second never returns unchanged=true and produces a fresh template id.
  • Wire-shape lock: skipIfUnchanged=false is identical to omitting it.

Tests are permissive on the outcome of the inspector decision (unchanged=true vs false) because the in-guest tracker is only active when envd was built with -tags inspector_bpf and the guest kernel supports BPF + soft-dirty. We assert structural guarantees that hold regardless of tracker state.

Asks for the maintainers

• Confirmation that e2b's microVM kernel ships with CONFIG_DEBUG_INFO_BTF, CONFIG_KPROBES, CONFIG_MEM_SOFT_DIRTY, CONFIG_BPF_SYSCALL. If not, the inspector ships in degraded mode (no win, no regression) until the kernel is rebuilt.
• Greenlight on the build pipeline change: developers running make generate/envd need clang≥10 + the vendored libbpf headers in this PR. The compiled .o is checked in, so CI building the envd binary doesn't need clang.
• Greenlight on the LaunchDarkly flag name (inspector-skip-unchanged) and the envd version-bump pattern.

Closes #2580.

Test plan

• Build envd with -tags inspector_bpf, confirm Status RPC reports bpf_loaded=true, soft_dirty_supported=true.
• Build envd without the tag, confirm Status reports degraded_reason="fs tracker disabled...".
• Run tests/integration/.../snapshot_skip_unchanged_test.go against a self-hosted stack.
• Sanity-check the new metric: e2b dashboard should show orchestrator_inspector_checkpoint_decisions_total{decision=...} after running the integration tests.
• Verify make test and make lint are clean across all packages.

🤖 Generated with Claude Code

[cla-bot]

We require contributors to sign our Contributor License Agreement, and we don't have @void-main on file. You can sign our CLA at https://e2b.dev/docs/cla . Once you've signed, post a comment here that says '@cla-bot check'

[qodo-code-review]

ⓘ You've reached your Qodo monthly free-tier limit. Reviews pause until next month — upgrade your plan to continue now, or link your paid account if you already have one.

[gemini-code-assist]

Code Review

The scanVMASoftDirty function in proc_tracker_linux.go hardcodes pageSize to 4096 bytes. This logic will fail on Linux kernels configured with larger page sizes, such as 16KB or 64KB common on ARM64, leading to incorrect pagemap offsets and potentially missed dirty pages. The system page size should be retrieved at runtime using os.Getpagesize() to ensure correctness across all supported architectures.