feat(scheduler): add resource-bounded grouped round-robin scheduling

[shudorcl]

What

This PR adds an opt-in grouped_round_robin scheduling strategy as a bounded workload-affinity building block for #15. Requests with the same rootfs image reference or exact template reference stay on one eligible READY node until the group's sandbox, CPU, or memory budget is reached, after which a new group is assigned through global round-robin.

The Gateway now propagates templateID through the internal scheduling hint, scheduling snapshots derive current node status from discovery state and heartbeat TTL, and the new strategy is covered by configuration, documentation, and focused tests.

Why

Plain round-robin can scatter repeated workloads across nodes and repeatedly warm the same image or template artifacts. Grouped placement improves the opportunity for local cache reuse while bounding concentration and preserving the existing node resource filter.

Related issue

Refs #15

This PR provides request-identity affinity but does not close the broader cache-aware scheduling issue.

Scope and non-goals

In scope: grouped placement by the first rootfs image or exact template reference, configurable group budgets, fresh READY-node enforcement, bounded in-memory group state, gateway hint propagation, and operator documentation.

Non-goals: inspecting or scoring actual node cache or P2P inventory, resolving mutable image tags or template aliases, considering attached-drive identities, adding projected hard-capacity admission, or persisting and reconciling group state across scheduler restarts.

Design and behavior changes

discovered nodes -> current status derivation -> existing node resource filter -> READY-only grouped round-robin -> selected node

Each workload identity has at most one open group. Requests reuse its node while the request fits the configured group budget; otherwise the group closes and the request starts a new group on the next eligible node. New groups for all identities share one stable global round-robin cursor, while missing or oversized identities use a separate round-robin fallback.

Placement accounting is serialized and updated during scheduling so concurrent bursts cannot overfill an open group while heartbeat metrics lag. Open state is capped at 10,000 keys with LRU eviction. Template requests do not carry CPU or memory values, so their groups use the sandbox-count limit only.

Compatibility and operations

• Public API or generated protocol: The public HTTP API is unchanged. The internal scheduler protobuf adds optional NewSandboxHint.template_id field 2, which is wire-compatible with mixed Gateway and Scheduler versions.
• Configuration or defaults: The default remains round_robin. Selecting grouped_round_robin requires scheduler.grouped_round_robin.max_sandbox_count > 0; CPU and memory group limits are optional. Equivalent environment overrides are documented.
• Snapshot manifest, artifact layout, or storage format: N/A; no snapshot or artifact format changes.
• Upgrade and rollback: No migration is required. Group state is in-memory and resets on restart; older components or missing hints degrade to ordinary round-robin placement.
• Host requirements, permissions, ports, or dependencies: No new host requirements, ports, services, or dependencies.

Validation

• make fmt
• make clippy
• make test-unit
• Relevant Rust integration tests
• make -C services test (required when services/ changes)
• Generated clients/server regenerated with the documented make target
• Documentation updated
• Benchmarks or performance comparison completed

Commands and results:

make -C services build
make -C services test
make -C services vet
(cd services && go test ./...)
(cd services && go test -race ./...)
git diff --check
# PASS locally

make -C services proto PROTOC=<protoc 3.21.12>
gofmt -w services/api/proto/scheduler.pb.go services/api/proto/scheduler_grpc.pb.go
# PASS in a temporary LF-normalized export; generated files were unchanged

# PASS: Services CI
https://github.com/shudorcl/AgentENV/actions/runs/30477945881

# PASS: AgentENV integration, single-node E2E, and docker-compose E2E
https://github.com/shudorcl/AgentENV/actions/runs/30477945814

Local make -C services fmt-check reports unrelated CRLF files because this WSL checkout uses core.autocrlf=true; the same commit passes fmt-check in Services CI.

Skipped checks and reasons:

• Root Rust formatting, Clippy, and unit-test targets were not run because this PR changes only the Go control-plane module and its generated Go protobuf.
• No benchmark was run because this PR does not claim a measured performance improvement; group keys and retained state are explicitly bounded.

Risks and reviewer notes

• Grouping uses exact, potentially mutable image or template references and is therefore an affinity heuristic rather than proof of cache availability.
• Group limits control affinity, not request admission. An individually oversized request is still placed once and immediately closes its group; hard node eligibility remains governed by the existing resource filter.
• Accounting happens before sandbox creation completes. A downstream creation failure can consume group budget and cause an earlier rotation, but cannot cause an open group to exceed its configured budget.
• Group state is process-local and protected by one mutex. The 10,000-key LRU bound limits memory growth, but scheduling through this strategy is intentionally serialized.
• Reviewers should start with services/scheduler/internal/strategy.go, services/gateway/internal/schedule_hint.go, services/scheduler/internal/node_registry.go, and services/shared/config/config.go.

Checklist

• The PR contains one coherent change and no unrelated formatting or refactoring.
• New behavior is covered by tests, or I explained why testing is impractical.
• Logs and examples contain no credentials, tokens, or private registry information.
• I did not manually edit generated code without updating its source and regenerating it.

[github-actions]

✅ OpenCodeReview: Review complete: 0 finding(s) across 7 selected item(s).

[shudorcl]

@blahgeek @LSX-s-Software This PR has been open for a week. CI is passing and the PR is mergeable, but it hasn’t received a review yet. Could you please take a look when you have a spare moment? If you’re unavailable, would you mind suggesting an alternative reviewer? Thanks!

[yingdi-shan]

Thanks for putting this together, and sorry for the delayed response.

Our main gap is the lack of a simulation framework for evaluating scheduling strategies against realistic workloads. We plan to collect representative workload traces and replay them against different strategies, which will let us compare CPU/memory balance, cache hit ratio, cold starts across workload patterns.

This does not need to be part of this PR. We can leave this strategy open and merge it once the simulation framework is in place.

[shudorcl]

@yingdi-shan

Thanks for the context, and sorry for the delayed follow-up. Is anyone already working on or designing the simulation framework?
If not, I’d be interested in taking it on. Since the requirements are still fairly high-level, I can first open a design/tracking issue to clarify the workload trace format, simulation model, metric definitions, and acceptance criteria before starting the implementation.
In particular, it would be helpful to confirm whether representative traces already exist, and whether the expected first version should be a lightweight deterministic simulator or an end-to-end replay against running services.