Skip to content

Add instance expiration controls - #434

Open
sjmiller609 wants to merge 1 commit into
mainfrom
hypeship/instance-ttl-reaper
Open

Add instance expiration controls#434
sjmiller609 wants to merge 1 commit into
mainfrom
hypeship/instance-ttl-reaper

Conversation

@sjmiller609

@sjmiller609 sjmiller609 commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • accept either a relative ttl or absolute expires_at when creating or updating an instance
  • treat update TTLs as relative to commit time and allow ttl: "0s" to disable expiration
  • persist and return the canonical expires_at deadline, returning null when disabled
  • serialize creation and expiration updates with reaper deletion so successful operations cannot race stale deletion decisions
  • bound each reaper deletion to 30 seconds and skip busy lifecycle locks until the next sweep
  • sweep once at startup, then once per minute
  • return 409 instance_expired once the current deadline has passed and 404 if deletion already completed
  • clear expiration metadata from snapshots and forked instances

Instances without an expiration are unchanged. The reaper uses the normal instance deletion path.

Testing

  • targeted API tests for relative TTL, absolute expiration, disabling, validation, consistent error mapping, and explicit null responses
  • targeted manager tests for persistence, deadline boundaries, startup sweeping, delete timeout continuation, and race ordering
  • snapshot and instance-fork expiration clearing tests
  • targeted race-detector tests for reaper lifecycle coordination
  • go test ./... -run '^$'
  • make oapi-generate (verified deterministic output)

The complete API suite was also attempted, but environment-dependent tests require mkfs.erofs and network bridge permissions unavailable locally. The targeted tests and repository-wide compile pass.


Note

Medium Risk
Adds automatic instance deletion and new create/update API fields, including lock races between the reaper and lifecycle operations. Incorrect expiration handling could delete live workloads.

Overview
Adds optional instance expiration: create and update accept mutually exclusive ttl (Go duration) or expires_at. Responses always include canonical expires_at (null when disabled). Update TTLs are relative to commit time; ttl: "0s" clears expiration.

A host TTL reaper sweeps about once a minute and deletes expired instances through the normal delete path. It uses per-instance locks so in-flight create/update wins over a stale reap, times out slow deletes, and records hypeman_instances_ttl_reaper_deletions_total. Past-deadline updates return 409 instance_expired; already-reaped instances return 404.

Forks and snapshot records drop expiration; restoring a snapshot keeps the current instance deadline.

Reviewed by Cursor Bugbot for commit 0336f8b. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown
-->

✱ stlc build

go code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

116 files generated at fff6296 (pushed)

go get github.com/kernel/hypeman-go-staging@fff62962588943aaf97919319f01d6d35fb907c1
python code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

230 files generated at f5d798b (pushed)

typescript code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

138 files generated at 67a02f7 (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note
LevelCodeMessageTargets
Build metadata
Buildbd_76BDJKvy-jagged-resin
Timestamp2026-08-20T15:13:54.770Z
stlc8413509
Spec hashb79c3a21fc1f
Config hash55e15f6f4434

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-08-20 15:14:19 UTC

@sjmiller609
sjmiller609 marked this pull request as ready for review August 20, 2026 14:11

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 843a3cd. Configure here.

Comment thread lib/instances/ttl_reaper.go
@sjmiller609
sjmiller609 force-pushed the hypeship/instance-ttl-reaper branch from 843a3cd to 469a295 Compare August 20, 2026 14:51
@sjmiller609 sjmiller609 changed the title Add instance TTL expiration Add instance expiration controls Aug 20, 2026
@sjmiller609
sjmiller609 force-pushed the hypeship/instance-ttl-reaper branch from 469a295 to 0336f8b Compare August 20, 2026 15:10
@sjmiller609
sjmiller609 requested a review from chruffins August 20, 2026 15:54
@chruffins

Copy link
Copy Markdown
Contributor

reviewed — one correctness issue and three follow-ups worth addressing.

Bugs

  • lib/instances/update.go:56-65,111 — TTL deadlines are calculated before environment/proxy updates and metadata persistence complete. A slow update can commit an already-expired deadline, violating the API contract that TTL starts when the update is committed. Calculate the relative deadline at the final metadata commit point.

  • lib/instances/ttl_reaper.go:63-66 — the 30-second delete timeout is not a hard bound. The normal delete path calls context-insensitive operations such as killHypervisor and deleteInstanceData/os.RemoveAll, so a stuck delete can block this sequential sweep and delay all later expirations.

Structural / Maintainability

  • cmd/api/api/instances.go:137-152, lib/instances/create.go:624-628 — expiration validation is split between the API and domain layers and already differs for ttl: "0s" plus expires_at: HTTP rejects any combination, while the domain request accepts zero TTL and uses the absolute deadline. Make TTL presence explicit in the domain request or centralize the policy so callers cannot observe different rules.

Questions

  • lib/instances/fork.go:281, lib/instances/snapshot.go:302,426 — is the differing expiration behavior intentional? Instance forks and snapshot forks clear expiration, while restoring a snapshot preserves the current instance deadline. The public API does not document these inheritance rules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants