Spotter

A Kubernetes controller that detects spot instance interruptions on GitHub Actions self-hosted runners and automatically retries failed workflow runs.

The Problem

When running GitHub Actions on self-hosted runners via ARC (Actions Runner Controller) on spot/preemptible instances, cloud providers can reclaim the instance at any time. This kills the runner pod mid-job, GitHub marks the job as failed, and there is no built-in retry mechanism.

How It Works

Spotter runs two controllers:

EventWatcher listens for Kubernetes Events emitted by node termination handlers (e.g., AWS NTH, Karpenter). When a spot reclaim is detected, the handler emits events like PreDrain or CordonAndDrain. Spotter caches the affected node.

EphemeralRunnerWatcher watches ARC EphemeralRunner CRs and their owned pods. It proactively caches runner-to-node mappings as pods get scheduled. When a CR is deleted (runner killed), it checks whether the node was spot-interrupted - either from the event cache or by detecting the node is gone/cordoned - and calls the GitHub API to rerun the workflow.

Features

• Automatic retry of failed workflows caused by spot interruptions
• Configurable rerun strategy - rerun only failed jobs or the entire workflow, per repo
• Cloud-agnostic - works with AWS NTH, GCP, and Azure termination handlers via configuration
• Fallback detection - detects spot kills via node state (gone/cordoned) when termination handler events are unavailable
• Notifications via Shoutrrr (Slack, Discord, Teams, and more)
• GitHub App and PAT authentication with automatic token refresh for App auth
• Rate limiting - configurable QPS and burst for GitHub API calls
• GHES support - configurable GitHub API base URL
• Prometheus metrics - spot detections, retry counts, API errors, reconcile duration
• Non-retryable error handling - gracefully handles "workflow already running" and similar GitHub API errors
• ConfigMap-backed retry state - survives pod restarts
• Dry-run mode - log what would be retried without calling the GitHub API
• Multi-arch Docker images (amd64/arm64)
• Helm chart with leader election for HA

Prerequisites

• Kubernetes cluster with ARC scale sets (v0.14+)
• A GitHub PAT or GitHub App with actions:write permission on the target repositories
• (Optional) A node termination handler that emits Kubernetes Events (e.g., AWS NTH with EMIT_KUBERNETES_EVENTS=true, or Karpenter with --interruption-queue). Spotter also works without one via node state fallback detection.

Installation

Helm

helm install spotter charts/spotter \
  --namespace kube-system \
  --set github.existingSecret=my-github-secret

With a GitHub token

kubectl create secret generic github-token \
  --namespace kube-system \
  --from-literal=github-token=ghp_your_token_here

helm install spotter charts/spotter \
  --namespace kube-system \
  --set github.existingSecret=github-token

With GitHub App auth

kubectl create secret generic github-app-key \
  --namespace kube-system \
  --from-file=github-app-key.pem=/path/to/private-key.pem

helm install spotter charts/spotter \
  --namespace kube-system \
  --set github.authType=app \
  --set github.appId=12345 \
  --set github.appInstallationId=67890 \
  --set github.existingAppSecret=github-app-key

Configuration

All configuration is done through Helm values or command-line flags.

Retry Behavior

Value	Default	Description
config.maxRetries	3	Max retries per workflow run
config.nodeCacheTTL	30m	How long to remember interrupted nodes
config.retryStateTTL	24h	How long to keep retry state records
config.rerunStrategy	failed	failed to rerun only failed jobs, all to rerun the entire workflow
config.rerunAllRepos	[]	Repos that always use the all strategy (e.g., ["myorg/repo1"])
config.dryRun	false	Log without calling the GitHub API
config.logLevel	info	Log level: debug, info, warn, error

Event Detection

Value	Default	Description
config.eventSources	["aws-node-termination-handler"]	K8s Event source names to match
config.eventReasons	["SpotInterruption", "SQSTermination", "PreDrain", "CordonAndDrain"]	Event reasons to match
config.eventMessagePatterns	["Spot Interruption"]	Message substrings for ambiguous reasons
config.watchNamespace	"" (all)	Namespace to watch

GitHub API

Value	Default	Description
github.authType	pat	pat or app
github.apiUrl	https://api.github.com	GitHub API base URL (for GHES)
github.apiQps	10	Rate limit (requests/sec)
github.apiBurst	20	Rate limit burst

Notifications

Value	Default	Description
notify.url	""	Shoutrrr notification URL
notify.existingSecret	""	Existing secret containing the notification URL
notify.existingSecretKey	notify-url	Key in the existing secret

Cloud Provider Examples

AWS with NTH (SQS mode):

config:
  eventSources: ["aws-node-termination-handler"]
  eventReasons: ["PreDrain", "CordonAndDrain"]

AWS with NTH (IMDS mode):

config:
  eventSources: ["aws-node-termination-handler"]
  eventReasons: ["SpotInterruption"]

AWS with Karpenter (native interruption handling):

config:
  eventSources: ["karpenter"]
  eventReasons: ["Disrupted", "DisruptionTerminating"]

Karpenter v0.19+ supports native spot interruption handling via EventBridge/SQS (requires --interruption-queue). If using Karpenter without this flag, Spotter's node state fallback (node gone/cordoned) detects interruptions automatically - no event source configuration needed.

GCP:

config:
  eventSources: ["gcp-node-termination-handler"]
  eventReasons: ["PreemptibleTermination"]

Azure:

config:
  eventSources: ["azure-scheduled-events"]
  eventReasons: ["SpotEviction"]

No termination handler (fallback only):

Spotter can detect spot interruptions without any termination handler by checking if the node is gone or cordoned. This works with any cloud provider or node management tool (Karpenter, Cluster Autoscaler, etc.). No event source configuration is needed - the fallback is always active.

Notifications

Spotter uses Shoutrrr for notifications. Pass any supported URL:

# Slack
--set notify.url='slack://token@channel'

# Discord
--set notify.url='discord://token@webhook-id'

# Microsoft Teams
--set notify.url='teams://group@tenant/altId/groupOwner?host=organization.webhook.office.com'

Or use an existing secret:

--set notify.existingSecret=my-notify-secret

Metrics

Spotter exposes Prometheus metrics on the configured metrics port (default :8080):

Metric	Type	Labels	Description
spotter_spot_interruptions_detected_total	Counter	node, reason, source	Spot interruptions detected
spotter_workflow_retries_total	Counter	owner, repo, reason	Workflow retries triggered
spotter_workflow_retries_exhausted_total	Counter	owner, repo	Retries exhausted (max reached)
spotter_retry_api_errors_total	Counter	owner, repo, status_code	GitHub API errors
spotter_github_api_calls_total	Counter	operation	Total GitHub API calls
spotter_reconcile_duration_seconds	Histogram	controller	Reconcile loop duration
spotter_reconcile_errors_total	Counter	reason	Reconcile errors by category
spotter_node_cache_size	Gauge		Nodes in the interruption cache

Enable the ServiceMonitor for Prometheus Operator:

metrics:
  serviceMonitor:
    enabled: true

License

Apache License 2.0 - see LICENSE for details.