Handle spending-limit errors gracefully with escalating cooldowns

[slacki-ai]

Summary

• Spending-limit errors (e.g. "Failed to start GPU" due to RunPod hourly spend cap) now trigger a 5-minute global provisioning pause instead of penalising individual GPU types. Jobs stay pending and retry automatically after the pause.
• Default cooldown reduced from 7 days to 1 hour for non-spending-limit hardware failures.
• Escalating cooldowns within the same UTC day: 1h → 6h → 2 days for repeated cooldowns on the same GPU type. Resets daily.

How it works

1. is_spending_limit_error() detects spending-limit patterns in RunPod error messages
2. On match, record_failure() sets a global pause timestamp instead of counting toward the per-hardware failure threshold
3. scale_workers() checks is_spending_limit_paused() at the top and returns early (with a log warning), keeping all jobs in pending state
4. For non-spending-limit failures, the existing threshold logic applies but uses an escalating cooldown schedule tracked per calendar day (UTC)

Configuration

All new values are env-configurable:

• OW_RUNPOD_SPENDING_LIMIT_PAUSE_SECONDS — global pause duration (default: 300s)
• OW_RUNPOD_HARDWARE_COOLDOWN_SECONDS — base cooldown (default: 3600s, down from 604800s)

Test plan

• 8 unit tests pass (3 existing + 5 new covering spending-limit detection, global pause, escalation, and daily reset)
• Deploy to staging and verify jobs stay pending when spending limit is hit
• Verify escalating cooldowns work by triggering repeated hardware failures

🤖 Generated with Claude Code