Fix(escalation) close session id rotation bypass of alert rate limiting - #71
Conversation
Added a global alert rate limiting mechanism to prevent abuse by rotating session IDs. This includes a new function to check global rate limits and integrates it into the existing alert logging process.
Added regression tests for global rate limiting to ensure that rotating session IDs do not bypass alert limits and that global limits function correctly under various conditions.
Added global rate limit configuration for alerts.
|
Warning Review limit reached
Next review available in: 58 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughAdds a global, cross-session alert rate-limit backstop to the escalation router, configured via new config.yaml fields, enforced alongside the existing per-session limiter with refined suppression reasons/logging, and validated by a new test suite covering bypass prevention and window expiration. ChangesGlobal Alert Rate Limit Backstop
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant EscalationRouter
participant PerSessionLimiter
participant GlobalLimiter
Caller->>EscalationRouter: escalate(session_id, event)
EscalationRouter->>PerSessionLimiter: check_rate_limit(session_id)
PerSessionLimiter-->>EscalationRouter: allowed/denied
alt session allowed
EscalationRouter->>GlobalLimiter: _global_rate_limit_allows()
GlobalLimiter-->>EscalationRouter: allowed/denied
end
EscalationRouter-->>Caller: alerts_allowed + reason (per-session/GLOBAL/none)
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/test_router.py (1)
137-159: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTest couples
global_maxto livesession_cap— fragile ifrate_limit_maxchanges.The test hardcodes
global_max=5but readssession_capfrom the live store singleton. The firstsession_capcalls consume both per-session and global slots, so the test only passes whensession_cap < 5. Ifrate_limit_maxinconfig.yamlis changed to ≥ 5, those initial calls exhaust the global limiter and the finalfresh["alerted"] is Trueassertion fails.Setting
global_maxrelative tosession_capeliminates the coupling.♻️ Proposed fix
- with patch.object(router_mod, "get_config", return_value=self._cfg(global_max=5)): + with patch.object(router_mod, "get_config", return_value=self._cfg(global_max=session_cap + 5)):🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_router.py` around lines 137 - 159, The test in test_session_limited_event_does_not_consume_global_slot is fragile because it hardcodes get_config(... global_max=5) while using the live session_cap from get_store()._rate_limit_max, so the first loop can exhaust the global limiter when session_cap changes. Update the test setup to derive global_max from session_cap (or otherwise keep it safely above the per-session cap) before calling escalate, so the fresh-session assertion remains valid regardless of config changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/test_router.py`:
- Around line 137-159: The test in
test_session_limited_event_does_not_consume_global_slot is fragile because it
hardcodes get_config(... global_max=5) while using the live session_cap from
get_store()._rate_limit_max, so the first loop can exhaust the global limiter
when session_cap changes. Update the test setup to derive global_max from
session_cap (or otherwise keep it safely above the per-session cap) before
calling escalate, so the fresh-session assertion remains valid regardless of
config changes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 3ac3ff5d-4620-4e13-8b86-c181b1c77dc2
📒 Files selected for processing (3)
humane_proxy/config.yamlhumane_proxy/escalation/router.pytests/test_router.py
Refactor alert rate limiting logic to include per-IP and global ceiling checks. Update comments for clarity on rate limiting mechanisms.
Added pytest fixtures to reset rate limiters between tests and refactored test methods to use the new fixture. Improved test coverage for rate limiting behavior with respect to session IDs and client IPs.
Added per-IP alert rate-limiting configuration.
Vishisht16
left a comment
There was a problem hiding this comment.
Good job with the work and thanks for the contribution
What
Adds a global, non-session-keyed rate-limit backstop for operator alerts
(Slack/Discord/Teams/PagerDuty/email), alongside the existing per-session
limiter.
Why
check_rate_limit()is keyed entirely onsession_id, which iscaller-supplied and unauthenticated —
middleware/interceptor.pyreads itstraight off the request body with no validation. That means the
per-session quota (
escalation.rate_limit_max, default 3/hour) resets forevery new
session_id, so an attacker can rotate the ID on every requestand trigger unlimited operator pages/notifications, even though no single
session ever exceeds its own quota.
For a tool whose entire purpose is alerting a human when someone's in
crisis, unlimited false pages is a real availability/trust problem — it
buries real alerts under noise (alert fatigue on the on-call channel).
How
_global_rate_limit_allows()inescalation/router.py: in-processsliding-window counter (deque of timestamps + lock), independent of
session_id.escalate()now requires both the existing per-session check AND thenew global check before firing alerts. Short-circuits so a
session-limited event doesn't also burn a global slot.
escalation:—global_rate_limit_max(default100) and
global_rate_limit_window_seconds(default 60). Setglobal_rate_limit_max: 0to disable.reason: "logged_alerts_globally_rate_limited"in the result dictso it's distinguishable from the existing per-session
"logged_alerts_rate_limited"in logs/tests.of either rate limit, per the existing design intent (only alerting
is throttled).
Known limitation
This backstop is in-process, so it's per-worker in a multi-process
deployment (gunicorn/uvicorn with multiple workers), not a hard global
cap across the whole fleet. It still closes the exploit for the default
single-process deployment. A Redis-backed version would be needed for a
true cross-process ceiling — happy to open that as a follow-up if wanted.
Testing
Added
TestGlobalRateLimitBackstopintests/test_router.py:session_idno longer yields unlimited alertsglobal_rate_limit_max: 0disables the backstoppytest -q→ 421 passed, 16 skipped, 0 failed.ruff checkclean.