Smart retry/simulator rl - #11
Open
JanStasz wants to merge 31 commits into
Open
Conversation
Add LiveMetricsBuffer that collects per-attempt events in a sliding window and computes aggregated metrics (success rate, latency percentiles, retry ratio, queue depth). Integrates into ServiceRuntime with opt-in activation via enable_live_buffer() to avoid overhead when not using RL.
Implement RetrySimEnv (gym.Env) that wraps the discrete-event simulator. The agent observes system metrics and adjusts retry parameters (max_attempts, backoff delay, budget ratio) at fixed intervals. Supports seeded episodes for reproducible comparisons.
…ripts Add train_rl_test.py for PPO training with Stable-Baselines3 and compare_rl_vs_baseline.py for evaluating learned policy against a static baseline on the same scenario.
…tion features Extend RandomScenarioSimEnv observation from 13 to 18 dimensions by appending 3 delta-trend features (success_rate, error_rate, retry_ratio) and 2 normalised current-action features (refill_rate_idx, capacity_idx). Improve the reward signal to include delta-success shaping and an action-change penalty for smoother policy behaviour. Adjust the discrete action maps (120→90, 100→80) for tighter exploration range. Add token_bucket_indices_from_physical() helper for mapping physical token-bucket params back to the nearest discrete action index.
…dirs Each training run now writes model weights, VecNormalize stats, eval logs, TensorBoard logs, and plots under simulator/trained_models/run_*/ instead of scattered top-level files. Add NormSyncCallback to keep eval env normalisation in sync with the training env during evaluation callbacks. Support --run-dir flag for evaluating a specific historical run, and --skip-training to evaluate the latest run without retraining.
…driving Replace the RetrySimEnv wrapper approach with direct ConfigLoader-based simulation driving for no-budget, static, and RL evaluation variants. This enables richer per-client metrics and correct budget manipulation. Add comprehensive comparison metrics: per-client success rates, load amplification, retry efficiency, fairness analysis, time-to-recover, and latency percentiles (p50/p95/p99).
Add analyze_policy_first_layer.py to compute mean |weight| per input dimension on the PPO actor's first layer, highlighting which observation features the policy attends to most. Add analyze_rollout_correlation.py to collect (obs, reward, action) over many episodes and compute Pearson/Spearman correlations plus mutual information between each feature and the reward signal. Update .gitignore to cover simulator/trained_models/, best_model/, and eval_logs/ directories.
Replace raw traffic-count observations with normalized health, pressure, and trend features, and fix retry-budget accounting edge cases that made the control loop noisy.
Add a smoother RL control variant that adjusts refill rate and bucket capacity by bounded index deltas instead of jumping directly across the action grid.
Introduce multi-client metastable retry-budget environments and benchmark scenarios that measure aggregate success, client fairness, retry efficiency, and recovery behavior.
Expand evaluation to compare no budget, YAML static, best static, and RL policies across metastable scenarios with reusable metrics and artifacts.
Add a three-knob controller that combines time-based refill with event-based retry capacity earned from successful attempts.
Replace client-centric reward with server-side signals (retry storm, budget saturation, action stability) and decouple decision cadence from observation/delta metric windows across train, eval, and benchmarks.
… rate Record caller-side retry admissions and rejections on IstioRetryBudget and log outcomes when client-managed retries hit the server budget gate. This supports budget_reject_rate in the RL observation vector.
…with prototype
Expand the retry-budget action grid to MultiDiscrete([6, 6]) with percent
{0,5,10,20,35,50} and minRetryConcurrency {0,1,2,3,5,8}, keeping (20, 3)
as the default baseline index.
Re-source all caller-derivable observation features from client attempt
history instead of the server live buffer: per-attempt success, retry
ratio, failure rates, p95 latency pressure, and fairness from observed
first-attempt share. Replace queue_utilization with budget_reject_rate.
Update reward, plotting ticks, and unit tests accordingly.
Requires retraining existing PPO models and shipping a new
vecnormalize_stats.pkl.
…n spaces Add a reference for the IstioRetryBudgetMetastableEnv controller: the 36-action MultiDiscrete([6, 6]) grid over retryBudget.percent and minRetryConcurrency, the 18-feature caller-side per-attempt observation vector, the decision/observation/delta timing parameters, and the compatibility notes for older checkpoints and VecNormalize stats.
Add an Istio retry-budget benchmark scenario whose YAML budget is intentionally too restrictive (0% / 0), giving the adaptive controller room to reopen retries when they are useful. Drive the static-budget baseline from each scenario's YAML instead of a hardcoded (20%, 3) so benchmarks report the budget actually configured per scenario.
… plotting Add tooling to interpret the trained controller: - analyze_istio_retry_budget_signals.py replays the model in the eval environment and combines model-internal feature importance with inference-time counterfactual ablations. - plot_istio_focused_benchmark.py plots success rate, load amplification, and RL actions over time for one benchmark scenario. - plot_top_signal_action_correlations.py plots action correlations for the top external telemetry signals per fault phase.
Organize the RL scripts, docs, optional dependencies, and shipped RB-RL.v5 artifacts so the DIRB controller can be trained, evaluated, and benchmarked reproducibly.
Adapt the RL retry-budget extension (DIRB / RB-RL.v5) to main's refactored simulator architecture: - ServiceRuntime split into mixins: re-wire DIRB runtime hooks (retry concurrency counters, live metrics buffer, Istio runtime-state refresh, admission recording, runtime policy controls) into service.py + service_attempts.py + service_middleware.py without regressing main's static _events/metrics semantics. - Config builders: add Istio retry-budget building to config/builders and keep ConfigLoader wrappers; schema keeps istio_retry_budget alongside main's arolla_retry_budget. - Policies: budgets moved to retry_controls.py; update RL imports and add IstioRetryBudget.applies_pre_queue_admission + **kwargs add_result for the new LoadLimiterMiddleware contract. - Tests: update RL control tests to the merged admission/live-buffer APIs.
Update the _phase_flags method to ensure that only "Partial Failure" fault windows are considered active. This change prevents load spikes from incorrectly influencing the post-fault recovery window used in the reward function, enhancing the accuracy of the simulation environment.
Keep normal YAML simulation runs behavior-identical to pre-RL by only honoring external_is_retry when an RL env has enabled the live buffer or an Istio retry budget is configured. Otherwise retry admission falls back to the original attempt > 1 classification.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(simulator): DIRB — RL controller for adaptive Istio retry-budget control
Summary
This branch adds DIRB (Dynamic Istio Retry Budget), a reinforcement-learning
extension to the discrete-event simulator that learns to tune Istio server-side
retry-budget parameters online and recover microservice systems from metastable
retry-amplification failures.
The extension is additive and fully opt-in: it adds an RL layer (metrics,
environments, training/eval/benchmark tooling, docs, and a trained model) on top
of the existing simulator. All RL machinery is dormant for normal YAML
simulation runs — non-RL experiments behave bit-for-bit identically to main.
The full project report (Smart Retry, PDF) is attached to this PR for context.
~94 files changed, +14,147 / -14 over 30 commits (merge-base 4d85579).
Motivation
Client-side retries defend against transient failures, but under partial overload
they add work to an already-saturated service and can push it into a metastable
state where goodput stays collapsed long after the original fault clears. Istio
exposes a server-side retry budget (percent, minRetryConcurrency) that caps active
retries, but in practice it is static: one setting must cover both transient
failures and real overload. This work asks whether an RL agent can dynamically
tune that budget from deployable mesh telemetry to preserve useful retries while
suppressing amplification during overload.
What's included
Simulator core (mutable Istio retry budget + caller-side telemetry):
policies/istio_retry_budget.py: Istio-style concurrency retry-budget limiter(l = max(m, b/100 * (n_active + n_pending))).
config/schema.py,config/loader.py,config/builders/: config model andYAML loader support for the Istio retry budget.
runtime/service.py:update_istio_retry_budget()(andupdate_token_bucket())so an RL environment can patch the budget mid-run, plus live-buffer telemetry
and retry-concurrency counters.
runtime/service_attempts.py,service_middleware.py: retry-admission trackingso budget reject rate becomes an observable signal.
metrics/live_buffer.py:LiveMetricsBuffersliding-window view of recentper-attempt metrics for online observation.
RL environments (
src/simulator/rl/):istio_retry_budget_env.py: the finalIstioRetryBudgetMetastableEnv— a6x6 MultiDiscrete action grid over (percent, minRetryConcurrency) and an 18-D
caller-side, per-attempt observation vector.
env.py,random_scenario_env.py,relative_action_env.py,metastable_relative_action_env.py,metastable_fairness_env.py,hybrid_metastable_env.py,hybrid_retry_budget.py.Training / evaluation / benchmark / analysis scripts (
bin/rl/):hybrid variants.
models.
analysis, and benchmark plotting.
Experiments, model, and docs:
experiments/yaml/rl/: token-bucket and Istio metastable/benchmark scenarios(load spike, partial failure, compound failure, switchback, fairness).
models/RB-RL.v5/: frozen PPO checkpoint, VecNormalize stats, training spec,and signal-analysis / benchmark results.
docs/rl/: RL README, action/observation space spec, and model docs.requirements-rl.txt,setup.py, and.gitignoreupdates for RL artifacts.Tests (
tests/unit/):test_istio_retry_budget_rl.py,test_metastable_env.py,test_rl_env_controls.pyverify the limiter, environment, and that RL actionsactually move the runtime budget. Full suite: 107 passing.
Design notes
per-request retry/backoff/drop decisions. It periodically (every 5 s) patches
budget parameters rather than sitting in the hot request path.
m in {0, 1, 2, 3, 5, 8}. Static baseline = (20%, 3) at index (3, 3).
level window, 5 s delta window), chosen so the same features can be scraped
from Envoy in the live mesh.
and recovery, penalizes retry amplification, overload, failures, budget
pressure, and unstable/oscillating actions.
the current 6x6 environment; RB-RL.v5 was retrained from scratch.
Behavior isolation for normal runs
The RL layer does not change normal (non-RL) simulation behavior:
calls
enable_live_buffer(); all_record_live_metricscalls are no-opsotherwise.
isinstance(limiter, IstioRetryBudget)check.update_*runtime controls are only invoked by RL envs._rl_retry_tracking_active(true only when the live buffer is enabled or anIstio budget is configured). For normal runs, retry classification falls back
to the original
attempt > 1rule, so every pre-existing load-limiteradmission decision is unchanged.
reported metrics (success rate, retry amplification, latency, etc.) for non-RL
runs are identical to main.
Caveats
some simulator/deployment mismatch remains.
a stricter "2 of 3 runs" criterion, recovery coverage drops); robustness is
future work.
Smart Retry Report.pdf