Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,27 @@ may break public API until 1.0.0 ships.

## [Unreleased]

### Changed

- **Breaking** — `RetryPolicy`'s canonical constructor gains a fourth `Duration`, `maxTotalDelay`,
after `maxDelay` (seven components, was six). Migration: prefer the new `RetryPolicy.builder()`
(starts from `defaults()`, one setter per knob), or insert `Duration.ofMinutes(1)` — the default —
as the fourth positional argument. Deconstruction patterns over the record gain a component too.
Pre-1.0 per [ADR-019](docs/adr/019-pre-10-stability-policy.md); see
[ADR-027](docs/adr/027-retry-policy-builder-and-budget.md).

### Added

- **`fanar-core`** — a total sleep budget for retries ([ADR-027](docs/adr/027-retry-policy-builder-and-budget.md)):
`RetryPolicy.maxTotalDelay()` (default 1 min — the worst case the other defaults already allowed, so
nothing changes at the defaults) bounds the sum of all sleeps within one call; a sleep that would exceed
it — computed back-off or honoured `Retry-After` hint — is never started, retrying ends and the exception
surfaces with the hint preserved, like the ADR-025 ceiling. Plus `RetryPolicy.builder()` and
`withMaxTotalDelay(...)`.
- **`fanar-spring-boot-4-starter`** — `fanar.retry.max-total-delay` (default `1m`); the `RetryPolicy` bean
is built through the builder, so a `max-delay` raised above the budget fails the context at startup
([ADR-020](docs/adr/020-spring-boot-4-starter.md), amended).

- **`fanar-core`** — rate-limit visibility ([ADR-026](docs/adr/026-rate-limit-telemetry.md)): the retry boundary
publishes `fanar.ratelimit.limit` / `.remaining` / `.reset` / `.policy` observation attributes from every response
that carries Fanar's rate-limit headers (successes and 429s alike; the last attempt's values win), and both HTTP
Expand Down
158 changes: 142 additions & 16 deletions core/src/main/java/qa/fanar/core/RetryPolicy.java
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,25 @@
* Retry configuration for the SDK's built-in retry interceptor.
*
* <p>Immutable and thread-safe. Construct via {@link #defaults()} or {@link #disabled()} for the
* canonical presets, or via the record constructor for full control. Derive variants through the
* {@code with*} methods — each returns a new record.</p>
* canonical presets, via {@link #builder()} (which starts from the defaults) for full control, or
* via the record constructor. Derive variants through the {@code with*} methods — each returns a
* new record. The builder is the stable way to set knobs: the canonical constructor changes
* arity whenever a knob is added (ADR-027).</p>
*
* <p>This type holds only the <em>configuration</em>. The retry loop — applying backoff, honoring
* {@code Retry-After}, computing jitter — lives in the SDK's internal retry interceptor.</p>
*
* <h2>Default policy</h2>
* <p>{@link #defaults()} returns 3 attempts, exponential backoff with
* {@link JitterStrategy#FULL full jitter}, base 500&nbsp;ms, cap 30&nbsp;s, multiplier 2.0, and
* the {@link #isDefaultRetryable default retryable predicate} (transient server-side errors and
* transport failures — never client-side or content-filter errors). The cap also bounds
* honoured server {@code Retry-After} hints: a hint above it ends retrying and the exception
* surfaces immediately with the hint preserved (ADR-025).</p>
* {@link JitterStrategy#FULL full jitter}, base 500&nbsp;ms, cap 30&nbsp;s, a total sleep budget of
* 1&nbsp;min, multiplier 2.0, and the {@link #isDefaultRetryable default retryable predicate}
* (transient server-side errors and transport failures — never client-side or content-filter
* errors). The cap also bounds honoured server {@code Retry-After} hints: a hint above it ends
* retrying and the exception surfaces immediately with the hint preserved (ADR-025). The budget
* bounds the <em>sum</em> of all sleeps within one call: a sleep that would push the total over it
* is never taken — retrying ends and the exception surfaces, hint preserved (ADR-027). At the
* defaults the budget equals the worst case the other knobs allow (two sleeps of at most 30&nbsp;s),
* so it only bites once {@code maxAttempts} or {@code maxDelay} are raised.</p>
*
* <h2>Validation</h2>
* <p>The canonical constructor validates all invariants at construction time. {@code with*}
Expand All @@ -34,19 +40,26 @@
* {@code Retry-After} hints — a hint above it ends retrying and the
* exception surfaces with the hint preserved (ADR-025); must be
* positive, ≥ {@code baseDelay}, and representable in milliseconds
* @param maxTotalDelay budget for the sum of all sleeps within one call — the next sleep
* (computed back-off or honoured hint) is taken only if the total stays
* within it, otherwise retrying ends and the exception surfaces with
* the hint preserved (ADR-027); must be positive, ≥ {@code maxDelay},
* and representable in milliseconds
* @param backoffMultiplier factor applied to the backoff on each retry; must be ≥ 1.0
* @param jitter jitter policy applied to the computed backoff
* @param retryable predicate deciding whether a given exception is worth retrying.
* Consulted only while the policy can still honour a retry: the
* attempt budget ({@code maxAttempts}) and the delay ceiling
* ({@code maxDelay}) end retrying regardless of its answer
* attempt budget ({@code maxAttempts}), the delay ceiling
* ({@code maxDelay}) and the total sleep budget ({@code maxTotalDelay})
* end retrying regardless of its answer
*
* @author Oussama Mahjoub
*/
public record RetryPolicy(
int maxAttempts,
Duration baseDelay,
Duration maxDelay,
Duration maxTotalDelay,
double backoffMultiplier,
JitterStrategy jitter,
Predicate<FanarException> retryable
Expand Down Expand Up @@ -78,6 +91,18 @@ public record RetryPolicy(
throw new IllegalArgumentException(
"maxDelay must be representable in milliseconds, got " + maxDelay);
}
Objects.requireNonNull(maxTotalDelay, "maxTotalDelay");
if (maxTotalDelay.isNegative() || maxTotalDelay.isZero()) {
throw new IllegalArgumentException("maxTotalDelay must be positive, got " + maxTotalDelay);
}
if (maxTotalDelay.compareTo(maxDelay) < 0) {
throw new IllegalArgumentException(
"maxTotalDelay (" + maxTotalDelay + ") must be >= maxDelay (" + maxDelay + ")");
}
if (maxTotalDelay.compareTo(MAX_REPRESENTABLE_DELAY) > 0) {
throw new IllegalArgumentException(
"maxTotalDelay must be representable in milliseconds, got " + maxTotalDelay);
}
if (backoffMultiplier < 1.0) {
throw new IllegalArgumentException(
"backoffMultiplier must be >= 1.0, got " + backoffMultiplier);
Expand All @@ -88,7 +113,7 @@ public record RetryPolicy(

/**
* The SDK's default retry policy: 3 attempts, exponential backoff with full jitter, base 500&nbsp;ms,
* cap 30&nbsp;s, multiplier 2.0, and the default retryable predicate.
* cap 30&nbsp;s, total sleep budget 1&nbsp;min, multiplier 2.0, and the default retryable predicate.
*
* @return a new policy instance with the documented defaults
*/
Expand All @@ -97,6 +122,7 @@ public static RetryPolicy defaults() {
3,
Duration.ofMillis(500),
Duration.ofSeconds(30),
Duration.ofMinutes(1),
2.0,
JitterStrategy.FULL,
RetryPolicy::isDefaultRetryable);
Expand All @@ -113,6 +139,7 @@ public static RetryPolicy disabled() {
1,
Duration.ofMillis(500),
Duration.ofSeconds(30),
Duration.ofMinutes(1),
2.0,
JitterStrategy.FULL,
RetryPolicy::isDefaultRetryable);
Expand Down Expand Up @@ -142,34 +169,133 @@ public static boolean isDefaultRetryable(FanarException e) {

/** @return a new policy with the given {@code maxAttempts}, all other fields unchanged */
public RetryPolicy withMaxAttempts(int maxAttempts) {
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, backoffMultiplier, jitter, retryable);
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, maxTotalDelay, backoffMultiplier, jitter, retryable);
}

/** @return a new policy with the given {@code baseDelay}, all other fields unchanged */
public RetryPolicy withBaseDelay(Duration baseDelay) {
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, backoffMultiplier, jitter, retryable);
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, maxTotalDelay, backoffMultiplier, jitter, retryable);
}

/** @return a new policy with the given {@code maxDelay}, all other fields unchanged */
public RetryPolicy withMaxDelay(Duration maxDelay) {
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, backoffMultiplier, jitter, retryable);
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, maxTotalDelay, backoffMultiplier, jitter, retryable);
}

/**
* @return a new policy with the given {@code maxTotalDelay}, all other fields unchanged
* @since 0.4.0
*/
public RetryPolicy withMaxTotalDelay(Duration maxTotalDelay) {
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, maxTotalDelay, backoffMultiplier, jitter, retryable);
}

/** @return a new policy with the given {@code backoffMultiplier}, all other fields unchanged */
public RetryPolicy withBackoffMultiplier(double backoffMultiplier) {
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, backoffMultiplier, jitter, retryable);
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, maxTotalDelay, backoffMultiplier, jitter, retryable);
}

/** @return a new policy with the given {@code jitter}, all other fields unchanged */
public RetryPolicy withJitter(JitterStrategy jitter) {
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, backoffMultiplier, jitter, retryable);
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, maxTotalDelay, backoffMultiplier, jitter, retryable);
}

/**
* @return a new policy with the given {@code retryable} predicate, all other fields unchanged;
* the attempt budget and the delay ceiling still apply regardless of its answer
*/
public RetryPolicy withRetryable(Predicate<FanarException> retryable) {
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, backoffMultiplier, jitter, retryable);
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, maxTotalDelay, backoffMultiplier, jitter, retryable);
}

/**
* Begin building a policy from {@link #defaults()}: every knob starts at its default and only
* the ones set change. Validation happens in {@link Builder#build()}, exactly as for the
* canonical constructor.
*
* @return a new builder
* @since 0.4.0
*/
public static Builder builder() {
return new Builder(defaults());
}

/**
* Fluent builder for {@link RetryPolicy} (ADR-027). Not thread-safe; build once.
*
* @since 0.4.0
*/
public static final class Builder {

private int maxAttempts;
private Duration baseDelay;
private Duration maxDelay;
private Duration maxTotalDelay;
private double backoffMultiplier;
private JitterStrategy jitter;
private Predicate<FanarException> retryable;

private Builder(RetryPolicy start) {
this.maxAttempts = start.maxAttempts();
this.baseDelay = start.baseDelay();
this.maxDelay = start.maxDelay();
this.maxTotalDelay = start.maxTotalDelay();
this.backoffMultiplier = start.backoffMultiplier();
this.jitter = start.jitter();
this.retryable = start.retryable();
}

/** @param maxAttempts total attempts including the first; {@code 1} disables retries */
public Builder maxAttempts(int maxAttempts) {
this.maxAttempts = maxAttempts;
return this;
}

/** @param baseDelay initial backoff delay */
public Builder baseDelay(Duration baseDelay) {
this.baseDelay = baseDelay;
return this;
}

/** @param maxDelay cap on one sleep and ceiling on honoured {@code Retry-After} hints (ADR-025) */
public Builder maxDelay(Duration maxDelay) {
this.maxDelay = maxDelay;
return this;
}

/** @param maxTotalDelay budget for the sum of all sleeps within one call (ADR-027) */
public Builder maxTotalDelay(Duration maxTotalDelay) {
this.maxTotalDelay = maxTotalDelay;
return this;
}

/** @param backoffMultiplier factor applied to the backoff on each retry */
public Builder backoffMultiplier(double backoffMultiplier) {
this.backoffMultiplier = backoffMultiplier;
return this;
}

/** @param jitter jitter policy applied to the computed backoff */
public Builder jitter(JitterStrategy jitter) {
this.jitter = jitter;
return this;
}

/** @param retryable predicate deciding whether an exception is worth retrying */
public Builder retryable(Predicate<FanarException> retryable) {
this.retryable = retryable;
return this;
}

/**
* Validate and build.
*
* @return the policy
* @throws IllegalArgumentException or {@link NullPointerException} on an invalid knob,
* exactly as the canonical constructor
*/
public RetryPolicy build() {
return new RetryPolicy(maxAttempts, baseDelay, maxDelay, maxTotalDelay, backoffMultiplier, jitter, retryable);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@
* sleep uses the hint instead of the computed back-off curve. A hint above {@code maxDelay}
* ends retrying immediately — no sleep, no burned attempt — and the exception surfaces with
* the hint preserved (ADR-025).</li>
* <li>Never sleeps past {@link RetryPolicy#maxTotalDelay()} in total for one call: when the next
* sleep — computed back-off or honoured hint — would push the cumulative sleep over the
* budget, retrying ends without sleeping and the exception surfaces with the hint preserved
* (ADR-027).</li>
* <li>Otherwise sleeps for {@code baseDelay * multiplier^(attempt-1)} capped at {@code maxDelay},
* with {@link JitterStrategy} applied (none / full / equal).</li>
* <li>Records {@link FanarObservationAttributes#HTTP_STATUS_CODE} for every response received
Expand Down Expand Up @@ -80,6 +84,7 @@ public RetryInterceptor(RetryPolicy policy) {
@Override
public HttpResponse<InputStream> intercept(HttpRequest request, Chain chain) {
int attempt = 0;
Duration slept = Duration.ZERO;
while (true) {
attempt++;
try {
Expand All @@ -97,11 +102,15 @@ public HttpResponse<InputStream> intercept(HttpRequest request, Chain chain) {
if (attempt >= policy.maxAttempts()
|| !policy.retryable().test(e)
|| (hint != null && hint.compareTo(policy.maxDelay()) > 0)) {
recordRetryCount(chain, attempt - 1);
throw e;
throw surfaced(chain, attempt, e);
}
Duration delay = hint != null ? hint : backoff(attempt);
if (slept.plus(delay).compareTo(policy.maxTotalDelay()) > 0) {
throw surfaced(chain, attempt, e);
}
chain.observation().event("retry_attempt");
sleepOrAbort(hint != null ? hint : backoff(attempt));
sleepOrAbort(delay);
slept = slept.plus(delay);
}
}
}
Expand Down Expand Up @@ -164,6 +173,12 @@ private static void recordRateLimit(Chain chain, HttpResponse<?> response) {
}
}

/** Retrying ends here: record the exit and hand the exception back to be thrown. */
private static FanarException surfaced(Chain chain, int attempt, FanarException e) {
recordRetryCount(chain, attempt - 1);
return e;
}

private static void recordRetryCount(Chain chain, int retries) {
chain.observation().attribute(FanarObservationAttributes.FANAR_RETRY_COUNT, retries);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,38 @@ void disabledPolicyStillMapsErrorsButNeverRetries() {
assertEquals(1, server.hits());
}

// --- total sleep budget (ADR-027) ---------------------------------------------------------

@Test
void retryAfterHintsBeyondTheTotalBudgetEndRetrying() {
// Two 429s each asking for 1 s: each within the 1 s ceiling, so ADR-025 alone would sleep
// both. A 1 s total budget admits the first hint (exactly the budget) and refuses the second
// — the exception surfaces after one retry, hint preserved, no further request is made.
server.enqueue(
Reply.of(429, "slow down", Map.of("Retry-After", "1")),
Reply.of(429, "slow down again", Map.of("Retry-After", "1")));
RecordingObservability obs = new RecordingObservability();
RetryPolicy budgeted = RetryPolicy.builder()
.maxDelay(Duration.ofSeconds(1))
.maxTotalDelay(Duration.ofSeconds(1))
.build();

long started = System.nanoTime();
FanarRateLimitException ex;
try (FanarClient client = client(budgeted, obs)) {
ex = assertThrows(FanarRateLimitException.class, () -> client.chat().send(ping()));
}
long elapsedMs = (System.nanoTime() - started) / 1_000_000L;

assertEquals(2, server.hits(), "one retry, then the budget ends it");
assertEquals(Duration.ofSeconds(1), ex.retryAfter(), "the refused hint is preserved");
assertTrue(elapsedMs >= 950, "the first hint must be slept, elapsed " + elapsedMs + " ms");
assertTrue(elapsedMs < 5_000, "the second must not, elapsed " + elapsedMs + " ms");
assertEquals(List.of("retry_attempt"), obs.events);
assertEquals(1, obs.attributes.get(FanarObservationAttributes.FANAR_RETRY_COUNT));
assertSame(ex, obs.errors.getFirst());
}

// --- rate-limit visibility (ADR-026) ------------------------------------------------------

@Test
Expand Down
Loading
Loading