Skip to content

Riptide creates a new FailsafePlugin and Failsafe instance per policy, bypassing Failsafe’s policy chaining and deadlock protection #1859

Open
Open
Riptide creates a new FailsafePlugin and Failsafe instance per policy, bypassing Failsafe’s policy chaining and deadlock protection#1859
Labels
Bug

Description

@rameshmalla
rameshmalla
opened on May 18, 2026

Description:

Currently, Riptide creates a new instance of FailsafePlugin for each policy, and each FailsafePlugin in turn creates its own Failsafe instance. As a result, multiple policies (such as Retry and Timeout) are applied as separate plugins rather than being composed into a single chained policy.

This implementation choice predates key evolutions in the Failsafe library, and it now causes us to bypass Failsafe’s deadlock prevention logic as documented in reference 3 and 4.

By splitting policies into separate plugins, each policy creates its own Failsafe instance, resulting in independent execution and future contexts for each policy. This means Riptide clients are not protected by Failsafe’s built-in guardrails designed to prevent deadlocks between certain policy combinations such as Timeout and Retry.

Expected Behavior:

Riptide should compose all configured Failsafe policies into a single chain within one FailsafePlugin/Failsafe instance per endpoint/client, rather than instantiating one plugin/instance per policy.

Example

val retryPolicy = RetryPolicy.builder<String>()
            .handle(Exception::class.java)
            .withDelay(Duration.ofSeconds(1))
            .withMaxRetries(3)
            .onRetry { event ->
                println("[Retry] Attempt #${event.attemptCount} | Reason: ${event.lastException?.message}")
            }
            .build()

        val timeout = Timeout.builder<String>(Duration.ofSeconds(2))
            .withInterrupt()
            .onFailure { println("[Timeout] Execution timed out!") }
            .build()
        
        Failsafe.with(timeout, retryPolicy )
            .getStageAsync { context -> fetchDataAsync() }
            .thenAccept { result ->
                println("✅ Result: $result")
            }
            .exceptionally { ex ->
                println("❌ Failed: ${ex.message}")
                null
            }

Actual Behavior:

Each policy is configured as its own FailsafePlugin, resulting in unchained, independent Failsafe instances that do not benefit from Failsafe’s coordinated policy execution and deadlock prevention.

Current behavior

// timeout
        val timeout = Timeout.builder<String>(Duration.ofSeconds(2))
            .withInterrupt()
            .onFailure { println("${Thread.currentThread().name} : [Timeout] Execution timed out!") }
            .build()
        // timeout

        // Retry Policy
        val retryPolicy = RetryPolicy.builder<String>()
            .handle(Exception::class.java)
            .withDelay(Duration.ofSeconds(1))
            .withMaxRetries(3)
            .onRetry { event ->
                println("[Retry] Attempt #${event.attemptCount} | Reason: ${event.lastException?.message}")
            }
            .build()

        //Failsafe with retry policy and the main task
        val retryStage = Failsafe.with(retryPolicy)
            .getStageAsync { context -> fetchDataAsync(atomicInteger.incrementAndGet()) }
        
        Failsafe.with(timeout )
            .getStageAsync { context -> retryStage }   // Returns CompletableFuture
            .thenAccept { result ->
                println("✅ Result: $result")
            }
            .exceptionally { ex ->
                println("❌ Failed: ${ex.message}")
                null
            }

References:

  1. FailsafePluginFactory : https://github.com/zalando/riptide/blob/main/riptide-spring-boot-autoconfigure/src/main/java/org/zalando/riptide/autoconfigure/FailsafePluginFactory.java#L45
  2. FailsafePlugin.java: https://github.com/zalando/riptide/blob/main/riptide-failsafe/src/main/java/org/zalando/riptide/failsafe/FailsafePlugin.java#L76
  3. Failsafe TimeoutExecutor Deadlock Guard: https://github.com/failsafe-lib/failsafe/blob/master/core/src/main/java/dev/failsafe/internal/TimeoutExecutor.java#L118
  4. Failsafe RetryPolicyExecutor deadlock guard: https://github.com/failsafe-lib/failsafe/blob/master/core/src/main/java/dev/failsafe/internal/RetryPolicyExecutor.java#L152

Steps to Reproduce

I used this test to reproduce this issue by adding some delays through debug points in


    @Test
    fun testFailsafeDeadlock(){


        // timeout
        val timeout = Timeout.builder<String>(Duration.ofSeconds(2))
            .withInterrupt()
            .onFailure { println("${Thread.currentThread().name} : [Timeout] Execution timed out!") }
            .build()
        // timeout

        // Retry Policy
        val retryPolicy = RetryPolicy.builder<String>()
            .handle(Exception::class.java)
            .withDelay(Duration.ofSeconds(1))
            .withMaxRetries(3)
            .onRetry { event ->
                println("[Retry] Attempt #${event.attemptCount} | Reason: ${event.lastException?.message}")
            }
            .build()

        //Failsafe with retry policy and the main task
        val retryStage = Failsafe.with(retryPolicy)
            .getStageAsync { context -> fetchDataAsync(atomicInteger.incrementAndGet()) }

        Failsafe.with(timeout )
            .getStageAsync { context -> retryStage }   // Returns CompletableFuture
            .thenAccept { result ->
                println("✅ Result: $result")
            }
            .exceptionally { ex ->
                println("❌ Failed: ${ex.message}")
                null
            }
            .join() // Block main thread until done
    }

    fun fetchDataAsync(incrementAndGet: Int): CompletableFuture<String> {
        return CompletableFuture.supplyAsync {
            println("[Service] Call $incrementAndGet Fetching data on thread: ${Thread.currentThread().name}")
            if(incrementAndGet==1){
                throw Exception("Failing")
            }
            Thread.sleep(1100)
            "Hello from async service!"
        }
    }

Debug points

  • FailsafeFuture L117
  • TimeoutExecutor L125
  • TimeoutExecutor L119
  • RetryPolicyExecutor L145

Context

This led to a deadlock in one of our services, which blocked the forked join pool from being used by other processes, leading to degradation of the performance of that service.

Thread dump for investigation


Found one Java-level deadlock:
=============================
"ForkJoinPool.commonPool-worker-3":
  waiting to lock monitor 0x000079025c01e1c0 (object 0x00001000f287dc00, a dev.failsafe.spi.FailsafeFuture),
  which is held by "http-my-service-234"

"http-my-service-234":
  waiting to lock monitor 0x00007902bc000eb0 (object 0x00001000f2882aa0, a dev.failsafe.spi.FailsafeFuture),
  which is held by "ForkJoinPool.commonPool-worker-3"

Java stack information for the threads listed above:
===================================================
"ForkJoinPool.commonPool-worker-3":
	at dev.failsafe.spi.FailsafeFuture.cancel(FailsafeFuture.java:74)
	- waiting to lock <0x00001000f287dc00> (a dev.failsafe.spi.FailsafeFuture)
	at dev.failsafe.spi.FailsafeFuture.lambda$propagateCancellation$0(FailsafeFuture.java:175)
	at dev.failsafe.spi.FailsafeFuture$$Lambda/0x000079042ccb5a70.accept(Unknown Source)
	at dev.failsafe.spi.FailsafeFuture.cancelDependencies(FailsafeFuture.java:129)
	- locked <0x00001000f2882aa0> (a dev.failsafe.spi.FailsafeFuture)
	at dev.failsafe.internal.TimeoutExecutor.lambda$applyAsync$2(TimeoutExecutor.java:133)
	- locked <0x00001000f2882be0> (a java.util.concurrent.atomic.AtomicReference)
	at dev.failsafe.internal.TimeoutExecutor$$Lambda/0x000079042cfe2000.call(Unknown Source)
	at dev.failsafe.internal.util.DelegatingScheduler.lambda$schedule$0(DelegatingScheduler.java:141)
	at dev.failsafe.internal.util.DelegatingScheduler$$Lambda/0x000079042ccaed68.call(Unknown Source)
	at io.opentelemetry.javaagent.bootstrap.executors.ContextPropagatingCallable.call(ContextPropagatingCallable.java:38)
	at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(java.base@21.0.10/ForkJoinTask.java:1456)
	at java.util.concurrent.ForkJoinTask.doExec(java.base@21.0.10/ForkJoinTask.java:387)
	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@21.0.10/ForkJoinPool.java:1312)
	at java.util.concurrent.ForkJoinPool.scan(java.base@21.0.10/ForkJoinPool.java:1843)
	at java.util.concurrent.ForkJoinPool.runWorker(java.base@21.0.10/ForkJoinPool.java:1808)
	at java.util.concurrent.ForkJoinWorkerThread.run(java.base@21.0.10/ForkJoinWorkerThread.java:188)
"http-my-service-234":
	at dev.failsafe.spi.FailsafeFuture.completeResult(FailsafeFuture.java:93)
	- waiting to lock <0x00001000f2882aa0> (a dev.failsafe.spi.FailsafeFuture)
	at dev.failsafe.AsyncExecutionImpl.complete(AsyncExecutionImpl.java:153)
	at dev.failsafe.AsyncExecutionImpl$$Lambda/0x000079042ccb4d08.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.internal.TimeoutExecutor.lambda$applyAsync$4(TimeoutExecutor.java:174)
	at dev.failsafe.internal.TimeoutExecutor$$Lambda/0x000079042cfe28e8.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.Functions.lambda$toAsync$5(Functions.java:202)
	at dev.failsafe.Functions$$Lambda/0x000079042ccb60f0.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.Functions.lambda$getPromiseOfStage$2(Functions.java:152)
	at dev.failsafe.Functions$$Lambda/0x000079042ccb5ca8.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.spi.FailsafeFuture.completeResult(FailsafeFuture.java:99)
	- locked <0x00001000f287dc00> (a dev.failsafe.spi.FailsafeFuture)
	at dev.failsafe.AsyncExecutionImpl.complete(AsyncExecutionImpl.java:153)
	at dev.failsafe.AsyncExecutionImpl$$Lambda/0x000079042ccb4d08.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$handleAsync$5(RetryPolicyExecutor.java:155)
	- locked <0x00001000f287dd50> (a java.util.concurrent.atomic.AtomicReference)
	at dev.failsafe.internal.RetryPolicyExecutor$$Lambda/0x000079042cd6da58.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(java.base@21.0.10/CompletableFuture.java:887)
	at java.util.concurrent.CompletableFuture.whenComplete(java.base@21.0.10/CompletableFuture.java:2357)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$handleAsync$6(RetryPolicyExecutor.java:150)
	at dev.failsafe.internal.RetryPolicyExecutor$$Lambda/0x000079042ccaf430.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$handleAsync$5(RetryPolicyExecutor.java:155)
	- locked <0x00001000f287dd50> (a java.util.concurrent.atomic.AtomicReference)
	at dev.failsafe.internal.RetryPolicyExecutor$$Lambda/0x000079042cd6da58.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(java.base@21.0.10/CompletableFuture.java:887)
	at java.util.concurrent.CompletableFuture.whenComplete(java.base@21.0.10/CompletableFuture.java:2357)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$handleAsync$6(RetryPolicyExecutor.java:150)
	at dev.failsafe.internal.RetryPolicyExecutor$$Lambda/0x000079042ccaf430.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$handleAsync$5(RetryPolicyExecutor.java:155)
	- locked <0x00001000f287dd50> (a java.util.concurrent.atomic.AtomicReference)
	at dev.failsafe.internal.RetryPolicyExecutor$$Lambda/0x000079042cd6da58.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(java.base@21.0.10/CompletableFuture.java:887)
	at java.util.concurrent.CompletableFuture.whenComplete(java.base@21.0.10/CompletableFuture.java:2357)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$handleAsync$6(RetryPolicyExecutor.java:150)
	at dev.failsafe.internal.RetryPolicyExecutor$$Lambda/0x000079042ccaf430.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.Functions.lambda$toAsync$5(Functions.java:202)
	at dev.failsafe.Functions$$Lambda/0x000079042ccb60f0.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.Functions.lambda$getPromiseOfStage$2(Functions.java:152)
	at dev.failsafe.Functions$$Lambda/0x000079042ccb5ca8.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.spi.FailsafeFuture.completeResult(FailsafeFuture.java:99)
	- locked <0x00001000f287c830> (a dev.failsafe.spi.FailsafeFuture)
	at dev.failsafe.AsyncExecutionImpl.complete(AsyncExecutionImpl.java:153)
	at dev.failsafe.AsyncExecutionImpl$$Lambda/0x000079042ccb4d08.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.spi.PolicyExecutor.lambda$applyAsync$2(PolicyExecutor.java:109)
	at dev.failsafe.spi.PolicyExecutor$$Lambda/0x000079042ccb5838.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.Functions.lambda$toAsync$5(Functions.java:202)
	at dev.failsafe.Functions$$Lambda/0x000079042ccb60f0.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture.complete(java.base@21.0.10/CompletableFuture.java:2179)
	at dev.failsafe.Functions.lambda$getPromiseOfStage$2(Functions.java:152)
	at dev.failsafe.Functions$$Lambda/0x000079042ccb5ca8.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@21.0.10/CompletableFuture.java:863)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@21.0.10/CompletableFuture.java:841)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@21.0.10/CompletableFuture.java:510)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(java.base@21.0.10/CompletableFuture.java:1773)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.10/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.10/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith(java.base@21.0.10/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.10/Thread.java:1583)

Your Environment

  • Version used: 4.5.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    Bug

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions