feat: add RETRY-conformance contract tests for FDv1 streaming and polling by tanderson-ld · Pull Request #404 · launchdarkly/sdk-test-harness

tanderson-ld · 2026-08-07T20:14:10Z

Summary

Adds contract tests exercising the RETRY specification's non-permanent-stop guarantee for the FDv1 streaming and polling data sources. Introduces two capability gates so SDKs can adopt independently:

retry-conformance-fdv1-streaming
retry-conformance-fdv1-polling

The new tests exercise the SDK's real production 5-minute extended-regime timing — no test-harness bypass, no compression knobs on the servicedef, no test-only surface in the SDK's public API. They are marked as long-running (t.LongRunning()) and only run when the harness is invoked with -enable-long-running-tests.

Legacy "do not retry after unrecoverable HTTP error" scenarios remain, gated behind the ABSENCE of the streaming capability so SDKs can migrate on their own schedule. Once every SDK reports the new capabilities, both the legacy tests and the gates can be removed.

What's added

Streaming (guarded by retry-conformance-fdv1-streaming + -enable-long-running-tests):

retry after unexpected HTTP error on initial connect (401/403/405)
retry after unexpected HTTP error on reconnect (401/403/405)
enters extended-regime backoff after unexpected HTTP error (paired-bound timing assertion)
does not permanently stop under sustained unexpected HTTP errors
returns to normal-regime backoff after healthy operation (streaming spec's activeSince reset: 75 s silent stream after an extended-regime reconnect, then next reconnect must be at normal-regime timing)

Polling (guarded by retry-conformance-fdv1-polling + -enable-long-running-tests):

retry after unexpected HTTP error on initial poll (401/403/405)
returns to normal-regime cadence after two consecutive successful polls (RETRY §1.8.1 count-based reset)

Design: long-running instead of compressed timings

An earlier revision of this PR exposed extended-regime timing knobs on the servicedef (ExtendedInitialDelayMS, ResetThresholdMS) so tests could compress the 5-minute extended-initial delay into a ~30-second observable window. Reviewers pushed back because that shape forced SDKs to expose corresponding test-only knobs on their public API surface just to be testable by the harness.

This revision drops all compression knobs and exercises real production timing instead. That makes each retry-conformance subtest take several minutes of wall clock, which is why the tests are marked t.LongRunning() and gated behind -enable-long-running-tests. Per-SDK CI schedules for the long-running suite are out of scope for this PR (planned follow-up work).

Framework support for t.LongRunning() and the -enable-long-running-tests flag was ported from v3 in #428, a prerequisite that landed before this PR's refactor.

Relationship to PR #401

Supersedes #401, which targeted v3 and exercised FDv2 code paths — out of scope for the initial RETRY-conformance work, which is FDv1-only. This PR targets v2 and drives the FDv1 streaming/polling paths in the SDK testservices. #401 will be closed with a pointer here.

SDK-2808

Test plan

Contract tests pass against go-server-sdk PR feat: Verify per-context private attributes are not applied to other contexts #429 (the reference server SDK implementation) at real production 5-minute extended-regime timing (~3-hour run with -enable-long-running-tests)
Legacy "do not retry" tests still pass for SDKs without the new capability (verified by SDK capability check)
Retry-conformance tests are correctly SKIPPED for SDKs without the capability
Retry-conformance tests are correctly SKIPPED when -enable-long-running-tests is not set

…ling Adds SDK-facing capability gates (retry-conformance-fdv1-streaming, retry-conformance-fdv1-polling) plus contract tests exercising the RETRY specification's non-permanent-stop guarantee on 401/403/405 responses. Streaming tests cover: retry after unexpected HTTP error on initial connect and on reconnect, extended-regime backoff timing (paired-bound assertion), and sustained-error survivability. Legacy "do not retry" tests are now gated behind the ABSENCE of the streaming capability so SDKs can migrate independently. Polling tests cover: retry after unexpected HTTP error on initial poll. Additional polling scenarios (extended-regime shape, two-consecutive- success reset) are deferred to follow-up tickets. New timing knobs on SDKConfigStreamingParams (ExtendedInitialDelayMS, ResetThresholdMS) and SDKConfigPollingParams (ExtendedInitialDelayMS) let tests compress the extended regime into an observable window.

The sustained-retry test used the default 5 s NewSDKClient wait, so any extended-regime retry attempts made during init were pre-buffered in the mock endpoint before the assertion loop began. An SDK that retried a few times and then permanently stopped could still satisfy the 3-iteration drain, defeating the intent of the test. - Set StartWaitTimeMS to 1 ms so NewSDKClient returns quickly and each RequireConnection observes a live retry attempt rather than draining a pre-filled buffer (mirrors the sibling extended-regime-backoff test). - Add a final RequireConnection after the loop -- the "one more still arrives at the end" check the existing comment already described but which was missing from the code. - Bump loop count from 3 to 5 and introduce a per-test timeout of 12 s that accommodates the extended-regime doubling backoff (500, 1000, 2000, 4000, 8000 ms gaps) so a broken SDK cannot slip through with a handful of early attempts. SDK-2808

joker23

we should update https://github.com/launchdarkly/sdk-test-harness/blob/v3/docs/service_spec.md as well to document this capability.

Per PR review: the evaluation assertions were reading before the connection assertions that prove the retry actually happened. Reorder so the tests read chronologically -- drain the expected connections first, then evaluate the flag as the payoff.

Per PR review: add sections for "retry-conformance-fdv1-streaming" and "retry-conformance-fdv1-polling" capabilities, and document the associated streaming (extendedInitialDelayMs, resetThresholdMs) and polling (extendedInitialDelayMs) configuration knobs the test harness uses to compress the extended regime into an observable window.

tanderson-ld · 2026-08-14T15:09:17Z

@joker23 Documented in 25df1a0. Added "retry-conformance-fdv1-streaming" and "retry-conformance-fdv1-polling" capability sections in docs/service_spec.md, plus the associated extendedInitialDelayMs / resetThresholdMs streaming knobs and the polling extendedInitialDelayMs knob in the config schema.

…ormance The two legacy "do not retry after unrecoverable HTTP error" tests were gated only by CapabilityClientEventSourceHTTPErrors. Under the RETRY specification, an SDK that declares retry-conformance-fdv1-streaming retries after those errors instead of permanently stopping, so the legacy tests would fail against a compliant client-side SDK. Adds the same skip-when-capability-present guard the server-side legacy tests already use, so the client-side suite is ready for future client SDK adoption of the RETRY capability.

## Summary Ports the framework scaffolding portion of [#314](#314) (Jason Bailey, "fix: Add support for optional long running tests") from v3 to v2. ## What ships - `-enable-long-running-tests` CLI flag (default false). Mirrors the `-enable-persistence-tests` shape. - `EnableLongRunningTests bool` on `ldtest.TestConfiguration`. - `(t *T) LongRunning()` method. Long-running tests call it at the top of their body; if the flag isn't set, the test skips with reason `"use -enable-long-running-tests to run"`. - Threaded through `RunSDKTestSuite` and `main.go`. ## What's deliberately not ported The original commit ([`01a245c`](01a245c)) also added 237 lines to `sdktests/common_tests_stream_fdv2.go`. That file exercises v3-only FDv2 shapes and has no place on v2. Skipped. ## Why now Scaffolding for follow-up work on [#404](#404). Review feedback there ([discussion](#404 (comment)), [discussion](#404 (comment))) argued against exposing an SDK-side servicedef knob to compress the extended-regime timing in the RETRY-conformance tests. The consensus was that those tests should run at real timing gated by the long-running-tests mechanism — which requires that mechanism to exist on v2 first. Once this lands, #404 rebases onto v2 tip and its RETRY-conformance tests get `t.LongRunning()` calls at the top of their bodies, plus timing rewritten to production defaults. ## Verification - `go build ./...` — clean. - `go run . -h` — new flag visible: `-enable-long-running-tests enable tests that take a long time to run (10+ seconds)`. - `go test ./...` — full suite green. No behavior change to existing v2 tests since none of them call `t.LongRunning()`. ## Test plan - [x] Confirm the port matches the shape of #314's framework portion. - [ ] Confirm the CLI flag help text is acceptable (verbatim from Jason's original). - [ ] Confirm no v3-only content bled through.  --- > [!NOTE] > **Overview** > Adds optional **long-running test** gating on v2, mirroring the existing `-enable-persistence-tests` pattern. > > The harness accepts `-enable-long-running-tests` (default false). That value flows through `main` → `RunSDKTestSuite` → `ldtest.TestConfiguration.EnableLongRunningTests`. Tests can call `t.LongRunning()` at the start; when the flag is off they skip with reason `use -enable-long-running-tests to run`. > > No existing v2 tests call `LongRunning()` yet, so default runs behave the same until follow-up tests adopt the helper. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 536af50. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>

…rmance-tests-v2

…ning gate Remove the servicedef knobs (ExtendedInitialDelayMS, ResetThresholdMS) that would require SDK-side test-harness bypasses in the public API. Instead, mark the retry-conformance streaming and polling tests as long-running so they exercise the SDK's real 5-minute extended-regime delay directly. Add regime-recovery tests: - streaming: verifies the healthy-operation reset per the streaming spec's activeSince mechanics (first SSE event sets marker; reset evaluated on next reconnection when >= 60 s have elapsed). - polling: verifies RETRY §1.8.1's count-based reset (two consecutive successful polls return the SDK to normal-regime cadence).

cursor

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9964faf. Configure here.}

- server_side_stream_retry.go "retry after unexpected HTTP error on initial connect": RequireConnection returns before the SDK reads the SSE payload, so an immediate EvaluateAllFlags could observe an empty store. Switch to pollUntilFlagValueUpdated, matching the sibling reconnect test. - server_side_poll_retry.go "returns to normal-regime cadence": dropping the second 401 that would spuriously fail a spec-conformant SDK. A 401 after the §1.8.1 reset re-enters extended regime under RETRY, so the following poll would take ~5 min, not the ~30 s the assertion expected. Prove the reset by observing that a post-reset success arrives at normal PollInterval cadence.

🤖 I have created a release *beep* *boop* --- ## [2.40.0](v2.39.0...v2.40.0) (2026-08-19) ### Features * add RETRY-conformance contract tests for FDv1 streaming and polling ([#404](#404)) ([ee4d85f](ee4d85f)) ### Bug Fixes * Retry the test harness download and surface curl errors ([#411](#411)) ([2a7533b](2a7533b)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).  --- > [!NOTE] > **Overview** > **Release 2.40.0** bumps the package version in `.release-please-manifest.json`, `main.go` (`versionString`), and documents the release in `CHANGELOG.md`. > > The changelog for this tag records two already-merged changes: **RETRY-conformance** contract tests for FDv1 **streaming** and **polling** (capability-gated, including long-running retry-regime checks), and a **downloader** fix that **retries** harness downloads and **surfaces curl errors** for easier CI debugging. > > There is no additional functional code in this PR diff beyond the version and changelog updates. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit b96a76c. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>  Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This was referenced Aug 7, 2026

feat: conform FDv1 streaming and polling to RETRY spec launchdarkly/go-server-sdk#429

Open

feat: RETRY-conformance contract tests for FDv1 streaming and polling #401

Closed

tanderson-ld marked this pull request as ready for review August 10, 2026 18:09

tanderson-ld requested a review from a team as a code owner August 10, 2026 18:09

cursor Bot reviewed Aug 10, 2026

View reviewed changes

Comment thread sdktests/server_side_stream_retry.go

Merge branch 'v2' into ta/sdk-2808-retry-conformance-tests-v2

7a11250

tanderson-ld mentioned this pull request Aug 12, 2026

feat(server): RETRY-spec conformance in FDv1 streaming and polling launchdarkly/java-core#200

Open

5 tasks

tanderson-ld and others added 2 commits August 12, 2026 14:37

Merge branch 'v2' into ta/sdk-2808-retry-conformance-tests-v2

5cbe223

kinyoklion reviewed Aug 12, 2026

View reviewed changes

Comment thread servicedef/service_params.go

joker23 reviewed Aug 12, 2026

View reviewed changes

keelerm84 reviewed Aug 12, 2026

View reviewed changes

Comment thread sdktests/server_side_poll_retry.go Outdated

Comment thread sdktests/server_side_stream_retry.go Outdated

Comment thread sdktests/server_side_stream_retry.go Outdated

tanderson-ld added 2 commits August 14, 2026 11:00

Merge branch 'v2' into ta/sdk-2808-retry-conformance-tests-v2

d749ec0

tanderson-ld requested review from joker23, keelerm84 and kinyoklion August 14, 2026 15:12

cursor Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread docs/service_spec.md Outdated

cursor Bot reviewed Aug 17, 2026

View reviewed changes

Comment thread sdktests/client_side_stream_retry.go

tanderson-ld mentioned this pull request Aug 18, 2026

chore: port long-running test framework support from v3 #428

Merged

3 tasks

tanderson-ld added 2 commits August 18, 2026 10:24

Merge remote-tracking branch 'origin/v2' into ta/sdk-2808-retry-confo…

8f02fba

…rmance-tests-v2

cursor Bot reviewed Aug 18, 2026

View reviewed changes

Comment thread sdktests/server_side_stream_retry.go Outdated

Comment thread sdktests/server_side_poll_retry.go

kinyoklion approved these changes Aug 19, 2026

View reviewed changes

tanderson-ld merged commit ee4d85f into v2 Aug 19, 2026
7 checks passed

tanderson-ld deleted the ta/sdk-2808-retry-conformance-tests-v2 branch August 19, 2026 19:04

github-actions Bot mentioned this pull request Aug 19, 2026

chore(v2): release 2.40.0 #414

Merged

Navigation Menu

Uh oh!

feat: add RETRY-conformance contract tests for FDv1 streaming and polling - #404

feat: add RETRY-conformance contract tests for FDv1 streaming and polling#404
tanderson-ld merged 11 commits into
v2from
ta/sdk-2808-retry-conformance-tests-v2

tanderson-ld commented Aug 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

joker23 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tanderson-ld commented Aug 14, 2026

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

tanderson-ld commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's added

Design: long-running instead of compressed timings

Relationship to PR #401

Test plan

Uh oh!

Uh oh!

Uh oh!

joker23 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tanderson-ld commented Aug 14, 2026

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tanderson-ld commented Aug 7, 2026 •

edited

Loading