Skip to content

feat: OpenFeature track and event delivery reliability - #252

Open
fabriziodemaria wants to merge 16 commits into
mainfrom
openfeature-track-context
Open

feat: OpenFeature track and event delivery reliability#252
fabriziodemaria wants to merge 16 commits into
mainfrom
openfeature-track-context

Conversation

@fabriziodemaria

@fabriziodemaria fabriziodemaria commented Aug 7, 2026

Copy link
Copy Markdown
Member

Summary

  • Align OpenFeature track() with Confidence event conventions (details → data, context merge, "value" / "context" overrides).
  • Improve event delivery reliability: optional interval flush, startup flush, stop/shutdown flush, provider shutdown()flush().

Test plan

  • Provider track tests
  • EventSenderEngineReliabilityTest
  • Full CI on combined branch

Supersedes #253 (reliability) — both changes are on openfeature-track-context.

@fabriziodemaria fabriziodemaria changed the title Align OpenFeature track() with Confidence event conventions feat(provider): align OpenFeature track() with Confidence event conventions Aug 7, 2026
@fabriziodemaria fabriziodemaria changed the title feat(provider): align OpenFeature track() with Confidence event conventions feat: OpenFeature track and event delivery reliability Aug 10, 2026
@vahidlazio

Copy link
Copy Markdown
Collaborator

sendChannel is still a rendezvous channel (capacity 0). When minBatchSizeFlushPolicy triggers a flush and an upload is already in flight, the writer blocks on sendChannel.send(SEND_SIG) and stops draining writeReqChannel. If stop() is called during that window, the 2s timeout fires before the writer can consume the remaining buffered events, and coroutineScope.cancel() drops them.

Consider making sendChannel buffered (e.g. Channel(Channel.UNLIMITED)) or switching to trySend — the sender already processes all ready files per invocation, so duplicate signals are harmless.

The stopDrainsQueuedEventsBeforeUploading test doesn't catch this because the engine is created without minBatchSizeFlushPolicy, so the writer never hits sendChannel.send().

@vahidlazio

Copy link
Copy Markdown
Collaborator

Here's a failing test that reproduces the issue under production conditions (batch flush policy + slow upload). It currently fails with expected:<12> but was:<10> — the writer blocks on the rendezvous sendChannel.send() while an upload is in flight, so 2 events never reach disk before the 2s stop timeout.

The fix should make this pass (e.g. buffering sendChannel or switching to trySend):

@Test
fun stopDrainsAllEventsWhenFlushPolicyBlocksWriter() {
    val slowUploader = SlowEventUploader(uploadDelayMillis = 3_000)
    val batchFlush = object : FlushPolicy {
        private var count = 0
        override fun reset() { count = 0 }
        override fun hit(event: EngineEvent) { count++ }
        override fun shouldFlush(): Boolean = count > 4
    }

    val engine = EventSenderEngineImpl(
        eventStorage = storage,
        clientSecret = "secret",
        uploader = slowUploader,
        flushPolicies = mutableListOf(batchFlush),
        dispatcher = Dispatchers.IO,
        sdkMetadata = SdkMetadata("id", "1.0"),
        debugLogger = null
    )

    repeat(12) { engine.emit("event-$it", mapOf(), mapOf()) }
    Thread.sleep(100)
    engine.stop()

    assertEquals(
        "All 12 events should be written to storage before stop() returns",
        12,
        storage.storedEventCount()
    )
}

private class SlowEventUploader(
    private val uploadDelayMillis: Long
) : EventSenderUploader {
    val uploadedEventNames = mutableListOf<String>()

    override suspend fun upload(events: EventBatchRequest): Boolean {
        kotlinx.coroutines.delay(uploadDelayMillis)
        synchronized(uploadedEventNames) {
            uploadedEventNames.addAll(
                events.events.map { it.eventDefinition.removePrefix("eventDefinitions/") }
            )
        }
        return true
    }
}

RecordingEventStorage also needs a helper:

fun storedEventCount(): Int = synchronized(this) {
    currentEvents.size + readyEvents.values.sumOf { it.size }
}

@fabriziodemaria

Copy link
Copy Markdown
Member Author

Addressed the sendChannel blocking issue in b6048ed:

  • sendChannel is now Channel(CONFLATED) with trySend(SEND_SIG) so the writer loop never suspends on flush signaling while upload holds uploadMutex
  • Added stopDrainsAllEventsWhenFlushPolicyBlocksWriter (your repro with batch flush policy + slow uploader) — asserts all 12 events reach storage before stop() returns

The earlier writeReqChannel UNLIMITED change only helps when the writer keeps consuming; blocking on rendezvous sendChannel.send() was the remaining gap.

fabriziodemaria added a commit that referenced this pull request Aug 17, 2026
Use a conflated sendChannel with trySend so the writer loop never
blocks on flush signaling while upload holds uploadMutex. Adds
stopDrainsAllEventsWhenFlushPolicyBlocksWriter repro from review.

Addresses vahidlazio feedback on PR #252.
fabriziodemaria added a commit that referenced this pull request Aug 19, 2026
Use a conflated sendChannel with trySend so the writer loop never
blocks on flush signaling while upload holds uploadMutex. Adds
stopDrainsAllEventsWhenFlushPolicyBlocksWriter repro from review.

Addresses vahidlazio feedback on PR #252.
@fabriziodemaria
fabriziodemaria force-pushed the openfeature-track-context branch from c214081 to dcc7cab Compare August 19, 2026 09:06
fabriziodemaria and others added 9 commits August 19, 2026 14:22
Merge per-event OpenFeature context with session context, map tracking details to event data, and allow explicit context overrides in the payload — matching Swift provider behavior.
Upload pending batches at startup, flush and upload on stop(), and add an optional periodic flush interval so low-volume events are not left on disk indefinitely.
Do not assign the companion Instance field; main never cached instances and
tests rely on a fresh engine per ConfidenceFactory.create() call.
Engine reliability fixes:
- Bound stop()'s final upload with a 2s timeout so it can never block
  the caller indefinitely; unsent batches are retried at next startup
- Serialize uploadReadyBatches with a mutex so the flush consumer,
  startup retry and stop() can no longer upload the same batch twice
- Skip uploading empty batches, so periodic flushing on an idle app no
  longer makes an HTTP request per interval
- Enqueue emit()/flush() synchronously on a buffered channel and drain
  it in stop(), so every event accepted before stop() reaches disk
- Open the current events file in append mode: a restart previously
  truncated the prior session's unsealed events

API and mapping fixes:
- Log a debug warning when event data's "context" field overrides the
  evaluation context; deprecate the now-unthrown InvalidContextInMessage
- Preserve Long/Float tracking values instead of mapping them to Null
- Reject non-positive eventFlushIntervalMillis in ConfidenceFactory
- Regenerate the API dump (create() overload was hand-edited, HttpError
  was missing)

Test fixes:
- Fix reliability tests: UnconfinedTestDispatcher is not a type, and
  advanceUntilIdle hangs forever on the self-rescheduling interval job
- Make the storage fake honor deletion and seal empty batches like the
  real implementation; assert exact upload counts
- Add tests for drain-on-stop, emit-after-stop, idle-interval no-upload
  and Long/Float value mapping
- Reset the shared minBatchSizeFlushPolicy in setup so
  EventSenderIntegrationTest is no longer order-dependent

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clarify the rendezvous-to-buffered channel change for stop() drain
reliability and trySend() semantics in response to PR review.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use a conflated sendChannel with trySend so the writer loop never
blocks on flush signaling while upload holds uploadMutex. Adds
stopDrainsAllEventsWhenFlushPolicyBlocksWriter repro from review.

Addresses vahidlazio feedback on PR #252.
@fabriziodemaria
fabriziodemaria force-pushed the openfeature-track-context branch from dcc7cab to aebae9a Compare August 19, 2026 12:25

@fabriziodemaria fabriziodemaria left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bugbot follow-up review: three reliability issues to address before merge.

Comment thread Confidence/src/main/java/com/spotify/confidence/EventSenderEngine.kt Outdated
Comment thread Confidence/src/main/java/com/spotify/confidence/PayloadMerger.kt Outdated
Comment thread Confidence/src/main/java/com/spotify/confidence/EventSenderEngine.kt Outdated
Drain accepted events asynchronously before bounding network delivery, and snapshot caller-owned payload maps so queued events remain stable.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread Confidence/src/main/java/com/spotify/confidence/Confidence.kt Outdated
Comment thread Confidence/src/main/java/com/spotify/confidence/PayloadMerger.kt Outdated
Comment thread Confidence/src/main/java/com/spotify/confidence/EventSender.kt Outdated
nicklasl and others added 2 commits August 24, 2026 14:01
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
nicklasl and others added 3 commits August 24, 2026 14:18
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants