Skip to content

feat(sync): idempotent import so retries do not duplicate IMAP messages #85

Description

@TDannhauer

Summary

Sync client commands (especially EAS 16 email Modify on Drafts) are applied to IMAP before the client has accepted the Sync response / advanced its SyncKey. If the response is lost, timed out, or silently rejected, the client retries the same command batch under the old SyncKey while the backend may already have written new messages — producing unbounded duplication (observed: Drafts growing from a clean folder toward tens of thousands of copies under Gmail Android FullDraftsUpSync).

Motivation

This showed up while investigating horde/ActiveSync#77. Streaming / transport fixes address client read timeouts, but they do not fix the durability mismatch:

  1. Client sends Modify (or Add) under SyncKey N
  2. Server commits to IMAP (often a replace that creates a new UID)
  3. Server builds the Sync response (new SyncKey N+1, <Replies>, possibly server→client <Add>s for the new UIDs)
  4. If the client never accepts that response, it stays on SyncKey N and re-sends the same commands
  5. A non-idempotent import path appends another generation of messages

True two-phase commit with the client is not possible over HTTP (the server cannot know the client accepted the body). The robust EAS approach is idempotent application of client commands plus Sync state that remembers what was already applied for a given SyncKey transition.

Proposed direction

  • Make Sync import of Modify/Add/Remove idempotent under retry of the same SyncKey / same client command identity (server id + change identity, or an applied-commands record for NN+1).
  • For Draft email Modify in particular: map the replaced IMAP UID correctly in sync state (old UID deleted/replaced, not an extra unmatched <Add> back to the client) so a successful round-trip does not amplify the client's local draft set.
  • Ensure a retried batch after a committed-but-unacked response returns the same successful replies without creating new IMAP messages.

Out of scope

Test plan (once implemented)

  • Simulate Drafts Modify batch, commit IMAP, abort response before client ack; retry same SyncKey — message count must not increase
  • Successful Drafts up-sync does not produce a flood of server→client <Add>s for the just-replaced UIDs
  • Gmail Android FullDraftsUpSync against a small Drafts folder does not grow the folder across failed/partial responses

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions