Skip to content

fix: upload CLI defaults to replay=True, add --no-replay opt-out#22

Open
colombod wants to merge 2 commits into
mainfrom
fix/upload-replay-default
Open

fix: upload CLI defaults to replay=True, add --no-replay opt-out#22
colombod wants to merge 2 commits into
mainfrom
fix/upload-replay-default

Conversation

@colombod
Copy link
Copy Markdown
Collaborator

Problem

The upload CLI never passed ?replay=true to the server. The server's in-memory
idempotency cache (7-day TTL) silently dropped session:start events on re-upload,
leaving Session nodes without started_at. This caused re-uploaded old sessions to
appear in Neo4j without their original timestamps.

Changes

uploader.pyrun_upload() gains replay: bool = True. When True (the
default), every POST is sent with ?replay=true, bypassing the server-side cache.
Set to False to restore the old deduplication behaviour.

cli.py--no-replay flag added. Default is replay=True (no flag needed).
_DETAILED_HELP IDEMPOTENCY section rewritten to accurately describe the new default:
cache is bypassed, Neo4j idempotency is provided by MERGE + SET n += row.props.

Why --no-replay instead of --replay

The upload CLI is always a deliberate replay of existing JSONL data. There is no
scenario where silently skipping events is the right default. --no-replay is the
explicit override for the rare case (e.g. uploading a live in-progress session).

Verification

  • 168/168 tests pass (uv run pytest -q)
  • Ruff: clean — All checks passed!
  • Pyright: 0 errors, 0 warnings, 0 informations
  • Red-green confirmed: test fails with assert None == {'replay': True} on main,
    passes on this branch
  • No Neo4j operations in any changed file

Diego Colombo and others added 2 commits May 21, 2026 11:20
The server /events endpoint accepts ?replay=true to bypass the 7-day
in-memory idempotency cache. The upload CLI now passes this by default
so re-uploading old sessions always lands correctly in Neo4j regardless
of how recently the same events were previously sent.

--no-replay restores the old cache-enforced deduplication for callers
that want explicit deduplication (e.g. uploading a session that is
currently being captured live).

Files changed:
- uploader.py: replay: bool = True param, params={"replay": True} on POST
- cli.py: --no-replay flag, replay=not args.no_replay forwarding,
  _DETAILED_HELP IDEMPOTENCY section rewritten to reflect new default
- tests/test_uploader.py: test default sends params={"replay": True}
- tests/test_cli.py: --no-replay behavioural tests
httpx serialises Python bool True to the string "True" (capital T) via str(),
but the server expects lowercase "true". This was causing the replay flag to
be silently ignored on POST /events calls.

Changes:
- uploader.py: Changed {"replay": True} to {"replay": "true"} and updated
  the type hint from dict[str, Any] | None to dict[str, str] | None.
- test_uploader.py: Updated the assertion to match the corrected string value
  and added an explanatory comment describing why the string form is required.
- cli.py: Corrected the help text reference from replay=True to replay=true
  and reformatted [--no-replay] to its own continuation line for consistency.

All 168 tests pass. Ruff clean. Pyright 0 errors.

Generated with Amplifier

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@bkrabach
Copy link
Copy Markdown
Collaborator

Reviewed. Diagnosis (7-day cache silently dedup'ing replays so Session.started_at goes missing) and fix (default replay=True, opt-out via --no-replay) both make sense. One pre-merge ask:

Could you link or quote the server-side /events POST handler to confirm ?replay=true triggers only idempotent Neo4j operations? The PR description states Neo4j writes are MERGE + SET n += row.props. If that's universally true across all event types — no counters, no list appends, no audit log inserts, no side-effecting webhooks — then replay=True by default is safe.

This is a server-repo question, not a critique of this PR. If the server is pure-MERGE: approve immediately.

(The new default has no interaction with the real-time hook path or CR-1 parent_id propagation — the upload CLI is offline-batch only.)

@colombod
Copy link
Copy Markdown
Collaborator Author

Verified against the server repo (amplifier-context-intelligence). The server is pure-MERGE — here's the evidence:

All Neo4j writes go through neo4j_store.py:_flush_body():

# Nodes (line 402-404)
"UNWIND $rows AS row "
"MERGE (n:Session {node_id: row.node_id, workspace: row.props.workspace}) "
"SET n += row.props"

# Non-session nodes (line 411-413) — same pattern
"MERGE (n {node_id: row.node_id, workspace: row.props.workspace}) "
"SET n += row.props"

# Edges (line 475-479) — same pattern
"MERGE (src)-[r:{edge_type}]->(dst) "
"SET r += row.props"

SET n += row.props on an existing node updates properties in place — no duplicate nodes, no counters, no list growth in Neo4j.

Label patches (lines 426, 440, 450): SET n:Label / REMOVE n:Label — idempotent label assignments, not additive writes.

In-memory counters (registry.py:66, iteration.py:74): events_processed += 1, iteration_count += 1 — these are in-memory Python state on the worker, never written to Neo4j, and reset on server restart. They do not accumulate across replay runs.

No webhooks or external HTTP calls in the handler chain — grepped handlers/**/*.py for httpx, requests, urllib, aiohttp: zero matches.

Blob store (blob_store.py:147): path.write_text(json.dumps(value), encoding='utf-8') — overwrites the file, not appends. Idempotent on replay.

One edge case to be aware of: neo4j_store.py:83-84 has a DETACH DELETE duplicate path that fires when concurrent workers race to MERGE the same node. This is a safety cleanup that leaves exactly one node — idempotent by design, not an audit log.

Conclusion: The server is pure-MERGE for all durable state. replay=True by default is safe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants