feat(guard): capture events with bounded batched delivery - #172
Conversation
There was a problem hiding this comment.
Arcjet Review — 🟡 Medium Risk
Decision: Approved
Rationale: New capture()/flush() feature with bounded batched background delivery. No auth, infra, migration, or dependency changes. No security-critical surface: metadata is documented as untrusted and never logged in diagnostics, no injection risk (protobuf serialization), no hardcoded secrets (test key 'test_key_123' is clearly a test fixture). The design is thoughtful — degrade-don't-fail normalization, bounded queue with drop-on-full, daemon worker thread, coalesced diagnostics, and comprehensive unit + integration tests including hostile-input tests. Medium risk chiefly due to concurrency complexity in the delivery layer (worker thread lifecycle, async loop rebinding, in-flight accounting). I am confident enough in correctness to approve, with a few inline questions/suggestions for the author to consider — most notably a potential race in ArcjetGuardSync._ensure_delivery under concurrent first-capture from multiple threads.
Summary of Changes
Adds an SDK-side capture() API (async + sync) that records visibility events fire-and-forget, queued and batched to a Capture RPC on a background worker (daemon thread for sync, asyncio task for async). Includes a flush() with a 1s default deadline, coalesced local diagnostics (AJ30xx codes) via the arcjet logger, and validation that degrades bad fields into local_warnings rather than raising. Adds _capture.py, _delivery.py, _diagnostics.py, wires them into ArcjetGuard/ArcjetGuardSync, and adds unit + integration tests plus README docs.
Notes
Total diff is ~1500 lines including tests and docs, above the 1000-line threshold — flagging for size, though the change is cleanly separated (new files + narrow additions to _client.py) and the tests cover behavior thoroughly, so automated review remains reliable here.
Review: 88fd27fb | Model: anthropic/claude-opus-4-7 | Powered by Arcjet Review
Adds `capture()` and `flush()` to both guard clients. `capture()` records something the application did, for visibility only: it never affects a decision, never raises, and returns without awaiting on either client. Delivery follows the shape the capture ADRs set out: - Bounded queue, dropping when full. Blocking for space would turn our telemetry into the caller's latency, since capture() runs on a request path inside someone else's application. - Batched, so a burst of calls costs one request instead of one each. - Send-once. A failed batch is dropped and never retried; a stale visibility event is worth less than the capacity a retry consumes. - `flush(timeout_ms)` drains queued and in-flight events and drops the remainder on expiry. There is no `close()`. Every drop is reported through the `arcjet` logger with a stable code — AJ3000 invalid input, AJ3001 queue full, AJ3002 send failed, AJ3003 flush deadline. Nothing is discarded silently. Capture is fire-and-forget, so there is no response to carry a warning back on; without a local channel a drop would be invisible. Diagnostics coalesce per code so a persistent problem cannot turn a best-effort drop into a logging incident, while the accumulated count is still reported. Each event sets `source` to "sdk". The producer decides this, not the caller, so it is not an argument; the server stores an absent source as NULL, meaning unknown, and never substitutes a default. There are two delivery drivers because arcjet-py has two clients: a daemon worker thread for the sync client and an asyncio task for the async one, so each uses its own transport and neither pays for the other's machinery. The async worker is created on first use and rebinds if the event loop changes, because a client outlives any single loop. Normalization validates each field independently, so one bad optional value drops only itself and is reported both locally and in the event's local_warnings. Only an unusable `action` drops the whole event, since an event that does not say what happened records nothing. Pre-epoch timestamps are rejected because the wire field is unsigned. Tests cover normalization, both delivery drivers, the diagnostics channel, and both clients end to end against a fake transport: 63 new tests.
Live testing against decide.arcjet.com showed a five-event burst followed by flush() arriving as five separate requests, ~10ms apart. The worker saw the flush flag and sent only the event it was already holding, abandoning everything queued behind it — so flush() sent one request per event. "Do not wait" and "do not batch" are different things. On the flush path the worker now stops waiting for new events but still sweeps up the backlog already in the queue, up to batch_size. Same for batch_delay_ms=0, which had the same one-request-per-event behaviour. This mattered most where a backlog is most likely: flush() is the shutdown path, so the old behaviour turned a queued burst into a burst of requests right when the process was trying to exit, with a flush deadline running. The existing test asserted only "fewer requests than events" and passed, because in-process sends complete instantly and the batch window closed before the flush flag was ever observed. The regression test blocks the first send to build a real backlog, which is the condition production hit; it fails with [1, 1, 1, 1, 1, 1] before this change and passes with [1, 5] after. The async driver was already correct — it drains the deque rather than breaking out early.
Real defects found in review:
- Race in the sync client's `_ensure_delivery`. Two threads calling capture()
for the first time could both see `_delivery is None`, each build a
SyncCaptureDelivery and start a worker thread, and one would be overwritten
— orphaning its thread along with any events queued on it, invisible to
flush(). Now double-checked locking, with an unlocked fast path so the
steady state has no lock traffic on the request path.
- The capture() docstrings claimed pre-epoch timestamps drop the event. They
do not: the field falls back to the current time and is reported as a
dropped field. Corrected on both clients.
- AJ1001 was defined twice, in `_capture` and `_diagnostics`. `_diagnostics`
now owns it and `_capture` re-exports, so they cannot drift if it is
renumbered.
- A test asserted `event.action is not None`, which is always true because a
protobuf string field defaults to `""`. It asserted nothing. Now checks
truthiness, and each branch of the outcome is asserted separately.
The race regression test is worth a note. Written the obvious way — threads on
a barrier, then assert a single delivery — it passed 20/20 against the
*unlocked* code, because construction completes before the GIL hands over. It
proved nothing. It now widens the window by sleeping inside the constructor and
counts constructions, which fails 10/10 without the lock ("8 deliveries were
constructed") and passes with it.
Also documented, no behaviour change:
- Cross-loop safety on the async worker: rebinding is safe for loops used one
after another, which is the supported case. Driving one client from two
concurrently running loops is not supported — two workers would pop the same
deque with `_in_flight` split between them. Cancelling the old task cannot
fix it from here, since cancelling a task on another running loop is not
thread-safe either.
- Why `capture()` reserves capacity before the put, and why that ordering is
the safe direction to be wrong for flush().
- Why the try in `normalize_capture_event` deliberately covers the isinstance
check: a hostile metaclass can make isinstance raise, and capture() must not
propagate that.
- The idle-window timeout in the async worker is an expected signal, not an
error, so its except has nothing to handle.
Declined: CodeQL's "statement has no effect" on the Protocol `...` bodies. That
is the PEP 544 convention and what this repo already uses at all twelve
existing sites, including the neighbouring guard() stubs in the same file;
`pass` appears as a protocol body nowhere. The rule fires only on the added
lines, not on the identical pre-existing ones.
e929fd6 to
46cd9bc
Compare
There was a problem hiding this comment.
Arcjet Review — 🟢 Low Risk
Decision: Approved
Rationale: Re-review of a previously Medium-risk PR. Every substantive item from the prior review was either fixed (race in _ensure_delivery, OPTION_DROPPED duplication, misleading docstrings, imprecise test assertion, missing comments on _outstanding ordering, empty except, and event-loop rebinding scope) or answered with a specific rationale I find convincing (per-batch vs per-event batch_delay, wide try in normalize_capture_event, Protocol ... convention). The most important fix — double-checked locking on the sync client's first-capture path — is backed by a regression test that fails 10/10 without the lock, which is much stronger than the naive form of that test. The author also caught and fixed a real batching regression on the sync flush path that I did not flag last time. No new risks introduced; risk is now Low.
Summary of Changes
Re-review of the capture()/flush() feature. All prior review feedback addressed: sync _ensure_delivery race fixed with double-checked locking (with a widened-window regression test), OPTION_DROPPED code deduplicated via _diagnostics, docstrings corrected, comments added around the deliberate wide try/except, _outstanding reserve-before-put ordering, and idle-timeout except. Event-loop rebinding documented as unsupported for concurrent loops. Author also fixed a real sync-flush batching bug (worker was sending one request per event when a flush was pending) with its own regression test.
Notes
Re-review. All previous inline concerns resolved or answered. No new concerns raised on the delta.
Review: 698df792 | Model: anthropic/claude-opus-4-7 | Powered by Arcjet Review
Adds
capture()andflush()to both guard clients.capture()records something the application did, for visibility only: it never affects a decision, never raises, and returns without awaiting on either client.Delivery follows the shape the capture ADRs set out:
flush(timeout_ms)drains queued and in-flight events and drops the remainder on expiry. There is noclose().Every drop is reported through the
arcjetlogger with a stable code — AJ3000 invalid input, AJ3001 queue full, AJ3002 send failed, AJ3003 flush deadline. Nothing is discarded silently. Capture is fire-and-forget, so there is no response to carry a warning back on; without a local channel a drop would be invisible. Diagnostics coalesce per code so a persistent problem cannot turn a best-effort drop into a logging incident, while the accumulated count is still reported.Each event sets
sourceto "sdk". The producer decides this, not the caller, so it is not an argument; the server stores an absent source as NULL, meaning unknown, and never substitutes a default.There are two delivery drivers because arcjet-py has two clients: a daemon worker thread for the sync client and an asyncio task for the async one, so each uses its own transport and neither pays for the other's machinery. The async worker is created on first use and rebinds if the event loop changes, because a client outlives any single loop.
Normalization validates each field independently, so one bad optional value drops only itself and is reported both locally and in the event's local_warnings. Only an unusable
actiondrops the whole event, since an event that does not say what happened records nothing. Pre-epoch timestamps are rejected because the wire field is unsigned.Tests cover normalization, both delivery drivers, the diagnostics channel, and both clients end to end against a fake transport: 63 new tests.