Implement support for replaying events

[zoldar]

Changes

This PR implements support for replaying events via existing ingestion pipeline. Although the current implementation is entirely contained in analytics repository, the eventual solution will be split among analytics and persistor.

The plan for release is following:

• ClickHouse database migration - merge, release and run; the only thing it does is renaming already existing columns in our cluster, which were added when we were recovering lost events from persistor some time ago
• Persistor logic changes - extend the ClickHouse schemas for event and session with replay_session_id field and use a different cache key for replayed events as well in CacheStore; the change will be merged and released as well
• Finally, analytics pipeline changes - a preview app will be created first to test recovery using an already setup staging instance of ingestvcr service; merging the PR will be held off until headers sanitisation will be implemented in our infra to prevent sending replay headers from outside our system

Implementation details

Recording

A simplified diagram of how recording is integrated into the event ingestion pipeline:

When the event payload reach Ingress proxy, the haproxy dispatches its body and headers to the agent process listening on a tcp socket. The agent deserialises the payload, assigns a random ID to the event and asynchronously persists the event in the store while immediately returning the newly assigned ID to the proxy. The proxy appends it as X-Replay-Event-ID header. As of now, it's not used and is ignored down the pipeline.

The analytics ingest pipeline processes the event normally without any additional processing and so does persistor, after which the event ends up batched and inserted into ClickHouse DB.

Replaying

When replaying, each payload will go through the same ingress proxy but it won't be sent to the recording agent on the basis of replay header being already present. The replay utility appends 3 following headers when sending the payloads:

• X-Replay-Event-ID (ignored for now, like in the case of recording)
• X-Replay-Session-ID
• X-Replay-Time

Once the payload reaches ingestion pipeline, it's decoded in the request building phase. There, for replayed event, the event timestamp is replaced with the one provided in X-Replay-Time and X-Replay-Session-ID is put under request.replay_session_id. The replay session ID is then carried over to event attributes.

The put_salts step is crucial in ensuring that replayed events are processed in complete isolation from other events even if some user sessions are still valid from the time of recording. Without that, the existing sessions will get corrupted. Processing replayed events in isolation is a trade-off we have to make - there's no feasible way to retroactively adjust existing sessions, especially somewhere in the middle.

The isolation is achieved by - instead of using the salt from the database - generating a pseudo-random hash on the basis of replay session ID using Plug.Crypto.KeyGenerator and mixing replay session ID with secret key base in a safe way. This brings another challenge, because the hash is generated using PBKDF2 with default number of 1000 iterations, which takes around 1-2 ms when tested locally. Fetching salt from ETS cache takes only a couple microseconds in comparison. Even reducing number of iterations to 1 (which raises safety concerns over risk of exposing secret key base too easily) the timing is still strongly in favor of ETS. To work around that, we use KeyGenerator's built-in ability to cache computations using an ETS table. A separate process manages the table and purges it every 2 hours, just to be safe (though cache table blowing up is unlikely, given session IDs can't be provided externally).

The last step is session management in persistor. Replayed sessions are stored and fetched using distinct shape of cache key, including replay session ID. Although risk of user ID collisions is low, it's still better to be on the safe side - that's one more level of ensuring isolation.

Lastly, when a new replayed session is provisioned, its replay session ID attribute is populated accordingly, like in the case of event.

Recording replay session ID allows relatively easy "rollback" of a botched replay and starting over again. Also, two different replays (with different session IDs), even if overlapping, are easy to tell apart and clean up properly as well. An interrupted session can also be safely resumed (the replay utility allows to provide an existing session ID; otherwise it generates a unique one on every run).

Tests

• Automated tests have been added

[aerosol]

Looks good

[coderabbitai]

Caution

Review failed

An error occurred during the review process. Please try again later.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch event-replay-poc

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Preview environment👷🏼‍♀️🏗️
PR-6491

[ukutaht]

Thanks for the writeup @zoldar! Based on the description do I understand correctly that X-Replay-Session-Id is constant for all events within the same replay run and generated randomly per replay run?

[zoldar]

Thanks for the writeup @zoldar! Based on the description do I understand correctly that X-Replay-Session-Id is constant for all events within the same replay run and generated randomly per replay run?

Correct. That is, unless the session ID is explicitly provided when running replay. This enables resuming interrupted session.

[ukutaht]

Thanks! I don't fully understand why a separate salts mechanism is needed here. From what I can tell its purpose is to ensure that user_ids from live ingestion vs different replay runs do not clash. But can that not be achieved by including the replay_session_id as an input to the user_id hash function?

[zoldar]

@ukutaht That's what I basically did initially when spiking it. However, after some more research, I've concluded this is not going to cut it. In SipHash, providing a secret key with good entropy is crucial. :crypto.strong_rand_bytes meets those requirements. replay_session_id is neither secret (it's persisted verbatim together with event and session) nor it has a good entropy (rand.UInt64 is used for generating it and the docs explicitly state it's pseudo random and not suitable for security-sensitive applications).

Using Plug.Crypto.KeyGenerator adds certain degree of complexity, but I think that in this particular case it's warranted, as ensuring the stats are properly anonymised is one of our core features.

[ukutaht]

So I was thinking of something like:

1. Live ingestion is generated with siphash(salt, user_agent <> ip <> domain)
2. Replay ingestion is generated with siphash(salt, user_agent <> ip <> domain <> replay_session_id) where the salt is the same salt as normal

Is this what you had spiked as well? I understand that a randomly generated session_id is not suitable for cryptographic purposes but that's the job of the salting mechanism we already have. The salt is generated with :crypto.strong_rand_bytes and discarded which provides the anonymity. The string in the second argument is purely provided to manage uniqueness. It's fine to include low-entropy inputs in the second string argument since it is not the part that is responsible for rendering keys irreversible.

I do see the need if you had spiked a version where the replay_session_id takes place of the salt for replayed events:

siphash(replay_session_id, user_agent <> ip <> domain)

In this version I agree the session_id is not strong enough. But then I would ask why not move the session_id to the second argument and keep the usual salt in the first.

[zoldar]

@ukutaht 🤦 Of course at first I've spiked the dumb version and somehow completely missed the possibility of altering hashed value in generate_user_id instead. Yeah, that will make it a bit simpler.

[zoldar]

Ah, okay, I think that has crossed my mind actually but I've discarded the idea because salts could be cycled in the middle of replay. That's a non-issue though. I don't know, either way. That. Was. Dumb. argh

[zoldar]

Simplified with a0579e5 and 5ef8918. Thanks.

[ukutaht]

@zoldar nice, thanks 👍

I've discarded the idea because salts could be cycled in the middle of replay. That's a non-issue though

Hmm yeah the ingestion pipeline is designed in theory to carry sessions over during salt rotation but I'm not sure how well it works in practice tbh.