Introduce support for MSC4429: Profile Updates for Legacy Sync by anoadragon453 · Pull Request #19556 · element-hq/synapse

anoadragon453 · 2026-03-13T20:57:08Z

This PR implements support for MSC4429: Profile Updates for Legacy Sync.

~~Recommended to review commit-by-commit.~~ Commit history has become messy, suggest reviewing as a single item.

Paired with matrix-org/complement#849 and branch https://github.com/matrix-org/sytest/tree/anoa/msc4429

Tracking issue for removing unstable identifiers in Synapse: #19891

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

Copilot

Pull request overview

Implements experimental MSC4429 support by introducing a new profile_updates stream (persisted + replicated) and surfacing profile field changes in legacy /sync when enabled.

Changes:

Add profile_updates stream tracking (DB schema, stream token field, notifier + replication stream plumbing).
Emit MSC4429 profile updates in /sync responses and add filter support for selecting profile fields.
Record profile field updates on profile mutations (set/unset displayname/avatar/custom fields, deactivation cleanup).

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/rest/client/test_rooms.py	Updates hardcoded stream tokens to match the new `StreamToken` shape.
tests/rest/admin/test_room.py	Updates hardcoded stream tokens to match the new `StreamToken` shape.
synapse/types/init.py	Adds `PROFILE_UPDATES` stream key and extends `StreamToken` serialization/parsing.
synapse/streams/events.py	Includes `profile_updates_key` in current/bounded stream tokens.
synapse/storage/schema/main/delta/94/01_profile_updates_seq.sql.postgres	Adds Postgres sequence for the new stream.
synapse/storage/schema/main/delta/94/01_profile_updates.sql	Adds `profile_updates` table + indexes.
synapse/storage/schema/init.py	Bumps schema version to 94 and documents the change.
synapse/storage/databases/main/profile.py	Adds profile update persistence/query APIs and stream ID generator wiring.
synapse/rest/client/versions.py	Advertises `org.matrix.msc4429` in `unstable_features`.
synapse/rest/client/sync.py	Encodes MSC4429 profile update payload into `/sync` response.
synapse/replication/tcp/streams/_base.py	Adds `ProfileUpdatesStream` replication stream.
synapse/replication/tcp/streams/init.py	Registers `ProfileUpdatesStream` for replication.
synapse/replication/tcp/handler.py	Replicates `ProfileUpdatesStream` from configured writers.
synapse/replication/tcp/client.py	Handles incoming profile update replication and notifies listeners.
synapse/notifier.py	Allows `StreamKeyType.PROFILE_UPDATES` to wake `/sync` waiters.
synapse/handlers/sync.py	Generates initial + incremental MSC4429 profile update sections in `/sync`.
synapse/handlers/profile.py	Records profile update markers on profile mutations and notifies.
synapse/config/experimental.py	Adds `msc4429_enabled` config flag.
synapse/api/filtering.py	Adds `/sync` filter support for selecting profile fields (MSC4429-gated).
docker/complement/conf/workers-shared-extra.yaml.j2	Enables MSC4429 in complement worker test config.
changelog.d/19556.feature	Adds changelog entry for MSC4429 experimental support.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Sytest changes for element-hq/synapse#19556

We create a stream table dedicated to tracking profile updates. The sync machinary can this stream to understand whether a profile update needs to be included in a client's sync response.

Add a new stream type for profile updates. This allows sync processes to determine which profile updates a given user has or hasn't seen yet.

Replicate changes to the profile updates stream so that sync workers know there's a new profile update, and wake up sync waiters accordingly. Updates are keyed by `room_id` - sync waiters should only wake up if a user they share a room with updates their profile.

Based on the provided since token and filter.

And update relevant documentation/integration test config.

@user

Prevent the `ProfileRestServlet` from handling requests intended for the `ProfileFieldRestServlet`. If a requester tried to call PUT `.../profile/@user:domain/(key_id?)` a worker that did not mount `ProfileFieldRestServlet`, they would then receive a `405 Method Not Allowed` instead of the expected `404 Not Found`.

Copilot

Pull request overview

Copilot reviewed 52 out of 52 changed files in this pull request and generated 4 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…_room_txn`

anoadragon453

LGTM otherwise (but I can't approve my own PRs).

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

reivilibre

(haven't finished my review but reached end of day; midway through synapse/handlers/sync.py)

reivilibre · 2026-07-15T16:33:47Z

+        if stream_id and profile_update_targets["rooms"]:
+            self._notifier.on_new_event(
+                StreamKeyType.PROFILE_UPDATES,
+                stream_id,


Hmmm. I think there's actually a problem here, but luckily it seems harmless for the current streams that seem to be doing it wrong: as far as I can tell, the returned stream_id is the highest stream ID of those just persisted by this transaction.

But that doesn't guarantee that the stream is actually at that position yet and I think on_new_event actually wants the actual new position as its new_token. (This only gets used for real purposes for a small handful of stream types)

I think this is supposed to rather be the get_current_token.

will track the other streams doing this wrong at #19969

Wonder what happens in a 2-writer setup; does on_new_event get triggered locally AND by the stream following? :S

Is it ok if we leave this for #19969 to handle, given there seems to be a bit of uncertainty involved?

reivilibre · 2026-07-15T17:02:59Z

+        # ExpiringCache((User, Device))
+        #   -> LruCache(
+        #       sha256(Other User ID + Field Name + Field value) -> bool
+        #   )
+        self.lazy_loaded_profile_fields_cache: ExpiringCache[
+            tuple[str, str | None], LruCache[str, bool]
+        ] = ExpiringCache(


Hmm, a few concerns with this cache:

if I set my field (1) A=1, then (2) A=2, then (3) A=1 again, does my 3rd update get skipped because the hash matches an existing entry in cache? It seems you'd have to rekey this cache like sha256(other_user_id || field_name) -> sha256(field value)

we could store the hash bytes as bytes for 50% memory savings :)

blake2b might be a preferable choice of hash (constraining ourselves to the hashlib) for performance whilst retaining the collision resistance property; I think an output size of 16 bytes (128b) would be fine (whilst saving us memory) and since it's a keyed hash, we can generate a random key on every startup (e.g. as a constant in this file) so that it's not externally predictable.

Separately, there is a doubt in my mind: what happens if someone repeats a sync exactly, because the HTTP response gets lost and so the request gets retried?
It sounds like by that point, the cache has already been updated and therefore the profile update is lost from the second sync.
Most of the time we (most deployments) have a ResponseCache so for 2 minutes the repeated sync will be fine, but 'hmm'.
If this is the same as what lazy_loaded_members_cache does then maybe we add a TODO: comment + issue and move on since that by itself might be a bigger problem.

Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>

Instead of passing in a previous membership or a list of target users down in the event internal metadata, use the state delta (or rather, previously calculated sliding sync table changes based on state delta) to figure out if a membership really changed. Then add the member lookups to get the right users to update for. Somewhat WIP, a few tests still fail and this commit misses a bunch of docstrings and/or cleanups of old functionality.

* Should also notify on KNOCK and INVITE (though calling them a join is a bit poor) * Remove unnecessary fetch of user rooms as we only need to notify the members of the current room in question

The MSC indicates we should send down profiles for room members when someone joins a room, but it's a bit of a question right now on how to do this.

…deactivation` We'll anyway have "left room" actions generated for the user for everyone who is interested, as the user parter pushes them out of the room. Thus we would be doing unnecessary extra work here.

We now calculate the targets when persisting the updates.

jaywink · 2026-07-17T14:09:25Z


+        if self._msc4429_enabled:
+            # Handle changes to the profile updates stream.
+            # We've already done a bunch of work calculating the changes needed


As discussed elsewhere, we should, assuming this is kept in this pr, track figuring out this properly in further pull requests, specifically:

is this actually catching all the possible situations, especially regarding a user being removed from a room?

can we get rid of piggybacking on the sliding sync table change results, since it's kind of ugly, and instead refactor both to use some common functionality?

Add experimental feature flag msc4429_enabled

632c83e

anoadragon453 requested a review from Copilot March 17, 2026 11:56

Copilot started reviewing on behalf of anoadragon453 March 17, 2026 11:57 View session

Copilot AI reviewed Mar 17, 2026

View reviewed changes

Comment thread synapse/replication/tcp/client.py Outdated

Comment thread synapse/storage/schema/main/delta/94/01_profile_updates.sql Outdated

Comment thread synapse/storage/databases/main/profile.py Outdated

Comment thread synapse/handlers/sync.py Outdated

anoadragon453 force-pushed the anoa/msc4429 branch 2 times, most recently from 83e699f to a846945 Compare March 17, 2026 16:21

anoadragon453 added a commit to matrix-org/sytest that referenced this pull request Mar 17, 2026

Allow stream_writer to write to the profile_updates stream

0b0ab30

Sytest changes for element-hq/synapse#19556

anoadragon453 added a commit to matrix-org/sytest that referenced this pull request Mar 17, 2026

Allow stream_writer to write to the profile_updates stream

ffdc272

Sytest changes for element-hq/synapse#19556

anoadragon453 mentioned this pull request Mar 17, 2026

Allow stream_writer to write to the profile_updates stream matrix-org/sytest#1420

Closed

anoadragon453 force-pushed the anoa/msc4429 branch from dc8e9f8 to a309372 Compare March 18, 2026 15:58

anoadragon453 marked this pull request as ready for review March 18, 2026 15:58

anoadragon453 requested a review from a team as a code owner March 18, 2026 15:58

MadLittleMods added the A-Profiles label Mar 18, 2026

anoadragon453 added 10 commits March 24, 2026 13:10

Schema for new profile updates stream table

d0b616d

We create a stream table dedicated to tracking profile updates. The sync machinary can this stream to understand whether a profile update needs to be included in a client's sync response.

Add profile updates stream

63cfbe7

Add a new stream type for profile updates. This allows sync processes to determine which profile updates a given user has or hasn't seen yet.

Implement replication

e45045d

Replicate changes to the profile updates stream so that sync workers know there's a new profile update, and wake up sync waiters accordingly. Updates are keyed by `room_id` - sync waiters should only wake up if a user they share a room with updates their profile.

Allow fetching latest changes via /sync and filtering

d9adfd8

Based on the provided since token and filter.

Add msc4429_enabled to Complement tests

30e8737

newsfile

cfb7034

Update unit tests with new token format

5b4d505

Constrain profile endpoint handling to profile_updates writer workers

86572c6

And update relevant documentation/integration test config.

Correct the Complement worker endpoint configuration

f27ca70

anoadragon453 force-pushed the anoa/msc4429 branch from a309372 to f27ca70 Compare March 24, 2026 13:11

anoadragon453 mentioned this pull request Mar 25, 2026

Remove redacted_because from internal unsigned. #19581

Merged

Half-Shot mentioned this pull request Mar 31, 2026

MSC4429: Profile Updates for Legacy Sync matrix-org/matrix-spec-proposals#4429

Open

MadLittleMods added the A-Sync label Apr 6, 2026

reivilibre requested review from reivilibre and removed request for a team April 7, 2026 16:20

erikjohnston reviewed Apr 9, 2026

View reviewed changes

Comment thread docs/upgrade.md Outdated

Comment thread synapse/handlers/sync.py Outdated

Comment thread synapse/handlers/sync.py Outdated

anoadragon453 requested a review from Copilot July 14, 2026 21:55

Copilot started reviewing on behalf of anoadragon453 July 14, 2026 21:56 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread synapse/storage/databases/main/profile.py Outdated

Comment thread synapse/storage/databases/main/profile.py Outdated

Comment thread synapse/handlers/sync.py

Comment thread synapse/config/workers.py Outdated

jaywink and others added 3 commits July 15, 2026 11:42

Apply suggestions from code review

3824af8

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Lint

34ce70f

Remove unnecessary select from `_record_profile_updates_for_user_left…

cbe7c81

…_room_txn`

anoadragon453 commented Jul 15, 2026

View reviewed changes

Comment thread synapse/config/workers.py Outdated

Update synapse/config/workers.py

edc8624

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

reivilibre requested changes Jul 15, 2026

View reviewed changes

jaywink and others added 13 commits July 15, 2026 23:13

Apply suggestions from code review

f31d9fb

Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>

Merge remote-tracking branch 'origin/anoa/msc4429' into anoa/msc4429

7afa195

Fix test_sync_response_always_includes_the_user_themselves

cb9e1e5

Run linter on all files

865273c

Revert Event.internal_metadata changes from 9b6ff35

a01fc2e

Cleanup and docstring updates

5cb6d2c

Merge remote-tracking branch 'origin/develop' into anoa/msc4429

0e840eb

Fix up profile updates for memberhip changes

489539f

* Should also notify on KNOCK and INVITE (though calling them a join is a bit poor) * Remove unnecessary fetch of user rooms as we only need to notify the members of the current room in question

Add a test covering invite/knock

5e7475d

Add a failing test for covering members of a room joined

8b0be80

The MSC indicates we should send down profiles for room members when someone joins a room, but it's a bit of a question right now on how to do this.

Remove writing to the profile updates stream in `delete_profile_upon_…

9deebfb

…deactivation` We'll anyway have "left room" actions generated for the user for everyone who is interested, as the user parter pushes them out of the room. Thus we would be doing unnecessary extra work here.

Remove get_targets_for_profile_updates

2d84759

We now calculate the targets when persisting the updates.

jaywink reviewed Jul 17, 2026

View reviewed changes

jaywink added 6 commits July 20, 2026 14:36

No need to replicate delete_profile_upon_deactivation call anymore

775e701

Remove unused fields in profile replication

0aa4d3f

Fix docstrings in ProfileUpdatesStream and ProfileUpdatesStreamRow

055fcac

Drop list copy

51f6482

Reformat a bunch of SQL in main/profile for consistency

c0eb236

Fix comment re SQLite and JSON

12d4f2d

jaywink mentioned this pull request Jul 20, 2026

Profile updates stream further improvements #19981

Open

Uh oh!

Conversation

anoadragon453 commented Mar 13, 2026 • edited by jaywink Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anoadragon453 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

reivilibre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

reivilibre Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

jaywink Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

reivilibre Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

jaywink Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

anoadragon453 commented Mar 13, 2026 •

edited by jaywink

Loading

jaywink Jul 17, 2026 •

edited

Loading