Skip to content

Fix PostgresSequenceGenerator.check_consistency(...) using stale value after auto-repair#19973

Draft
MadLittleMods wants to merge 2 commits into
developfrom
madlittlemods/auto-repaired-inconsistent-stream-error
Draft

Fix PostgresSequenceGenerator.check_consistency(...) using stale value after auto-repair#19973
MadLittleMods wants to merge 2 commits into
developfrom
madlittlemods/auto-repaired-inconsistent-stream-error

Conversation

@MadLittleMods

@MadLittleMods MadLittleMods commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Fix PostgresSequenceGenerator.check_consistency(...) using stale value after auto-repair

Spawning from a problem that @gaelgatelement ran into #element-backend-internal:matrix.org but I don't think this PR necessarily fixes that problem.

IncorrectDatabaseSetup: Postgres sequence 'device_lists_sequence' is inconsistent with associated stream position of 'device_lists_stream' in the 'stream_positions' table. 
2026-07-16 16:30:12,335 - synapse.config.logger - 377 - WARNING - main - ***** STARTING SERVER *****
2026-07-16 16:30:12,336 - synapse.config.logger - 378 - WARNING - main - Server /usr/local/lib/python3.13/site-packages/synapse/app/homeserver.py version 1.156.0+pro.1
2026-07-16 16:30:12,336 - synapse.config.logger - 383 - WARNING - main - Copyright (c) 2023 New Vector, Inc
2026-07-16 16:30:12,336 - synapse.config.logger - 384 - WARNING - main - Licensed under the AGPL 3.0 license. Website: https://github.com/element-hq/synapse
2026-07-16 16:30:12,336 - synapse.config.logger - 387 - INFO - main - Server hostname: goldstar.element.dev
2026-07-16 16:30:12,336 - synapse.config.logger - 388 - INFO - main - Public Base URL: https://matrix.goldstar.element.dev/
2026-07-16 16:30:12,336 - synapse.config.logger - 389 - INFO - main - Instance name: master
2026-07-16 16:30:12,336 - synapse.config.logger - 390 - INFO - main - Twisted reactor: EPollReactor
2026-07-16 16:30:12,336 - synapse.app.homeserver - 417 - INFO - main - Setting up server
2026-07-16 16:30:12,336 - synapse.server - 635 - INFO - main - Setting up.
2026-07-16 16:30:12,351 - synapse.storage.databases - 92 - INFO - main - [database config 'master']: Checking database server
2026-07-16 16:30:12,352 - synapse.storage.databases - 95 - INFO - main - [database config 'master']: Preparing for databases ['main', 'state']
2026-07-16 16:30:12,352 - synapse.storage.prepare_database - 132 - INFO - main - ['main', 'state']: Checking existing schema version
2026-07-16 16:30:12,354 - synapse.storage.prepare_database - 136 - INFO - main - ['main', 'state']: Existing schema is 94 (+7 deltas)
2026-07-16 16:30:12,354 - synapse.storage.databases.main - 401 - INFO - main - Checking database for consistency with configuration...
2026-07-16 16:30:12,354 - synapse.storage.prepare_database - 430 - INFO - main - Applying schema deltas for v94
2026-07-16 16:30:12,355 - synapse.storage.prepare_database - 565 - INFO - main - Schema now up to date
2026-07-16 16:30:12,356 - synapse.storage.databases - 110 - INFO - main - [database config 'master']: Starting 'main' database
2026-07-16 16:30:12,376 - synapse.storage.util.id_generators - 92 - INFO - main - Initialising stream generator for push_rules(id): 1
2026-07-16 16:30:12,376 - synapse.storage.util.id_generators - 92 - INFO - main - Initialising stream generator for push_rules_enable(id): 1
2026-07-16 16:30:12,382 - synapse.storage.databases.main.event_push_actions - 1238 - INFO - main - Searching for stream ordering 1 month ago
2026-07-16 16:30:12,382 - synapse.storage.databases.main.event_push_actions - 1242 - INFO - main - Found stream ordering 1 month ago: it's 2
2026-07-16 16:30:12,383 - synapse.storage.databases.main.event_push_actions - 1245 - INFO - main - Searching for stream ordering 1 day ago
2026-07-16 16:30:12,383 - synapse.storage.databases.main.event_push_actions - 1249 - INFO - main - Found stream ordering 1 day ago: it's 2
2026-07-16 16:30:12,386 - synapse.storage.util.id_generators - 92 - INFO - main - Initialising stream generator for access_tokens(id): 1
2026-07-16 16:30:12,386 - synapse.storage.util.id_generators - 92 - INFO - main - Initialising stream generator for refresh_tokens(id): 1
2026-07-16 16:30:12,389 - synapse.storage.util.id_generators - 92 - INFO - main - Initialising stream generator for event_reports(id): 1
2026-07-16 16:30:12,390 - synapse.storage.util.id_generators - 92 - INFO - main - Initialising stream generator for room_reports(id): 1
2026-07-16 16:30:12,390 - synapse.storage.util.id_generators - 92 - INFO - main - Initialising stream generator for user_reports(id): 1
2026-07-16 16:30:12,391 - synapse.app._base - 251 - ERROR - main - Exception during startup
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/synapse/app/homeserver.py", line 486, in main
    setup(hs)
    ~~~~~^^^^
  File "/usr/local/lib/python3.13/site-packages/synapse/app/homeserver.py", line 422, in setup
    hs.setup()
    ~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/synapse/server.py", line 637, in setup
    self.datastores = Databases(self.DATASTORE_CLASS, self)
                      ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/__init__.py", line 119, in __init__
    main = main_store_class(database, db_conn, hs)
  File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/main/__init__.py", line 179, in __init__
    super().__init__(database, db_conn, hs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 133, in __init__
    super().__init__(database, db_conn, hs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/main/experimental_features.py", line 44, in __init__
    super().__init__(database, db_conn, hs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/main/devices.py", line 2806, in __init__
    super().__init__(database, db_conn, hs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/main/devices.py", line 110, in __init__
    self._device_list_id_gen = MultiWriterIdGenerator(
                               ~~~~~~~~~~~~~~~~~~~~~~^
        db_conn=db_conn,
        ^^^^^^^^^^^^^^^^
    ...<18 lines>...
        writers=hs.config.worker.writers.device_lists,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/synapse/storage/util/id_generators.py", line 308, in __init__
    self._sequence_gen.check_consistency(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        db_conn,
        ^^^^^^^^
    ...<3 lines>...
        positive=positive,
        ^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
    raise IncorrectDatabaseSetup(
    ...<2 lines>...
    )
synapse.storage.engines._base.IncorrectDatabaseSetup: 
Postgres sequence 'device_lists_sequence' is inconsistent with associated stream position
of 'device_lists_stream' in the 'stream_positions' table.

This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.

A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:

    DELETE FROM stream_positions WHERE stream_name = 'device_lists_stream';

This will need to be done every time the server is restarted.

**********************************************************************************
 Error during initialisation:
     Traceback (most recent call last):
       File "/usr/local/lib/python3.13/site-packages/synapse/app/homeserver.py", line 486, in main
         setup(hs)
         ~~~~~^^^^
       File "/usr/local/lib/python3.13/site-packages/synapse/app/homeserver.py", line 422, in setup
         hs.setup()
         ~~~~~~~~^^
       File "/usr/local/lib/python3.13/site-packages/synapse/server.py", line 637, in setup
         self.datastores = Databases(self.DATASTORE_CLASS, self)
                           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/__init__.py", line 119, in __init__
         main = main_store_class(database, db_conn, hs)
       File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/main/__init__.py", line 179, in __init__
         super().__init__(database, db_conn, hs)
         ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
       File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 133, in __init__
         super().__init__(database, db_conn, hs)
         ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
       File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/main/experimental_features.py", line 44, in __init__
         super().__init__(database, db_conn, hs)
         ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
       File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/main/devices.py", line 2806, in __init__
         super().__init__(database, db_conn, hs)
         ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
       File "/usr/local/lib/python3.13/site-packages/synapse/storage/databases/main/devices.py", line 110, in __init__
         self._device_list_id_gen = MultiWriterIdGenerator(
                                    ~~~~~~~~~~~~~~~~~~~~~~^
             db_conn=db_conn,
             ^^^^^^^^^^^^^^^^
         ...<18 lines>...
             writers=hs.config.worker.writers.device_lists,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         )
         ^
       File "/usr/local/lib/python3.13/site-packages/synapse/storage/util/id_generators.py", line 308, in __init__
         self._sequence_gen.check_consistency(
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
             db_conn,
             ^^^^^^^^
         ...<3 lines>...
             positive=positive,
             ^^^^^^^^^^^^^^^^^^
         )
         ^
       File "/usr/local/lib/python3.13/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
         raise IncorrectDatabaseSetup(
         ...<2 lines>...
         )
     synapse.storage.engines._base.IncorrectDatabaseSetup:
     Postgres sequence 'device_lists_sequence' is inconsistent with associated stream position
     of 'device_lists_stream' in the 'stream_positions' table.
 
     This is likely a programming error and should be reported at
     https://github.com/matrix-org/synapse.
 
     A temporary workaround to fix this error is to shut down Synapse (including
     any and all workers) and run the following SQL:
 
         DELETE FROM stream_positions WHERE stream_name = 'device_lists_stream';
 
     This will need to be done every time the server is restarted.
 
 
 There may be more information in the logs.
**********************************************************************************

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
    • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
  • Code style is correct (run the linters)

@MadLittleMods MadLittleMods added the A-Workers (includes replication) label Jul 16, 2026
@MadLittleMods MadLittleMods changed the title Fix check_consistency using stale value after auto-repair Fix PostgresSequenceGenerator.check_consistency(...) using stale value after auto-repair Jul 16, 2026
MadLittleMods
MadLittleMods commented Jul 16, 2026
View reviewed changes
Comment thread synapse/storage/util/sequence.py
Comment on lines 161 to 164
# If `is_called` is False then `last_value` is actually the value that
# will be generated next, so we decrement to get the true "last value".
if not is_called:
last_value -= 1

@MadLittleMods MadLittleMods Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally separate: We should move this above, next to where we first pull out last_value and is_called

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

A-Workers (includes replication)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MadLittleMods