`Gauge` callbacks aren't thread safe

[MadLittleMods]

Problem

RuntimeError: dictionary changed size during iteration

Traceback (most recent call last):
  File "/home/synapse/.local/share/uv/python/cpython-3.14.6-linux-x86_64_v2-gnu/lib/python3.14/wsgiref/handlers.py", line 137, in run
    self.result = application(self.environ, self.start_response)
                  ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/exposition.py", line 158, in prometheus_app
    status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression)
                              ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/exposition.py", line 122, in _bake_output
    output = encoder(registry)
  File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/openmetrics/exposition.py", line 61, in generate_latest
    for metric in registry.collect():
                  ~~~~~~~~~~~~~~~~^^
  File "/home/synapse/src/synapse/metrics/__init__.py", line 145, in collect
    for metric in REGISTRY.collect():
                  ~~~~~~~~~~~~~~~~^^
  File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/registry.py", line 94, in collect
    yield from collector.collect()
               ~~~~~~~~~~~~~~~~~^^
  File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 91, in collect
    for suffix, labels, value, timestamp, exemplar, native_histogram_value in self._samples():
                                                                              ~~~~~~~~~~~~~^^
  File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 275, in _multi_samples
    for suffix, sample_labels, value, timestamp, exemplar, native_histogram_value in metric._samples():
                                                                                     ~~~~~~~~~~~~~~~^^
  File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 268, in _samples
    return self._child_samples()
           ~~~~~~~~~~~~~~~~~~~^^
  File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 497, in samples
    return (Sample('', {}, float(f()), None, None),)
                                 ~^^
  File "/home/synapse/src/synapse/util/batching_queue.py", line 121, in <lambda>
    ).set_function(lambda: sum(len(q) for q in self._next_values.values()))
                           ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/synapse/src/synapse/util/batching_queue.py", line 121, in <genexpr>
    ).set_function(lambda: sum(len(q) for q in self._next_values.values()))
                                               ~~~~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: dictionary changed size during iteration

Relevant code:

synapse/synapse/util/batching_queue.py

Lines 119 to 121 in 1979fcc

	number_queued.labels(
	name=self._name, **{SERVER_NAME_LABEL: self.server_name}
	).set_function(lambda: sum(len(q) for q in self._next_values.values()))

Spotted because these issues popped up in Sentry but they just show some pieces of the error stack,

• https://sentry.tools.element.io/organizations/element/issues/11253798/
• https://sentry.tools.element.io/organizations/element/issues/11253797/
• https://sentry.tools.element.io/organizations/element/issues/11253795/

Then dove into the logs from matrix.org

Potential suspects

Same as #18764

I'm guessing the actual problem occurs when using the dedicated metrics listener type which runs on a different thread. So when we collect from the other thread, sometimes we collide with the dictionary changing size on the main thread.

Potential solutions

We probably need to use threading.Lock() like we do for InFlightGauge