Skip to content

Exit cleanly on SIGTERM instead of crashing during teardown - #207

Merged
willyborankin merged 1 commit into
Aiven-Open:masterfrom
mbertheau:mbertheau-shutdown-double-unregister
Aug 25, 2026
Merged

Exit cleanly on SIGTERM instead of crashing during teardown#207
willyborankin merged 1 commit into
Aiven-Open:masterfrom
mbertheau:mbertheau-shutdown-double-unregister

Conversation

@mbertheau

Copy link
Copy Markdown
Contributor

Stopping journalpump killed it with a traceback and exit code 1, which systemd records as a service failure:

KeyError: 10
  journalpump.py run -> journalpump.py _close_stale_readers
  -> journalpump.py unregister_from_poll -> select.poll.unregister

unregister_from_poll treated "fd is a key in reader_by_fd" as "fd is still in the poller". Unregistering writes _STALE_FD and leaves the key; only the run loop deletes it, after the poll results are processed. shutdown() unregisters every reader and also queues it stale, and _close_stale_readers() unregisters that queue again, so a SIGTERM arriving after the loop already purged its markers makes the second unregister raise.

Keying the guard on ownership (get(fd) is reader) makes unregistering idempotent, and leaves alone an fd number that has meanwhile been recycled by a different reader. The regression test drives the real run loop and raises SIGTERM at the end of the first iteration, where the reported traceback places it.

Stacked on #206, so until that merges the diff here carries its commits too. The fix itself is the single commit a8da352.

Made with Cursor

Stopping journalpump killed it with a traceback and exit code 1, which
systemd records as a service failure:

    KeyError: 10
      journalpump.py run -> journalpump.py _close_stale_readers
      -> journalpump.py unregister_from_poll -> select.poll.unregister

unregister_from_poll treated "fd is a key in reader_by_fd" as "fd is
still in the poller". Unregistering writes _STALE_FD and leaves the
key; only the run loop deletes it after processing poll results.
shutdown() unregisters every reader and queues it stale;
_close_stale_readers() unregisters the queue again. If SIGTERM arrives
after the loop already purged those markers, the second unregister
raises KeyError and systemd records exit 1.

Key the guard on ownership (`get(fd) is reader`) so a stale marker no
longer looks registered, and a recycled fd belonging to a different
reader is left alone. Deleting the key would also make the guard
honest, but the marker is how the loop tells "unregistered during this
batch" from "never ours". Dropping the second unregister would satisfy
today's callers and leave the trap armed.

The test raises SIGTERM at the end of the first iteration, where the
traceback placed it.

Co-authored-by: Cursor <cursoragent@cursor.com>
@mbertheau
mbertheau force-pushed the mbertheau-shutdown-double-unregister branch from a8da352 to 7a8522d Compare August 25, 2026 07:54
@mbertheau
mbertheau marked this pull request as ready for review August 25, 2026 07:55
@willyborankin
willyborankin merged commit f4f67f0 into Aiven-Open:master Aug 25, 2026
6 checks passed
@mbertheau
mbertheau deleted the mbertheau-shutdown-double-unregister branch September 7, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants