Reap datalogger connections that are abandoned without a FIN - #13
Open
FezVrasta wants to merge 1 commit into
Open
Reap datalogger connections that are abandoned without a FIN#13FezVrasta wants to merge 1 commit into
FezVrasta wants to merge 1 commit into
Conversation
grottserver never closes an idle socket. Its select loop sets only SO_REUSEADDR on the listener and nothing at all on accepted connections, so a session the datalogger walks away from without a FIN stays ESTABLISHED for the lifetime of the process, and its send_queuereg entry is never reclaimed. That would be survivable if dataloggers opened connections freely, but they allocate source ports from a small fixed pool - a ShineLan-X was observed cycling through exactly seven (1029-1035). Every orphan permanently consumes one. Once all seven 4-tuples are held open here the datalogger cannot open a connection at all: it stops reconnecting, stops sending, and the feed goes completely silent. Restarting grottserver drops every server-side socket and frees the whole pool at once, which is why a restart appears to fix it and why the silence returns 30-45 minutes later once the pool is spent again. Observed on a ShineLan-X in offline/grottserver mode: after 15 hours of total silence the datalogger reconnected within 15 seconds of a restart on three separate occasions, and /proc/net/tcp showed the orphaned sockets idle with no queued bytes, no timer, and no retransmits - not blocked, simply never closed. Enable SO_KEEPALIVE on accepted connections so the kernel reaps them without operator intervention. With 120s idle and 3 probes at 30s a vanished peer is detected in about 3.5 minutes. The probes are also traffic, so a stateful firewall stops expiring a session that is legitimately quiet between records. The TCP_* socket options are Linux-specific and applied through getattr guards, so platforms lacking them still get SO_KEEPALIVE at the system default interval. A failure to set any option is logged and swallowed rather than refusing the connection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
grottservernever closes an idle socket. Its select loop sets onlySO_REUSEADDRon the listener and nothing on accepted connections, so a session the datalogger abandons without a FIN staysESTABLISHEDfor the lifetime of the process and itssend_queueregentry is never reclaimed.That would be survivable if dataloggers opened connections freely, but they allocate source ports from a small fixed pool — a ShineLan-X was observed cycling through exactly seven (1029–1035). Every orphan permanently consumes one. Once all seven 4-tuples are held open, the datalogger cannot open a connection at all: it stops reconnecting, stops sending, and the feed goes completely silent.
Evidence
Observed on a ShineLan-X in offline/
grottservermode. After ~15 hours of total silence — no data records, not even a connection attempt — the datalogger reconnected within 15 seconds of agrottserverrestart, on three separate occasions:/proc/net/tcpshowed the orphaned sockets idle — no queued bytes, no timer, no retransmits:Not blocked, not stalled — simply never closed. A restart drops every server-side socket and frees the whole pool at once, which is why a restart appears to fix it and why the silence returns once the pool is spent again.
Fix
Enable
SO_KEEPALIVEon accepted connections so the kernel reaps them without operator intervention. With 120 s idle and 3 probes at 30 s, a vanished peer is detected in about 3.5 minutes. The probes are also traffic, so a stateful firewall stops expiring a session that is legitimately quiet between records.The
TCP_*options are Linux-specific and applied throughgetattrguards, so platforms lacking them still getSO_KEEPALIVEat the system default interval. A failure to set any option is logged and swallowed rather than refusing the connection.After the change, on the same hardware:
Close connectionentries appear for the first time, and the pool holds steady at 2 of 7 rather than draining to zero.Tests
Five new tests in
tests/test_grottserver_keepalive.pycover the option being set, the configured probe timings, the guard for platforms lacking theTCP_*knobs, and an AST check that the accept path actually calls the helper.No regressions: clean
mastergives 294 passed, this branch gives 299 passed, with the same pre-existing failures intest_addon_runtime_config.pyandtest_release_validation.py(the latter requires GitHub API credentials) on both.Notes
Independent of #12 — the leak is in
masterregardless of offline mode, so this can merge in either order. The add-on version is deliberately not bumped and the changelog entry is filed under Unreleased, matching the existing release process.