Skip to content

Reap datalogger connections that are abandoned without a FIN - #13

Open
FezVrasta wants to merge 1 commit into
Herbertmt978:masterfrom
FezVrasta:fix/grottserver-keepalive-accepted-connections
Open

Reap datalogger connections that are abandoned without a FIN#13
FezVrasta wants to merge 1 commit into
Herbertmt978:masterfrom
FezVrasta:fix/grottserver-keepalive-accepted-connections

Conversation

@FezVrasta

Copy link
Copy Markdown

grottserver never closes an idle socket. Its select loop sets only SO_REUSEADDR on the listener and nothing on accepted connections, so a session the datalogger abandons without a FIN stays ESTABLISHED for the lifetime of the process and its send_queuereg entry is never reclaimed.

That would be survivable if dataloggers opened connections freely, but they allocate source ports from a small fixed pool — a ShineLan-X was observed cycling through exactly seven (1029–1035). Every orphan permanently consumes one. Once all seven 4-tuples are held open, the datalogger cannot open a connection at all: it stops reconnecting, stops sending, and the feed goes completely silent.

Evidence

Observed on a ShineLan-X in offline/grottserver mode. After ~15 hours of total silence — no data records, not even a connection attempt — the datalogger reconnected within 15 seconds of a grottserver restart, on three separate occasions:

restart first reconnect
14:39:55Z 14:40:13Z
15:01:36Z 15:01:47Z
16:00:27Z 16:00:42Z

/proc/net/tcp showed the orphaned sockets idle — no queued bytes, no timer, no retransmits:

port   state    tx_queue  rx_queue  timer   when(s)  retrans
1030   ESTAB    0         0         none    0.0      0
1033   ESTAB    0         0         none    0.0      0
1029   ESTAB    0         0         none    0.0      0

Not blocked, not stalled — simply never closed. A restart drops every server-side socket and frees the whole pool at once, which is why a restart appears to fix it and why the silence returns once the pool is spent again.

Fix

Enable SO_KEEPALIVE on accepted connections so the kernel reaps them without operator intervention. With 120 s idle and 3 probes at 30 s, a vanished peer is detected in about 3.5 minutes. The probes are also traffic, so a stateful firewall stops expiring a session that is legitimately quiet between records.

The TCP_* options are Linux-specific and applied through getattr guards, so platforms lacking them still get SO_KEEPALIVE at the system default interval. A failure to set any option is logged and swallowed rather than refusing the connection.

After the change, on the same hardware: Close connection entries appear for the first time, and the pool holds steady at 2 of 7 rather than draining to zero.

port=1031  ESTAB  timer=KEEPALIVE  when=78.9s
port=1030  ESTAB  timer=KEEPALIVE  when=7.2s
held: 2 of 7-port pool

Tests

Five new tests in tests/test_grottserver_keepalive.py cover the option being set, the configured probe timings, the guard for platforms lacking the TCP_* knobs, and an AST check that the accept path actually calls the helper.

No regressions: clean master gives 294 passed, this branch gives 299 passed, with the same pre-existing failures in test_addon_runtime_config.py and test_release_validation.py (the latter requires GitHub API credentials) on both.

Notes

Independent of #12 — the leak is in master regardless of offline mode, so this can merge in either order. The add-on version is deliberately not bumped and the changelog entry is filed under Unreleased, matching the existing release process.

grottserver never closes an idle socket. Its select loop sets only
SO_REUSEADDR on the listener and nothing at all on accepted connections,
so a session the datalogger walks away from without a FIN stays
ESTABLISHED for the lifetime of the process, and its send_queuereg entry
is never reclaimed.

That would be survivable if dataloggers opened connections freely, but
they allocate source ports from a small fixed pool - a ShineLan-X was
observed cycling through exactly seven (1029-1035). Every orphan
permanently consumes one. Once all seven 4-tuples are held open here the
datalogger cannot open a connection at all: it stops reconnecting, stops
sending, and the feed goes completely silent. Restarting grottserver
drops every server-side socket and frees the whole pool at once, which is
why a restart appears to fix it and why the silence returns 30-45 minutes
later once the pool is spent again.

Observed on a ShineLan-X in offline/grottserver mode: after 15 hours of
total silence the datalogger reconnected within 15 seconds of a restart
on three separate occasions, and /proc/net/tcp showed the orphaned
sockets idle with no queued bytes, no timer, and no retransmits - not
blocked, simply never closed.

Enable SO_KEEPALIVE on accepted connections so the kernel reaps them
without operator intervention. With 120s idle and 3 probes at 30s a
vanished peer is detected in about 3.5 minutes. The probes are also
traffic, so a stateful firewall stops expiring a session that is
legitimately quiet between records.

The TCP_* socket options are Linux-specific and applied through getattr
guards, so platforms lacking them still get SO_KEEPALIVE at the system
default interval. A failure to set any option is logged and swallowed
rather than refusing the connection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant