Skip to content

Sweep every stale socket on the port, and only when the bind says so - #110

Merged
mgrossmann merged 1 commit into
mainfrom
issue-109-stale-port-sweep
Aug 22, 2026
Merged

Sweep every stale socket on the port, and only when the bind says so#110
mgrossmann merged 1 commit into
mainfrom
issue-109-stale-port-sweep

Conversation

@mgrossmann

@mgrossmann mgrossmann commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Closes #109.

The stale-port sweep in socket_thread() stopped at the first match and ran in
front of every bind, whether or not the bind would have succeeded. Both are
fixed; the misleading comment above it is replaced with what the socket table
actually is.

What changed

Bind first, sweep only on EADDRINUSE. The sweep is a recovery step for a
port that is demonstrably blocked, not something a healthy start walks through.
It used to run before every bind, where the only thing it could do was close a
socket belonging to something alive on that port.

Close every match. A dead FTPD leaves the listener and every connection it
had accepted -- an accepted socket's local address is the listen port too, so
getsockname() matches it -- and each one holds the port. One FTPD053I per
socket, plus a summary line from two upwards, then one 2 second settle (not per
socket) and an immediate rebind. The 10 second retry stays as the fallback.

No bind failure reported on a start that recovers. FTPD051E moved behind
the sweep, so a successful recovery shows the FTPD053I lines and FTPD054I and
nothing alarming in front of them. errno is captured immediately after each
bind(), since getsockname() and closesocket() in the sweep sit between the
failure and the message reporting it.

The caller's own descriptor is skipped. This used to run before socket()
had been called; it now runs with a live socket the caller is about to bind, and
a failed bind() may still leave that descriptor in the table carrying a local
port.

The bound address is compared, not just the port. SRVBIND can name one
interface (#76), and a leftover socket on a different one cannot be what made our
bind fail. INADDR_ANY collides in both directions. The rule is stated once in
ftpd_adr_conflicts() (ftpd#adr.c) so it can be tested.

The comment. It claimed the fd table is per address space and that a new STC
inherits fds from a prior run of the same jobname. Neither is true: the X'75'
table lives in Hercules, global to the emulator. That is why a dead STC's
leftovers are reachable at all, and why the sweep has to be bounded. What bounds
it is the ENQ FTPD.PORT.nnnnn (no second FTPD on this port) plus EADDRINUSE
(the port really is blocked). Beyond that it stays indiscriminate, because the
table exposes no owner -- a live listener on this port that is not an FTPD does
lose its socket. That boundary is now written into the code rather than implied.

Relation to HTTPD

Same defect, and deliberately the same shape as the fix mvslovers/httpd#224 just
merged (mvslovers/httpd@2f2c41c): bind first, sweep on EADDRINUSE only, close
all, skip own descriptor, 2 second settle, existing retry as fallback, errno
captured early.

Two FTPD-only additions. The address comparison has no HTTPD counterpart --
do_bind() there binds INADDR_ANY unconditionally, so there is nothing to
compare. And the summary line, which HTTPD does not have.

htons() here versus HTTPD's bare in->sin_port == port is not a difference:
libc370 defines htons/ntohl as (x) on big-endian S/370.

Verification

Host: make clean, make test-host 78/78 (TSTADR 35, six of them new for
ftpd_adr_conflicts()). Note the project cflags in project.toml carry
neither -Wall nor -Werror, contrary to the root CLAUDE.md -- CI does not
enforce them. Both changed sources were compiled by hand with -Wall -Werror,
clean.

Run on MVS (mvsdev, MVS/CE, 2026-08-22.) Throwaway STC on port 2122 from the
mbt staging library, so the live FTPD on 2121 and FTPD.LINKLIB were untouched.
Same scenario twice, once against each build: start, connect 3 clients
(FTPD015I ACTIVE SESSIONS: 3 / 10), C FTPDT, restart.

The leak was confirmed independently before each restart -- address space gone
from D A,L, yet a TCP connect to 2122 still succeeded with no banner (listener
alive in Hercules, nothing in MVS accepting) and all three accepted connections
still ESTABLISHED in host netstat. Four stale sockets, not one.

With the fix (179cbac):

FTPD053I CLOSING STALE SOCKET 3 ON 192.168.0.233 PORT 2122
FTPD053I CLOSING STALE SOCKET 4 ON 0.0.0.0 PORT 2122
FTPD053I CLOSING STALE SOCKET 5 ON 192.168.0.233 PORT 2122
FTPD053I CLOSING STALE SOCKET 6 ON 192.168.0.233 PORT 2122
FTPD053I 4 STALE SOCKETS CLOSED ON PORT 2122
FTPD054I LISTENING ON ANY PORT 2122

Under a second, no FTPD051E, no FTPD051I EADDRINUSE, RETRYING IN 10S.
Socket 4 is the listener on 0.0.0.0 (SRVBIND=ANY); 3, 5 and 6 are the accepted
connections on the interface address -- so getsockname() does fill sin_addr
for another address space's sockets. A first start on a free port swept nothing.

Without the fix (c63024d, the current FTPD.LINKLIB), identical scenario:

FTPD053I CLOSING STALE SOCKET 3 ON PORT 2122
FTPD051E BIND() FAILED ON ANY PORT 2122, ERRNO=48
FTPD051I EADDRINUSE, RETRYING IN 10S
FTPD051E BIND() RETRY FAILED, ERRNO=48

It closed one socket -- an accepted connection, not the listener -- then break,
and never listened at all. Note the first match is not the listener, which is
what makes stopping at it useless.

Restarting the fixed build afterwards swept exactly the three sockets the old
one had left (4, 5, 6) and came up. /P then left 2122 refusing
connections, so a clean shutdown closes the listener properly.

Not exercised on MVS: ftpd_adr_conflicts() only ran with want == 0
(SRVBIND=ANY), where everything on the port collides. The specific-interface
branch -- skipping a socket bound to a different address -- is covered by TSTADR
only.

Test STC proc and config data set were deleted afterwards; port 2122 refuses
connections and the live FTPD on 2121 answers normally.

The sweep in socket_thread() stopped at the first match.  An FTPD that ended
without closing its sockets leaves behind the listener AND every connection it
had accepted -- an accepted socket's local address is the listen port too, so
getsockname() matches it -- and each one keeps the port occupied.  Closing one
was almost never enough.

It also ran unconditionally in front of every bind, even one that would have
succeeded, where all it could do was close a socket belonging to something
alive on that port.  Now it runs only after bind() actually failed with
EADDRINUSE: every match is closed, one FTPD053I each, then a 2 second settle
and one immediate rebind, with the 10 second retry kept as the fallback rather
than the first resort.

A start that recovers this way reports no bind failure at all.  FTPD051E moved
behind the sweep, so the console shows the FTPD053I lines and FTPD054I -- an
E-message in front of a successful start only alarms.  errno is captured
immediately after each bind(), because getsockname() and closesocket() in the
sweep sit between the failure and the message that reports it, and either
clobbers it.

close_stale_port() takes the caller's own descriptor and skips it.  That
matters now in a way it did not before: the sweep used to run before socket()
had been called, and it now runs with a live socket the caller is about to
bind.  A failed bind() may still leave that descriptor in the table carrying a
local port, and closing it would leave the caller binding a closed socket.

The bound address is compared too, not just the port.  SRVBIND can name one
interface (#76), and a leftover socket on a different one cannot be what made
our bind fail -- it is not ours to close.  INADDR_ANY collides in both
directions, which ftpd_adr_conflicts() in ftpd#adr.c states once so the rule is
testable: TSTADR covers ANY over ANY, ANY against one interface either way, and
two different interfaces.

The comment above the sweep claimed the socket fd table is per address space
and that a new STC inherits fds from a prior run of the same jobname.  Neither
is true.  The X'75' table lives in Hercules and is global to the emulator,
which is exactly why a dead STC's leftovers are reachable here at all -- and
why the sweep has to be bounded.  What bounds it is the pair of guards around
it: the startup ENQ FTPD.PORT.nnnnn rules out another FTPD on this port, and
EADDRINUSE rules out a port that was not actually blocked.  It stays
indiscriminate about ownership beyond that, because the table exposes no owner:
a live listener on this port that is not an FTPD does lose its socket.

Same defect and same shape as mvslovers/httpd#224.

Closes #109

Claude-Session: https://claude.ai/code/session_01A6q4hLaokSJBeifJDkNJtH
@mgrossmann
mgrossmann merged commit c739353 into main Aug 22, 2026
1 check passed
@mgrossmann
mgrossmann deleted the issue-109-stale-port-sweep branch August 22, 2026 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stale port sweep closes only one socket and runs unconditionally Stale listen port recovery closes only one socket, and runs at the wrong time

1 participant