fix: bound remote pubsub subscriptions to prevent memory exhaustion

[acul71]

Summary

• Fix memory exhaustion / DoS via unbounded remote topic subscriptions (GHSA-4f8r-922h-2vgv).
• Delete empty peer_topics entries on unsubscribe and peer removal (disconnect / blacklist).
• Enforce inbound RPC size (1 MiB) and per-RPC / per-peer subscription limits (5000 / 100 by default).

Fixes #1349

Related

• GHSA-4f8r-922h-2vgv
• Fix memory leak due to retention of empty topic maps go-libp2p-pubsub#710
• Security: GossipSub Memory Exhaustion via Unbounded Remote Topic Subscriptions cpp-libp2p#349

Test plan

• pytest tests/core/pubsub/test_pubsub_subscription_limits.py -v
• pytest tests/core/pubsub/test_pubsub.py::test_handle_subscription -v
• pytest tests/core/pubsub/test_pubsub.py::test_peers_subscribe -v
• make lint && make typecheck && make test && make linux-docs

Made with Cursor

[acul71]

Status update — GHSA-4f8r-922h-2vgv / #1349

cc @seetadev @sumanjeet0012 for review — this addresses the GossipSub subscription-flood memory exhaustion pattern reported across the libp2p ecosystem (js GHSA-4f8r-922h-2vgv, go go-libp2p-pubsub#710, cpp cpp-libp2p#349).

What’s been accomplished

Core mitigations (shared Pubsub layer — applies to GossipSub and FloodSub):

1. Empty topic map cleanup — peer_topics entries are removed when the last peer unsubscribes or disconnects (_remove_peer_from_topic, _clear_peer_from_all_topics), matching go-libp2p-pubsub Implement validation throttler for message validation in Pubsub #710.
2. Inbound RPC size limit — 1 MiB default on pubsub read (read_varint_prefixed_bytes + max_inbound_rpc_size), aligned with outbound DefaultMaxMessageSize.
3. Per-RPC subscription cap — default 5000 entries/RPC (js-libp2p post-15.0.23 default); over-limit RPCs are rejected.
4. Per-peer subscription cap — default 100 distinct topics/peer (go downstream / Filecoin-style limit); excess subscribes are ignored with a warning.
5. Shutdown cleanup — peer_topics and per-peer subscription counts cleared in Pubsub.run() finally (js topics.clear() parity).
6. Optional allowed_topics allowlist — remote subscribes outside the set are ignored (js allowedTopics / go allowlist filter parity).

Tests — tests/core/pubsub/test_pubsub_subscription_limits.py (12 cases), plus updates to test_handle_subscription and test_peers_subscribe. Coverage includes:

• Unsubscribe / disconnect / _handle_dead_peer / blacklist teardown cleanup
• Per-RPC and per-peer limits (reject + at-limit accept)
• Oversized inbound frames
• Idempotent resubscribe counting
• allowed_topics accept/reject/unsubscribe

CI — make lint, make typecheck, make test (2893 passed), make linux-docs all green locally.

What’s still missing / out of scope for this PR

Gap	Notes
PoC-scale regression test	No js-style flood test (~349k unique topics / heap measurement). Current tests use small N; sufficient for CI regression, not a full advisory reproduction.
Full resource manager	#572 remains open; this PR does not implement rcmgr-style global limits.
Score / blacklist on violation	Subscription floods are logged and dropped; no GossipSub scorer penalty or automatic blacklist (deferred for interoperability).
Inbound frame size vs js	py-libp2p caps pubsub RPC at 1 MiB; js advisory references 4 MiB LP frames — still blocks single-frame mega-RPC, but not identical.
Rich subscription filter API	allowed_topics covers the allowlist case; no full go-style SubscriptionFilter plugin interface.

Ecosystem parity (rough)

• go-libp2p-pubsub Implement validation throttler for message validation in Pubsub #710: matched + stronger default limits baked in
• js-libp2p 15.0.23: ~95% — core three defects addressed; PoC test and 4 MiB LP default differ

Happy to adjust default limits, add scorer penalties, or scope a follow-up PoC test if maintainers prefer. Thanks for any guidance.

[acul71]

Status update — rebased onto main (blockers resolved)

cc @seetadev @sumanjeet0012 — branch has been rebased onto current main and force-pushed (--force-with-lease). Sync is now 0 behind / 5 ahead, with no merge conflicts.

What changed vs prior tip

1. Rebased onto main; folded limits/cleanup into main’s _forget_subscription / _forget_all_subscriptions / _replay_recent_messages (kept metrics + mcache replay).
2. Oversize inbound RPC now logs, resets the stream, and calls _handle_dead_peer (no auto-blacklist).
3. Empty topic keys deleted on forget; subscription counts popped at zero.
4. Tests retargeted to _forget_*, assert peer teardown on oversize, condition waits instead of fixed sleeps; updated replay-gate assertions for empty-key deletion.

Validation (local)

• make lint ✅
• make typecheck ✅
• make test ✅ — 3198 passed, 21 skipped
• make linux-docs ✅
• Newsfragment newsfragments/1349.bugfix.rst present (Fixes Security: GossipSub memory exhaustion via unbounded remote topic subscriptions #1349)

Ready for another look when convenient. Thanks!