Skip to content

Validate Nitter response content to prune broken instances#642

Merged
dreikanter merged 3 commits into
masterfrom
fix-nitter-instance-content-validation
May 24, 2026
Merged

Validate Nitter response content to prune broken instances#642
dreikanter merged 3 commits into
masterfrom
fix-nitter-instance-content-validation

Conversation

@dreikanter

Copy link
Copy Markdown
Owner

Summary

Fixes the Feedjira::NoParserAvailable errors (538+ occurrences) affecting every Nitter-backed feed.

Root cause: the Nitter subsystem validated only HTTP status, never response content. A Nitter instance returning HTTP 200 with a non-RSS body (an HTML rate-limit/error page) passed both the availability check and the loader's status check. The HTML was then handed to FeedjiraProcessor, raising NoParserAvailable. Because the response was 2xx, register_error never fired, so the broken instance was never suspended — it stayed in the pool and corrupted ~1/N of every Nitter feed's pulls. This is why feeds like neural-machine got fresh posts when they drew a healthy instance but errored when they drew a poison one (instance-level, not account-level).

Fix — validate that the response is actually a parseable feed, in both places:

  • NitterInstanceAvailabilityChecker#available? now requires the sample RSS to parse, so poison instances are marked disabled on pool refresh.
  • NitterLoader now treats unparseable content as an instance error (register_error + raise NitterLoader::Error), so a poison instance that slips through gets suspended after MAX_ERRORS and the failure is correctly attributed and self-heals.
  • New FeedContent.parseable? helper, shared by both.

Healthy instances keep serving the live feeds; broken ones get pruned.

Also disables 4 inactive Nitter feeds. Their X source accounts have produced no content for 16 months to 4 years (verified against freefeed cadence and source research): catscafe, extrafabulous, jakelikesonions, nathanwpyle. Each carries a disabling_reason. The live feeds (neural-machine, warandpeas, yesbut, safelyendangered) stay enabled; their 2h refresh interval is already well within Nitter rate limits.

Checklist

  • Updated CHANGELOG.md under Unreleased, or change is internal-only

@dreikanter
dreikanter merged commit e364695 into master May 24, 2026
6 checks passed
@dreikanter
dreikanter deleted the fix-nitter-instance-content-validation branch May 24, 2026 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant