Validate Nitter response content to prune broken instances#642
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the
Feedjira::NoParserAvailableerrors (538+ occurrences) affecting every Nitter-backed feed.Root cause: the Nitter subsystem validated only HTTP status, never response content. A Nitter instance returning HTTP 200 with a non-RSS body (an HTML rate-limit/error page) passed both the availability check and the loader's status check. The HTML was then handed to
FeedjiraProcessor, raisingNoParserAvailable. Because the response was 2xx,register_errornever fired, so the broken instance was never suspended — it stayed in the pool and corrupted ~1/N of every Nitter feed's pulls. This is why feeds likeneural-machinegot fresh posts when they drew a healthy instance but errored when they drew a poison one (instance-level, not account-level).Fix — validate that the response is actually a parseable feed, in both places:
NitterInstanceAvailabilityChecker#available?now requires the sample RSS to parse, so poison instances are markeddisabledon pool refresh.NitterLoadernow treats unparseable content as an instance error (register_error+ raiseNitterLoader::Error), so a poison instance that slips through gets suspended afterMAX_ERRORSand the failure is correctly attributed and self-heals.FeedContent.parseable?helper, shared by both.Healthy instances keep serving the live feeds; broken ones get pruned.
Also disables 4 inactive Nitter feeds. Their X source accounts have produced no content for 16 months to 4 years (verified against freefeed cadence and source research):
catscafe,extrafabulous,jakelikesonions,nathanwpyle. Each carries adisabling_reason. The live feeds (neural-machine,warandpeas,yesbut,safelyendangered) stay enabled; their 2h refresh interval is already well within Nitter rate limits.Checklist
CHANGELOG.mdunderUnreleased, or change is internal-only