WordCounter - Paste a link throws 500 error but also many sites prevent this from working

[kanarinka]

Pasting a link seems to throw a 500 internal server error with SOME web pages but not others.

For example:

DOES WORK
Frankenstein from the Gutenberg project- https://www.gutenberg.org/cache/epub/84/pg84.txt
Blog post https://www.stephendiehl.com/posts/ai_bubble/

DOES NOT WORK
https://en.wikipedia.org/wiki/Manifesto_of_Futurism

• Wikipedia links seem inconsistent, sometimes 500 error and sometimes seems like we are getting a message from them to not scrape their content

Generally speaking we are getting way more "get out of here, bot scraper" type of messages that make this function less useful. Even Ethan's blog for example does this - https://ethanzuckerman.com/2026/01/21/where-should-we-live-a-new-class-on-cities-migration-and-climate-change/

We might want to consider removing this functionality? @rahulbot what do you think?

[jsaikali]

I was able to fix this for Wikipedia by adding headers (this fix will go out this week). The issue was that some sites don't like unsigned bots. With the contact information it seems to work for Wikipedia

I.e. code including a header:

def download_webpage(url: str) -> dict:
    """
    Fetch a webpage and return a dict with keys 'title' and 'text' using readability.
    """
    try:
        # Headers required to avoid 403 from some sites; identify as a bot with contact info
        headers = {
            "User-Agent": "DataBasicWordCounter/1.0 (contact: rahul@databasic.io)"
        }
        response = requests.get(url, headers=headers, timeout=15)
        response.raise_for_status()
        doc = Document(response.text)
        return {"title": doc.title(), "text": doc.summary()}
    except Exception:
        logger.exception("Failed to download or parse webpage %s", url)
        return {"title": "", "text": ""}