Reddit Subreddit Scraper

Reddit Subreddit Scraper for extracting post listings, scores, upvote ratios, comment counts and authors from Reddit.com. This repo has a free Reddit subreddit scraping script you can run right now, and a Reddit subreddit data API that returns a listing as 13 fields of structured JSON per post, with real vote data on every row.

This repo reads public subreddits. Every capture, example and screenshot on this page targets a large public community. Post authors are pseudonyms the listing publishes, so they are in the JSON; they are not rendered into any image here, and nothing in this repo is built around a particular person's account.

Last updated: 2026-07-20. Working against Reddit.com as of July 2026, and re-verified whenever Reddit changes their markup.

This repo covers one endpoint in depth. For posts with their comment threads, search and public user feeds, see the Reddit Scraper repo.

Every JSON block on this page was captured from the live API on 2026-07-20. Arrays are trimmed where marked and each block says exactly what was cut; every field shown is verbatim, nulls included. Full uncut samples are committed in reddit_subreddit_scraper_api_data/, so you can diff this page against them. Every code example calls the actual API and is runnable from reddit_subreddit_scraper_api_codes/.

pip install requests
export CHOCODATA_API_KEY="your_key"     # free: 1,000 requests, one-time, no card
python reddit_subreddit_scraper_api_codes/subreddit.py programming

Those three lines return this, live from Reddit.com. One post object, complete, all 13 fields, nothing cut:

{
  "id": "t3_1v1mpxw",
  "title": "Zig proposes introducing an actually memory safe (unlike Rust) compilation mode inspired by Fil-C at ~1-6x performance penalty",
  "score": 162,
  "num_comments": 168,
  "upvote_ratio": 0.7035175879396985,
  "awards": 0,
  "author": "Usual-Amount-264",
  "author_id": "t2_2he243jc5c",
  "subreddit": "programming",
  "permalink": "https://www.reddit.com/r/programming/comments/1v1mpxw/zig_proposes_introducing_an_actually_memory_safe/",
  "external_url": "https://codeberg.org/ziglang/zig/issues/36237",
  "domain": "codeberg.org",
  "created": "2026-07-20T14:12:27.152000+0000"
}

162 points with a 0.70 upvote ratio and 168 comments is not a quiet post, it is an argument: roughly three in ten voters disagreed and then wrote about it 168 times. The score alone cannot tell you that. Multiply it by 24 posts a page:

That is the whole point of this repo. What follows is the free path, the wall it runs into, and the endpoint reference.

---

Contents

• Free Reddit Subreddit Scraper
• Avoid getting blocked when scraping a subreddit
  • Using the Chocodata Reddit Subreddit Scraper API
• Reddit Subreddit Scraper API reference
  • Quickstart · Authentication · Parameters · Errors · Rate limits and concurrency
  • Subreddit: post listings, scores, upvote ratios and comment counts
• Page through a subreddit and export every post to CSV
• Measured latency
• License

---

Free Reddit Subreddit Scraper

Append .json to any Reddit listing URL and Reddit will render it as JSON. No key, no OAuth app, no cost. Everyone reaches for this first, so it goes first here too:

python free_scraper/reddit_subreddit_free_scraper.py programming

Source: free_scraper/reddit_subreddit_free_scraper.py. It requests https://www.reddit.com/r/{sub}/{sort}.json, walks data.children[].data, and emits name, title, score, num_comments, upvote_ratio, author, created_utc, permalink, domain per post.

The one thing it does right, that a lot of scrapers get wrong, is parse before it judges. A listing page can carry the string "blocked" somewhere in its own JavaScript and still hand back 25 usable posts, so a script that scans the body for that word before parsing invents failures that never happened. This one extracts the posts first and only calls a request refused when there are none; if posts come back it prints them and exits 0.

After running the command, your terminal should look something like this:

No posts come back. The next section is what we measured.

Avoid getting blocked when scraping a subreddit

We pointed that script at Reddit from a residential connection while writing this page. All 5 requests to reddit.com/r/{sub}/hot.json, spread over two subreddits, three user agents and both the www and old hosts, came back identical:

What we measured	Value
HTTP status	403
Response size	189,908 bytes
Content-Type	text/html (from a URL ending in .json)
Content-Encoding	none
<title>	none: the body opens straight into <body class=theme-beta>
JSON parse	failed
posts returned	0, on 5 of 5

Every request in that table is recorded in free_scraper_runs.json. Note what is not happening: Reddit is not pretending. There is no 200-that-is-really-a-block and no fake listing. You get a straight 403. The traps are elsewhere.

What bites you	Why	What it costs you
A .json URL that answers with HTML	The 403 comes back as 190 KB of text/html, no <title>, no Content-Encoding. Call .json() on it and you get an exception, not an empty listing.	The traceback points at your JSON parsing, so the first hour goes into the parser when the problem is access.
The User-Agent is not the lever	Three headers (the python-requests default, desktop Chrome, a descriptive script string) produced the same 403 down to the same 189,908-byte body, on both hosts and both subreddits.	Rotating headers buys nothing here, and Reddit's terms ask you not to "misrepresent or mask either the user agent or OAuth identity" (redditinc.com/policies/data-api-terms, section 2.8, retrieved 2026-07-20) in the first place.
The open surface has no vote data	/r/programming/.rss on the same connection, same day, answered 200 with 34,628 bytes. It carries titles, authors, permalinks and timestamps, and no score, no upvote_ratio, no num_comments and no awards anywhere in it.	You ship a "working" scraper that silently cannot answer the only question you had: is this post big or not? A listing without vote data is a list of headlines.
A listing page is not a page of results	r/programming hot served 24 unique posts to us, not the 25 the limit suggests, and controversial served 7. The number moves with the sort and the community.	Sizing a job off limit overestimates what one call returns. Count total_results, and page if you need more (see the field notes).
Half the signal is in a field the page rounds off	Scores in our sample listing ran from 0 to 694 while upvote_ratio ran from 0.35 to 1.00. Three of the 24 posts sat at score 0, and their ratios were 0.43, 0.39 and 0.35: not ignored, actively disagreed with.	Ranking or filtering on score alone throws away the contested posts, which on most subreddits are the interesting ones.
Limits are Reddit's call, not a published constant	"Reddit may set and enforce limits on your use of the Data APIs ... in our sole discretion" (redditinc.com/policies/data-api-terms, section 2.9, retrieved 2026-07-20).	Capacity you cannot plan against, and terms that can move under you.

Worth heading off one myth: "every free Reddit tool on GitHub is dead" is not what we found. JosephLai241/URS (1,008 stars, last pushed 2026-07-11) is actively maintained and does not meet the 403 at all, because it is PRAW on top of Reddit's official API with an OAuth app you register yourself. That is a different route from the one measured here. The 403 above is specifically what the no-key .json path gets, which is the path this repo's free script takes.

---

Using the Chocodata Reddit Subreddit Scraper API

This is the managed path, and what the rest of this repo documents. The Chocodata Reddit Subreddit Scraper API returns a subreddit listing as parsed JSON with score, upvote_ratio, num_comments and awards on every post, across all five Reddit sorts, and runs at your plan's concurrency with no OAuth app to register. It reads the shreddit surface Reddit's own web client reads, which is the one that carries vote data, and it names the surface it used on every response. Free for the first 1,000 requests.

---

Reddit Subreddit Scraper API reference

Below is the Reddit Subreddit Scraper API reference to get you started: authentication, the parameters, error handling, concurrency, and the endpoint itself.

Quickstart

curl "https://api.chocodata.com/api/v1/reddit/subreddit?api_key=YOUR_KEY&subreddit=programming"

import requests

r = requests.get(
    "https://api.chocodata.com/api/v1/reddit/subreddit",
    params={"api_key": "YOUR_KEY", "subreddit": "programming"},
    timeout=90,
)
d = r.json()
top = d["posts"][0]
print(d["total_results"], top["score"], top["upvote_ratio"], top["title"][:40])
# 24 162 0.7035175879396985 Zig proposes introducing an actually mem
# (that score was 162 at capture time and will not be 162 when you run it)

After running the command, your terminal should look something like this:

Get a key at chocodata.com (1,000 requests, one-time, no card).

Authentication

Every request carries api_key as a query parameter, and that is all the authentication there is. No OAuth handshake, no app to register, no token to refresh, and no Reddit account of yours in the loop.

Parameters

Param	Type	Required	Default	Description
api_key	string	yes	-	Your Chocodata API key.
subreddit	string	yes	-	The community name. A leading r/ is accepted and stripped, and casing does not matter here: programming, Programming and r/programming all returned the same 24 posts and the same canonical subreddit: "programming" in the response.
sort	enum	no	hot	hot, new, top, rising or controversial. rising returned a set identical to hot on 3 of 3 back-to-back pairs, same 24 ids in the same order, so treat it as an alias rather than a second listing.
t	enum	no	none	Time window for top and controversial: hour, day, week, month, year, all. Worth sending: sort=top with no t returned 7 to 8 posts on 4 of 4 calls, while top with t=week or t=all returned 24 on 4 of 4.
limit	integer	no	25	Caps how many posts come back, 1 to 75. It truncates and never extends: limit=5 returned 5, and limit=75 returned the same 24 the page had.
after	string	no	none	Pagination cursor. after_cursor in the response is null, so build this yourself: it is base64 of the last post's id. See the field notes, it works.

Each request costs 5 credits (= 1 request). Responses are billed only on success (2xx).

Errors

Status	error code	Meaning	Billed	What to do
400	invalid_params	subreddit is missing or the wrong type. The body names the exact path that failed.	no	Fix the query string.
401	INVALID_API_KEY	Key missing, unrecognised, or revoked.	no	Check api_key. Get one at chocodata.com.
402	INSUFFICIENT_CREDITS	Balance exhausted.	no	Top up or upgrade at chocodata.com.
429	RATE_LIMITED	Over your plan's concurrency.	no	Back off and retry; see Rate limits.
502	-	Reddit did not serve a usable listing for this request. retryable: true.	no	Retry shortly, and check the name resolves to a public community.

Auth and billing errors nest under error.code in uppercase; scrape-layer errors are flat with a lowercase error string. A bad key, verbatim:

{"error":{"code":"INVALID_API_KEY","message":"Api key not recognised."}}

A missing parameter, verbatim. It names the exact path that failed:

{
  "error": "invalid_params",
  "issues": [
    {
      "code": "invalid_type",
      "expected": "string",
      "received": "undefined",
      "path": ["subreddit"],
      "message": "Required"
    }
  ]
}

Both captured bodies, with their requests and statuses, are in errors.json. Every status in the table maps to a one-line message in this repo's scripts, so a typo'd key ends in a sentence rather than a traceback:

Rate limits and concurrency

Requests are not capped per minute. What is capped is concurrency, the number you can have in flight at any one moment.

Plan	Concurrent requests
Free	10
Vibe	30
Pro	50
Custom	100 to 500+

Go over it and the next request is refused with 429 rather than queued. The call is a synchronous GET, so there is nothing to poll: no webhook, no callback, no job id. The timeout=90 in the examples leaves room for the occasional call that has to be re-attempted upstream and runs long (see Measured latency).

Sizing: at Pro (50 concurrent) and the measured 2.4s median, one worker pool sustains roughly 50 / 2.4 = 21 requests per second. Fan out with a thread pool:

from concurrent.futures import ThreadPoolExecutor
import requests

def one(sub):
    r = requests.get("https://api.chocodata.com/api/v1/reddit/subreddit",
                     params={"api_key": KEY, "subreddit": sub}, timeout=90)
    return sub, (r.json()["total_results"] if r.ok else None)

with ThreadPoolExecutor(max_workers=10) as pool:   # <= your plan's concurrency
    for sub, n in pool.map(one, subreddits):
        print(n, sub)

---

Subreddit: post listings, scores, upvote ratios and comment counts

One page of a subreddit's listing, in any of Reddit's five sorts, with real vote data on every post.

Param	Type	Required	Default	Description
subreddit	string	yes	-	Community name, with or without r/. Casing does not matter.
sort	enum	no	hot	hot, new, top, rising, controversial.
t	enum	no	none	hour, day, week, month, year, all, for top and controversial.
limit	integer	no	25	1 to 75. Truncates only.
after	string	no	none	Cursor: base64 of the previous page's last post id.

curl "https://api.chocodata.com/api/v1/reddit/subreddit?api_key=YOUR_KEY&subreddit=programming"

Real response, with posts cut to 1 of 24. The envelope is complete and that post object is complete, all 13 fields verbatim (full sample):

{
  "subreddit": "programming",
  "sort": "hot",
  "t": null,
  "after_cursor": null,
  "total_results": 24,
  "posts": [
    {
      "id": "t3_1v1mpxw",
      "title": "Zig proposes introducing an actually memory safe (unlike Rust) compilation mode inspired by Fil-C at ~1-6x performance penalty",
      "score": 162,
      "num_comments": 168,
      "upvote_ratio": 0.7035175879396985,
      "awards": 0,
      "author": "Usual-Amount-264",
      "author_id": "t2_2he243jc5c",
      "subreddit": "programming",
      "permalink": "https://www.reddit.com/r/programming/comments/1v1mpxw/zig_proposes_introducing_an_actually_memory_safe/",
      "external_url": "https://codeberg.org/ziglang/zig/issues/36237",
      "domain": "codeberg.org",
      "created": "2026-07-20T14:12:27.152000+0000"
    }
  ],
  "_source": "shreddit"
}

upvote_ratio is the field worth automating for. Reddit shows you a score and, on a good day, a rounded percentage; here it is a float, per post, on every row.

Field notes:

1. after_cursor comes back null, and pagination still works. It was null on 41 of 41 listing responses in this session. The cursor Reddit's own listing uses is just base64 of the last post's fullname, and id is already that fullname: t3_1v1mpxw encodes to dDNfMXYxbXB4dw==. Send it as after and the next page follows on. We chained it four pages deep on two subreddits and got 97 unique posts from r/programming and 96 from r/askscience, with zero repeats on either:

   

   Every page of that run is in pagination.json, and paginate_to_csv.py does it for you.

2. A cursor Reddit does not accept silently re-serves page 1. We sent after=not-a-real-cursor and got 200 with the same 24 posts as an uncursored call, not an error. A paging loop that trusts the response will spin on page one forever, so stop when a page brings no new ids. That is the check paginate_to_csv.py uses.

3. _source names the surface that answered, and it decides whether the vote fields are real. It read shreddit on 40 of the 41 listings we captured. On the 41st it read rss-fallback, and on that surface score, num_comments, upvote_ratio, awards and author_id come back null rather than guessed, with a _rss_limitations object spelling out which fields those are. Same post, same day, both surfaces committed side by side in subreddit.json and subreddit_rss_fallback.json:

   {
     "subreddit": "programming",
     "sort": "top",
     "t": null,
     "after_cursor": null,
     "title": "top scoring links : programming",
     "description": "Computer Programming",
     "icon": "https://www.redditstatic.com/icon.png/",
     "total_results": 8,
     "posts": [
       {
         "id": "t3_1v1mpxw",
         "title": "Zig proposes introducing an actually memory safe (unlike Rust) compilation mode inspired by Fil-C at ~1-6x performance penalty",
         "score": null,
         "num_comments": null,
         "upvote_ratio": null,
         "awards": null,
         "author": "Usual-Amount-264",
         "author_id": null,
         "subreddit": "programming",
         "permalink": "https://www.reddit.com/r/programming/comments/1v1mpxw/zig_proposes_introducing_an_actually_memory_safe/",
         "external_url": "https://codeberg.org/ziglang/zig/issues/36237",
         "domain": "codeberg.org",
         "created": "2026-07-20T14:12:27+00:00"
       }
     ],
     "_source": "rss-fallback",
     "_rss_limitations": {
       "surface": "rss-atom",
       "unavailable_fields": ["score", "ups", "downs", "upvote_ratio", "num_comments", "awards", "total_awards_received", "author_karma", "author_cake_day", "flair"],
       "note": "This response used the .rss/Atom fallback (the shreddit HTML surface that carries vote data was unreachable on this request). RSS does not expose any vote/award/comment counts or author karma - those fields are null (not fabricated). Identity, author, timestamps, permalinks, post/comment bodies ARE provided."
     }
   }

   (The envelope is complete; only the posts array is cut, to 1 of 8, and that post object is verbatim. This surface adds title, description and icon that the shreddit surface does not.) The practical rule: branch on _source, or test score is not None, before you write a row into a dataset. A null score on this endpoint means "not on this surface", never "zero".

4. total_results is what came back, not what exists. 24 on hot for r/programming, 25 on a paged call, 7 on controversial, 8 on top with no t. Reddit serves a listing page, and the page size moves with the sort and the community. Count the array.

5. created is an ISO timestamp with milliseconds on the shreddit surface (2026-07-20T14:12:27.152000+0000) and to the second on the rss fallback (2026-07-20T14:12:27+00:00). Parse it as ISO 8601 rather than slicing it.

6. Classify a post on domain, not on external_url. domain is the outbound host, or self.{subreddit} for a text post. external_url is the full link, and it is the less reliable of the two: our r/programming capture has 0 text posts and 24 outbound links, but external_url is null on 2 of them while domain still names the host (youtu.be and mohamed.computer). Both are in subreddit.json if you want to look. Read domain for the "what kind of post is this" branch and treat external_url as best effort.

7. awards was 0 on all 24 posts in the sample listing, and author is the pseudonym as Reddit publishes it, "[deleted]" when the account is gone.

8. Scores drift, so two captures of one listing will not match. The r/programming post used throughout this page reads 162 in subreddit.json and 196 in the terminal screenshot above: same post, same afternoon, two observations minutes apart, both real. Do not treat a mismatch between this page and your own run as an error: that drift is the signal you are collecting, and it is why the endpoint is worth putting on a schedule.

Runnable: reddit_subreddit_scraper_api_codes/subreddit.py

---

Page through a subreddit and export every post to CSV

Getting more than one page out of a subreddit is the job people actually have, so it is in the repo end to end rather than as a snippet. paginate_to_csv.py builds the cursor from the previous page's last post, walks as many pages as you ask for, drops duplicates, and writes a CSV:

python reddit_subreddit_scraper_api_codes/paginate_to_csv.py programming 4 out.csv

That is a real run: 97 unique posts, every one carrying vote data, four API requests. The new column in that output is the guard rail. It counts ids not seen on an earlier page, and the loop stops when it hits zero, which is what a rejected cursor looks like from the outside.

Each row carries page and source alongside the 13 post fields, so a row with a null score is traceable to the surface that produced it rather than being an unexplained gap in your dataset.

One API request per page, so the arithmetic is simple: four pages of one subreddit daily is 4 requests a day, and the 1,000 free requests cover about 250 days of that, one-time. Point it at a set of competitors' communities and the CSV is the report.

---

Measured latency

Real end-to-end wall-clock, measured from a laptop against the live API on 2026-07-20 across 5 subreddits and all five sorts. This includes the upstream fetch, the anti-bot handling and the parse:

Endpoint	Median	Range	n
/reddit/subreddit	2.4s	1.5 to 16.9s	41

Read the range, not just the median. 34 of those 41 calls came back inside 4 seconds, and the 16.9s outlier is the number worth sizing for: requests that have to be re-attempted upstream land at that end of the spread and still come back with real data. Absorbing those attempts is a good part of what you are paying for. Every measurement is committed in latency.json. Small sample from one laptop on one afternoon, so size your timeouts off the tail rather than the median, and reproduce it with the scripts in this repo.

---

License

MIT. See LICENSE.