Building a Reddit Dataset From the API: What You Get That a Static Dump Does Not

Every page that ranks for "reddit dataset" hands you a file.

TL;DR: We measured both sides on 2026-09-02. One live call per community across ten communities returned 1,000 posts, 100 per call, 10 of 10 answering HTTP 200, each record carrying 22 top-level fields. The median newest record was 12.03 minutes old and the freshest was 21 seconds old. The freshest public monthly dump on the same day covered data through 2026-07-31, which is 32.73 days back, because those archives publish a median of 9 days after the month they cover ends across 15 consecutive releases. We then re-read the same 1,000 records 11.4 minutes later and 496 of them had already changed score or comment count. The archive still wins one column: 25 of those 1,000 year-old top posts no longer name an author through the live API, and a dump written a year ago still does.

We went looking for the page that teaches building one instead, and it is not there. The top ten results for the head term on 2026-09-02 were a subreddit, three forum threads asking where to get data, an academic corpus loader, a Stanford embedding set, a GitHub repository last touched in 2016, a Kaggle listing, a TensorFlow catalog entry and a vendor selling a packaged Reddit dataset on a data marketplace. Ten results, every one of them a thing you download or a person asking what to download. Nobody on that page shows you the other option.

Not affiliated with Reddit Inc. redditapis.com is an independent third-party REST proxy for Reddit's data API. This guide is vendor-neutral: it reports our own measurements with their dates and sample sizes, names the places where a static archive is genuinely the better tool, and says plainly which figures are measured and which are cited.

This post is that missing page, and it does not argue by assertion. Every number below came out of a pass we ran on 2026-09-02: ten communities, 1,000 posts, a re-read of the same 1,000 ids eleven minutes later, 1,000 more posts from the past year, five sequential pages against one community, and a read of the public archive release feed itself. The raw output and the falsification condition for each pass are recorded, and each figure says whether it was measured, derived or cited.

---

---

What actually is a Reddit dataset in 2026?

It is one of two very different things, and the phrase hides the difference. A Reddit dataset is either a file somebody exported at a moment in the past, or a store you keep filling from a live source.

• The export. A corpus written once, covering a stated date range, distributed as a download. Its contents never change, which is both the feature and the limit.
• The store. A table you fill from a live source on a cadence, whose rows you can re-read and correct. Its coverage grows and its mutable fields stay current, at the cost of running something.

Both are legitimate. They answer different questions, they cost different things, and almost every published guide describes only the first one. The word "dataset" pulls toward the file, because that is what the word has meant since researchers first started sharing corpora, and the pull is strong enough that people building live pipelines still describe what they are doing as "getting the dataset".

The distinction shows up the moment you ask a question with a tense in it. What are people saying about this product this week is a live question, and no file answers it. What did discussion of this topic look like across 2019 is a historical question, and a live API answers it slowly and expensively while a file answers it in one scan. Most of the confusion in this subject comes from people asking a live question and reaching for a historical tool because that is what the search results offered.

The autocomplete tail for the head term makes the pull visible. On 2026-09-02, Google's own suggestions for "reddit dataset" were: reddit datasets, reddit dataset kaggle, reddit dataset download, reddit dataset for sentiment analysis, reddit dataset pushshift, reddit dataset huggingface, reddit dataset github, reddit dataset.csv, reddit dataset pyg, reddit dataset graph. Ten suggestions, of which seven name a static download source or a file extension. That is not Google being wrong. It is Google reflecting what people actually type, which is a request for a file, because for most of the last decade a file was the only thing on offer.

That changed and the guidance did not. Our own state of the public JSON endpoints covers what happened to the free path, and the hub this post sits under covers the full method landscape. The short version is that the bulk archive lane narrowed sharply, the credentialed API lane stayed open, and the published advice mostly still points at the lane that narrowed.

How fresh is the data one live API call returns?

Twelve minutes old at the median, and twenty one seconds old at best. On 2026-09-02 we made one listing call per community against ten communities, at limit=100 sorted by new, and measured the gap between the read time and the newest created_utc in the returned page. The median across those ten calls was 12.03 minutes. The fastest community, r/AskReddit, returned a post that had existed for 0.35 minutes. The slowest, r/webscraping, returned one 504.24 minutes old, and that is a property of how quiet the community is rather than anything about the API.

AGE OF THE NEWEST RECORD ONE CALL RETURNED

Freshness by community, one pass, ten calls

Point	Value (min)
r/AskReddit	0.35 min
r/MachineLearning	0.8 min
r/technology	3.1 min
r/datasets	6.77 min
r/SaaS	9.03 min
r/LocalLLaMA	15.03 min
r/redditdev	28.14 min
r/dataisbeautiful	223.01 min
r/Python	406.8 min
r/webscraping	504.24 min

Here is the whole pass rather than a summary of it, including the window each page covered and what each request cost in wall clock:

What one live call returned, per community, read in one pass

Community	Posts returned	Window covered	Age of newest record	Request latency	Source
r/AskReddit	100	0.47 hours	0.35 min	5.011 s	measured
r/MachineLearning	100	407.56 hours	0.80 min	16.665 s	measured
r/technology	100	29.45 hours	3.10 min	2.854 s	measured
r/datasets	100	537.38 hours	6.77 min	4.364 s	measured
r/SaaS	100	18.46 hours	9.03 min	3.128 s	measured
r/LocalLLaMA	100	38.49 hours	15.03 min	4.551 s	measured
r/redditdev	100	2,692.91 hours	28.14 min	7.596 s	measured
r/dataisbeautiful	100	145.86 hours	223.01 min	29.406 s	measured
r/Python	100	851.40 hours	406.80 min	12.805 s	measured
r/webscraping	100	1,150.16 hours	504.24 min	4.407 s	measured

n = 1,000 · as of 2026-09-02

Two things in that table deserve more attention than the freshness column. The first is that all ten calls returned exactly 100 items, which is the ceiling on this endpoint and is worth internalising early because it shapes everything downstream. The second is the window column, which ranges from 0.47 hours to 2,692.91 hours. The same 100 records covered under half an hour in one community and 112 days in another, a ratio of 5,729.6 to one. If you build a collection loop that pages a fixed number of posts per community per run, you have silently built ten different sampling regimes and you will not notice until somebody compares two communities and gets nonsense.

Latency is the number we are least confident in and it is reported plainly for that reason. Median 4.781 seconds, mean 9.08 seconds, fastest 2.854 seconds, slowest 29.406 seconds. That is one client on one machine on one network path making sequential requests on one afternoon, which is exactly the wrong shape of sample for a performance claim. We publish it because a real number with its limits stated is more useful than silence, not because it is a benchmark. If you want that properly, the latency, uptime and cost benchmark is the post that does it with a sampling design built for the question.

How fresh is the newest public Reddit data dump?

Thirty two days old, and that is the freshest it gets. The largest actively maintained public monthly archive publishes as a release feed, so the cadence is not something you have to take anyone's word for. We read it directly through the GitHub API on 2026-09-02 and got fifteen consecutive monthly releases, the newest being the July 2026 data, published on 2026-08-09.

The person who runs the most visible self-hosted archive project describes the cadence plainly, and it is worth reading in his words rather than ours:

"the data from 2025-12 has been released already, it is usually released every month, it just needs to be split and reprocessed for 2025 by watchful1."

Which means that on the day we ran the live pass, nothing in the freshest public dump was dated later than 2026-07-31. Against our read moment that is 32.73 days, or 47,126.6 minutes. Set beside the live median of 12.03 minutes, the ratio is 3,917 to one, derived as 47,126.6 divided by 12.03. Against the fastest community in our basket it is 134,648 to one.

None of that is a criticism of the people running those archives. The cadence is regular, the releases arrive, and the median lag from the end of a covered month to its publication is 9 days across all fifteen releases we read, with a minimum of 6 and a maximum of 16. That is a well-run monthly process. The arithmetic is simply that no monthly process can hand you a record from this morning, and a lot of published advice talks about these archives as though it could.

When each monthly public dump actually appeared

Release	Data covered through	Published	Days after the month ended	Source
2026_07	2026-07-31	2026-08-09	8	measured
2026_06	2026-06-30	2026-07-12	11	measured
2026_05	2026-05-31	2026-06-11	10	measured
2026_04	2026-04-30	2026-05-10	9	derived
2026_03	2026-03-31	2026-04-10	9	measured
2026_02	2026-02-28	2026-03-08	7	measured
2026_01	2026-01-31	2026-02-07	6	measured
2025_12	2025-12-31	2026-01-10	9	derived
2025_11	2025-11-30	2025-12-17	16	measured
2025_10	2025-10-31	2025-11-11	10	measured
2025_09	2025-09-30	2025-10-12	11	measured
2025_08	2025-08-31	2025-09-12	11	measured
2025_07	2025-07-31	2025-08-08	7	measured
2025_06	2025-06-30	2025-07-07	6	measured
2025_05	2025-05-31	2025-06-08	7	measured

n = 15 · as of 2026-09-02

The maintainer of the most visible self-hosted archive project said the cadence out loud in January 2026, in a Hacker News thread that reached 286 points, when somebody asked the obvious question:

https://news.ycombinator.com/item?id=46602324

The question was whether the underlying data updates on a cadence or is a fixed historical snapshot, and whether self-hosters would need to re-download for fresh content. The answer, from the project author, was that the December 2025 data had been released and that it is usually released every month, with a further step where a second maintainer splits and reprocesses it. That is the honest, unvarnished description of the pipeline from the person running it, and it is the number we use for the dump side of every comparison in this post.

The same question keeps getting asked on Reddit itself, which is a decent sign that the answer is not obvious to buyers. On the thread announcing a self-hosted archive tool, one commenter asked whether the data only ran from 2005-06 to 2022-12 and whether new data would be updated at all:

r/

r/pushshift·u/Yekab0f

redarc - A selfhosted Pushshift alternative

00

Open on Reddit

That is the first question a person asks about an archive, and it should be.

There is a second lag inside the first one, and it is the reason the effective staleness is worse than the release date suggests. The monthly release is the raw month. Turning it into the per-community splits that most people actually download is a separate step run by a different maintainer, and on the Hacker News thread the project author noted that the 2025 data still needed to be split and reprocessed at the time he wrote. So the chain is: the month ends, the release appears a median of 9 days later, the split appears some further interval after that, and only then does the practical artefact exist. Every one of those steps is somebody doing unpaid work well, and none of them makes the file newer.

The practical consequence is that you should never date a hybrid corpus by the release you downloaded. Date it by the newest record actually inside the file, check that number rather than assuming it, and store it beside the data. A corpus whose stated coverage comes from a filename rather than from a scan of its own contents is a corpus whose freshness nobody has ever verified. We keep a current view of what replaced the older bulk archives if you want the full landscape of who publishes what, and the Arctic Shift path for agent workloads covers the specific archive whose release feed we measured here.

How much does a stored Reddit row change after you save it?

Half of them changed inside twelve minutes. This is the measurement that turned the argument from a preference into a number, and it is simple enough to reproduce in an afternoon. We took the exact 1,000 post ids from the freshness pass, waited, and re-read all 1,000 through the bulk id endpoint. The gap was 11.4 minutes. 496 of 1,000 records had changed their upvotes or their comments value.

Broken out, 443 records changed score and 116 changed comment count, with 63 doing both. The largest single score move was plus 458 points on a post in r/technology, and the largest comment move was plus 40 on a post in r/AskReddit. The rate is not uniform: 69 of 100 moved in r/technology and 13 of 100 in r/datasets.

The same 1,000 records, re-read 11.4 minutes later

Community	Rows re-matched	Score changed	Comment count changed	Unchanged	Largest score move	Source
r/technology	100	69	23	31	+458	measured
r/dataisbeautiful	100	67	9	31	+44	measured
r/AskReddit	100	21	61	39	+6	measured
r/LocalLLaMA	100	56	14	39	+8	measured
r/webscraping	100	59	0	41	+5	measured
r/MachineLearning	100	56	0	44	+10	measured
r/Python	100	55	0	45	+4	measured
r/redditdev	100	29	0	71	+3	measured
r/SaaS	100	18	9	76	+6	measured
r/datasets	100	13	0	87	+3	measured

n = 1,000 · as of 2026-09-02

The obvious objection to a single eleven-minute window is that we picked it, so we ran the same comparison again at a longer gap. Re-reading the identical 1,000 ids 38.8 minutes after first capture returned 565 changed records against 435 unchanged, up from 496. Score movement rose from 443 records to 496 and comment movement from 116 to 165, and the largest single score move grew from plus 458 to plus 1,523 on the same community. The curve goes one way, which is what it should do if this is decay rather than noise. A second window returning fewer changed records than the first would have meant our comparison was broken rather than that the data had settled.

The confound here is real and we are going to say it before anyone else does. The first pass sorted by new, so this is a sample of young posts, and young posts are the fastest moving records on the platform. A row about a three year old thread does not behave like this. The honest claim is narrower than the headline: a stored row about a NEW post decays this fast, which matters enormously if your pipeline collects new posts, which is what almost every pipeline does.

The implication for dataset design is direct. Any field that can move needs a refresh path or a timestamp saying when it was true, and preferably both. A score column with no as-of time is not data, it is a rumour with a decimal point. This is why the collection design later in this post has two loops rather than one, and why bulk fetching by id is the endpoint that does the unglamorous half of the work.

What does a Reddit post record contain, field by field?

Twenty two fields at the top level, and they were identical across all 1,000 records we read. That consistency is worth stating on its own, because a schema that varies by community or by post type is a genuine engineering problem and this one did not vary at all in our sample.

ONE POST RECORD, READ LIVE

The 22 fields a listing call hands you per post

Community listing, 2026-09-02

TOP-LEVEL FIELDS

22

LEAF FIELDS WITH NESTING

26

RECORDS PER CALL

100

BYTES PER RECORD

1,621

upvotes, comments and upvote_ratio give you live engagement

created_utc plus created give you both epoch and ISO time

locked, stickied, spoiler and over_18 carry current moderation state

is_crosspost and crosspost_origin carry lineage, not just content

author_info nests fullname, premium, is_blocked and flair

No field tells you what this post scored yesterdayHistory is something you store, never something a read returns

No field tells you the post was edited or whenThe post shape does not carry an edited marker, the comment shape does

Field list read off the live response on 2026-09-02 and verified identical across all 1,000 post records in the pass. Bytes per record is derived as 1,620,777 response bytes divided by 1,000 records, with compression disabled.

The fields are author, author_info, comments, created, created_utc, crosspost_origin, id, is_crosspost, is_self, link_url, locked, name, over_18, permalink, spoiler, stickied, subreddit, text, title, upvote_ratio, upvotes and url. The author_info field is itself an object carrying four more, fullname, premium, is_blocked and flair, so a post is 26 leaf fields if you count that way. Payload worked out at 1,620.8 bytes per record, derived from 1,620,777 response bytes across 1,000 records with compression disabled.

Grouping those 22 by what they do is an editorial call rather than a measurement, so treat the split as our classification rather than the platform's: five identity and origin fields, five content and link fields, four moderation state flags, three engagement fields, three type and relationship fields, and two timing fields. What that grouping makes obvious is how much of a post record is state rather than content. Seven of the 22 fields describe the current condition of the post rather than what it says, and every one of those seven is a field that can change after you store it.

There is a second thing that grouping reveals, and it is the whole reason the comparison in this post is not just about dates. If a field is not in a file, you cannot get it out of the file. A person on the archive threads found that the hard way and asked about it directly, having noticed that the Posts rows in a monthly dump did not carry a removed field even though he collected it himself in his own process. That is not a complaint about the archive. It is the structural point: an export carries the fields whoever built it chose to carry, and a re-read of a live source carries whatever the source returns today.

Why does a comment record carry three times as many fields as a post?

Because comments live inside a moderation and threading system that posts mostly sit above. A comment record read from the comment tree endpoint on 2026-09-02 carried 70 top-level fields, against a post's 22, a user profile's 19 and a community about record's 17.

What each record type actually carries, counted off live responses

Record type	Fields returned	Endpoint read	What the extra fields buy you	Source
Post	22	Community listing	Engagement, moderation state, crosspost lineage and authorship in one object	measured
Comment	70	Post comment tree	Thread structure, award state, edit and removal markers, moderation fields	measured
User	19	User profile	Account age and split karma totals, the only real cohort variable available	measured
Community	17	Community about record	Subscriber count, creation date, type and description for sizing	measured

n = 4 · as of 2026-09-02

The extra 48 fields on a comment are not padding. They fall into five families, none of which a post record needs:

• Threading. parent_id and the reply structure, which is what lets you rebuild a conversation rather than a bag of text.
• Edit and removal state. edited, removal_reason, collapsed_reason and the approval fields, all of which move after the record is written.
• Moderator speech. distinguished and the mod-note fields, which mark which comments carry authority.
• Award and flair machinery. Counts and template ids that describe how a community decorated the comment.
• Author context. Flair type, richtext and CSS class, which are community-scoped rather than account-scoped.

The point of that list is not the field names. They carry the parent relationship that lets you reconstruct a thread, the edit marker, the removal and approval state, the distinguished flag that marks moderator speech, award counts, collapse reasons and the author flair machinery. If your dataset is about discussion rather than about submissions, that is the record type you want, and it changes the storage maths: a comment corpus is both wider and far more numerous than a post corpus over the same window.

This is also the clearest single argument against treating any Reddit corpus as interchangeable with any other. Two datasets both called "Reddit data" can differ by a factor of three in how much they know about each row, and a schema mismatch on ingestion is the most common way that difference becomes somebody's afternoon. The comment endpoint guide covers the tree structure, and comment search covers finding comments across communities rather than inside one thread.

The practical consequence for a build is that you should decide which record type is your spine before you write a schema. A posts-spine corpus with comments attached is a different shape from a comments-spine corpus with post metadata denormalised onto each row, and converting between them later means a full rewrite of every query you have written in the meantime.

How many records does one call return, and how far back does it reach?

One hundred, always, and how far back those hundred reach depends entirely on the community. This is the single most misunderstood property of listing endpoints and it breaks more analyses than any other, because the count is fixed and the window is not.

Measured on 2026-09-02, the same 100-post page at limit=100 covered 0.47 hours in r/AskReddit and 2,692.91 hours in r/redditdev. That is 28 minutes against 112 days from an identical request. The median across the ten was 276.71 hours. If you build a job that pulls one page per community per hour, you are oversampling the quiet communities by a factor in the thousands and undersampling the fast ones so badly that you miss most of what happens.

The fix is to sample by time window rather than by record count. Decide the period you care about, page until the oldest record in your last page precedes it, and record how many calls that took. That figure, calls per community per window, is the number you actually need for capacity planning, and it is different for every community in your basket.

That walkthrough is a good illustration of the older shape of this work, pulling Reddit data through an R package, and it is also a small demonstration of the staleness problem in a different medium: it carries a 2023 update in its own title because the thing it teaches keeps moving underneath it. Tutorials go stale the same way corpora do.

How do you page through more than one hundred posts?

With the cursor the response hands you, the after value, passed back verbatim on the next request. We tested it five pages deep against r/technology on 2026-09-02 and it walked cleanly: 500 unique ids, no duplicates, no gaps, 24.14 seconds of wall clock, and a non-null cursor still present on page five.

Five sequential pages against one community, timed

Page	Items returned	Cumulative unique ids	Latency	Response bytes	Source
1	100	100	4.129 s	83,868	measured
2	100	200	4.267 s	84,474	measured
3	100	300	4.774 s	84,413	measured
4	100	400	5.480 s	84,199	measured
5	100	500	5.420 s	83,786	measured

n = 500 · as of 2026-09-02

Twenty point seven one records per second is the honest throughput figure for sequential paging from one client. It is not fast, and it should not be, because a listing endpoint is a read against a live system rather than a bulk export. What it is, is predictable: a hundred records per call, roughly five seconds per call, and a cursor that keeps working. For a corpus of a hundred thousand posts that is a thousand calls and something under two hours of wall clock, which most people find surprisingly manageable once they have stopped imagining they need the whole platform.

The two mistakes we see most often with cursors are worth naming. The first is treating the cursor as an offset and trying to compute it, which does not work because it is opaque and the endpoint documentation says so. The second is discarding it between runs, which turns every scheduled job into a first page and guarantees you re-collect the same recent records forever while the older ones you wanted stay out of reach. Store the cursor with the job state. The pagination guide covers both in detail, and the rate limit guide covers how fast you can reasonably run the loop.

One more property worth knowing before you design around it: latency crept up across our five pages, from 4.129 seconds on page one to 5.420 seconds on page five. Five samples is nowhere near enough to call that a trend and we are not calling it one. It is in the table because it is what we measured, and because if you see the same shape at page fifty it is worth investigating rather than assuming.

What happens to a Reddit record after a year?

Some of it goes away, and this is the column where a static archive genuinely beats a live read. We pulled the top 100 posts of the past year from each of the same ten communities on 2026-09-02, 1,000 records in total, all HTTP 200. 25 of those 1,000 now return a deleted or empty author. 7 are locked. None returned a removal marker in the body text, which is a null result worth stating rather than omitting.

What a year does to a record, read through the live API today

Community	Top posts read	Author no longer present	Locked now	Oldest post in sample	Source
r/technology	100	9	2	364.2 days	measured
r/Python	100	4	0	364.2 days	measured
r/AskReddit	100	4	4	358.9 days	measured
r/dataisbeautiful	100	2	0	363.2 days	measured
r/redditdev	100	2	1	363.4 days	measured
r/LocalLLaMA	100	2	0	361.1 days	measured
r/MachineLearning	100	1	0	355.0 days	measured
r/datasets	100	1	0	361.0 days	measured
r/SaaS	100	0	0	360.6 days	measured
r/webscraping	100	0	0	363.9 days	measured

n = 1,000 · as of 2026-09-02

Read the direction of that finding carefully, because it is easy to get backwards. This does not show the API losing data. It shows the API reflecting deletions the platform honoured, which is the correct behaviour and in most jurisdictions the required one. An archive written a year ago holds the original author name because it was written before the deletion, and that is exactly why an archive is the right tool for some questions and the wrong tool for others.

Somebody on Hacker News made this case better than we can:

"Reddit is becoming maddening to use because half the old links I click have comments overwritten with garbage out of protest for something. Ironically the original content is available in these archives (which are used for AI training) but now missing for actual users like me just trying to figure out how someone fixed their printer driver 2 years ago."

That comment is worth reading in full alongside the thread it sits in:

https://news.ycombinator.com/item?id=46606273

That is a real cost of a live-only pipeline and it deserves to sit in the post rather than in a footnote. The corollary is also worth stating: if your dataset needs to preserve content that authors later remove, you are making a decision with consequences beyond engineering, and the right time to think about it is before you build rather than after somebody asks.

The 9-of-100 figure in r/technology against 0-of-100 in r/SaaS and r/webscraping is a real spread and we do not have a confident explanation for it. Larger, more contentious communities plausibly generate more account deletions, but ten communities is far too small a sample to assert that, so it is reported as an observation rather than a finding.

Which endpoint should you point at which part of your dataset?

Five surfaces, and each one answers a different question. Choosing wrongly is the most common reason a build takes three times longer than it should, because the wrong endpoint makes you page for data that another endpoint would have handed you in one call.

In short:

• Community listing for the spine of the corpus, up to 100 posts per call with a cursor.
• Search for topic corpora that span communities rather than sit inside one.
• Bulk id for the refresh loop, up to 100 record ids per call.
• Comment tree for discussion work, 70 fields per comment record.
• Community about for sizing metadata, 17 fields in one object.

The community listing endpoint is the spine. It returns up to 100 posts per call with a cursor, and it is what you page on a schedule to keep a community's history filling in. The search endpoint is for topic corpora that span communities, and it is the right tool when your dataset is defined by a subject rather than by a place. The bulk id endpoint takes up to 100 comma-separated record ids in a single call and hands back their current state, which makes it the refresh loop. The comment tree endpoint gives you the 70-field comment records for a thread. The community about record gives you 17 fields of metadata for sizing.

THE TWO LOOPS A LIVING DATASET NEEDS

Where each endpoint feeds the corpus

new rowsbackfillidsupdatesthreadsmetadata

01ListingUp to 100 new posts per call

02SearchTopic backfill across communities

03Bulk idUp to 100 records refreshed per call

04Comments70 fields per item

05Community17 field profile for sizing

06Your corpusStored, stamped, queryable

Author history is the fifth surface and it is the one people forget. Reading an account's own submissions or comments is how you build cohort and overlap analysis, and it is the only route to questions like which other communities the people posting here also post in. It is also the most expensive per unit of insight, because it is one call per account, so budget it deliberately rather than discovering it mid-build.

There is a sixth thing that is not an endpoint at all and belongs in this list anyway: a push channel. If your dataset needs to react to new content rather than merely accumulate it, polling is the wrong shape and the webhook and polling comparison covers the trade properly. Monitoring with webhooks is the build guide for that half.

How do you actually build the collection loop?

Two loops, running on different cadences, writing to the same table. Almost every home-built Reddit pipeline we have seen ships the first one and skips the second, which is why their engagement columns quietly drift away from reality.

The collection loop pages new posts. Pick the time window first, page until the oldest record in the page predates your watermark, upsert every record keyed on its id, and store the cursor with the job state so the next run resumes rather than restarts. The refresh loop takes ids you already hold, batches them a hundred at a time into the bulk id endpoint, and writes the current score and comment count back. How often you run the refresh loop is a real design decision and our mutability measurement gives you the input for it: at 11.4 minutes, half of a fresh cohort had already moved.

A reasonable default that falls out of that number is to refresh aggressively for the first day of a record's life, then back off sharply. A record's score moves most in its first hours and then settles, so a fixed refresh interval either wastes most of its calls on old records or misses most of the movement on new ones. Age-tiered refresh is more code and considerably fewer calls.

The person who built the original large Reddit archive described his own first collection method in a public thread in November 2025, and it is a useful reality check on how exotic any of this is:

r/

r/redditdev·u/One_Performance_9259

Is there a way to (legally) scrape more than 2000 posts on Reddit?

00

Open on Reddit

"You can ask for posts by their ids going backwards. You can't specify a specific subreddit with this method but you can go back as far as you want. When I created Pushshift, my first large Reddit dump was done by using the sequential IDs directly."

He walked the sequential record ids backwards. The canonical bulk archive that half this SERP is descended from was itself assembled by reading a live API in a loop, which is the strongest possible argument that the archive lane and the API lane were never really two different lanes. One of them just got exported to a file.

For the code shape, the Python tutorial walks the request layer, and the analytics dashboard build walks the storage and rollup layer with a schema you can copy. If you would rather not write either, the build versus buy comparison is the honest version of that decision including the cases where building wins.

How should you store a dataset whose rows keep moving?

Append the history, project the current state, and never let a mutable field sit in a table without a timestamp beside it. Three objects do it: a records table for what never changes, an observations table for what does, and a current view that joins the latest observation onto each record. That is the whole design, and it is boring on purpose.

The shape, in the smallest form that works:

records      (id PK, subreddit, author_first_seen, created_utc,
              title, body, first_collected_at)
observations (record_id FK, observed_at, upvotes, comments,
              upvote_ratio, locked)
current      VIEW: records JOIN latest observation per record_id

The concrete version: one table holds immutable facts about a record, the id, the author as first seen, the created timestamp, the title and body as first collected, the community. A second table holds observations, one row per read, carrying the record id, the observed score, the observed comment count, the observed lock state and the time of the read. A view joins the latest observation onto each record to give you the current-state table everyone actually queries.

That shape costs you one extra table and buys three things that are genuinely hard to retrofit. You can answer questions about trajectory rather than only about state, which is where most of the interesting analysis lives. You can date every figure you publish, which is what makes a dataset citable rather than merely large. And when a field turns out to have been wrong, you can see when it changed instead of guessing.

The alternative, which is what most pipelines do, is a single wide table that gets overwritten in place. It is simpler for a week. Then somebody asks how fast this post grew, and the honest answer is that the information was collected and then deliberately discarded, four hundred and forty three times in eleven minutes in our sample alone.

Storage volume is not the constraint people expect. At 1,620.8 bytes per post record, a million posts is about 1.6 GB of raw JSON, and the observation table is far smaller per row because it holds five columns rather than 22. A comment corpus is a different matter, given 70 fields per record and a much higher record count per community, and that is the case where you should think about column stores before you think about anything else.

What does a static dump still do better?

Three things, and pretending otherwise would make this post worse. A static archive beats a live read on history, on completeness of the record after somebody deletes it, and on cost per unit of volume once you are working at the scale of billions of rows rather than millions. Each of those is a real advantage and each one is measurable, so they are worth stating precisely rather than conceding vaguely.

• History. One scan over local storage beats paging a live endpoint backwards for years, and for deep history the cursor will not take you there in practice anyway.
• The deleted record. 25 of 1,000 year-old top posts no longer name an author through the live API. A file written before those deletions still does.
• Cost per unit of volume. At 20.71 records per second per client, a billion-row corpus is not something you page for.

THE SAME SIX QUESTIONS, ASKED OF BOTH SOURCES

What each source can and cannot answer

	Live managed API	Monthly public dump
Newest record available	12.03 min old, median of 10	32.73 days old at best
Current score and comment count	Yes, on every read	Frozen at export time
A field nobody exported	Request it again	Not recoverable from the file
Storage before the first query	None, query on demand	Hundreds of gigabytes
Small or new communities	Any community by name	Top splits only
Author of a since-deleted post	No, reflects the deletion	Yes, holds the original

History is the obvious one. If your question is about 2014, a live listing endpoint is a bad way to reach it: you would page for a very long time, and for many communities the cursor will not take you that far in practice. An archive scan answers the same question in one pass over local storage. The academic corpus loaders and precomputed embedding sets that rank on this term exist precisely because researchers needed exactly that and building it live was never the right shape.

Completeness of the deleted record is the finding from our churn pass, running the other way. Twenty five of a thousand year-old top posts no longer name an author through the live API. An archive written before those deletions still does, which is simultaneously the archive's strongest technical argument and the source of every ethical question anyone has ever asked about these corpora.

Cost per unit of volume is real too. If you want billions of records, paging a live API at roughly twenty one records per second per client is not the tool, and nobody should pretend it is. That is what the AI training corpus guide exists to cover, and its answer is a hybrid: bulk archive for the historical base, live API for the current tail, with a documented schema alignment between them. The same shape applies to retrieval-augmented generation over Reddit, where the historical base is a one-time cost and the freshness of the tail is the whole product.

What a dump does not do better is anything with a tense in the present. The comparison grid above is not a scorecard with a winner. It is a routing table.

The hybrid shape deserves one more sentence because it is what most serious builds converge on and it is rarely described honestly. You take the archive once, for the base, and you accept its cutoff as a known boundary rather than pretending it is the present. You run the live loop forward from that boundary. The hard part is neither half, it is the seam: the two sources use different field names for the same facts, they disagree about deleted records by construction, and a naive union produces duplicate rows for every record that appears in both. Deduplicate on the record id, prefer the live row for any mutable field, keep the archive row for anything the live read no longer returns, and write down which source each column came from. That last step is the one people skip, and it is the one that makes the resulting corpus explicable a year later.

What does a Reddit dataset cost you to hold, either way?

Less than people expect on the live side, and considerably more than the download page implies on the archive side, because the download is only the first bill. A post record is 1,620.8 bytes, so a million posts is roughly 1.6 GB of raw JSON, which is a laptop-sized problem. A bulk corpus is a terabyte-sized one, and the storage is only the first of three costs it carries.

On the live side, the arithmetic is straightforward. 1,620.8 bytes per post record means a hundred thousand posts is roughly 162 MB of raw JSON, and a million is about 1.6 GB. Collection at 20.71 records per second from one client puts that million at roughly 13.4 hours of sequential wall clock, derived as 1,000,000 divided by 20.71 divided by 3,600. Both of those are laptop-sized problems, and neither requires you to hold anything you did not ask for.

On the archive side, the first bill is bandwidth and disk and it is large. The threads are full of people meeting it. One person coming to the torrents for the first time put it like this:

"Hey guys i have never worked with Torrents. Is there a way to avoid downloading ALL comments and submissions? It show that it would be approximatly 2 TB which i dont have on my computer left. I only need submissions and comments from March 2021 and March 2022.."

He needed two months and was quoted two terabytes. Another, on a thread announcing a full submission corpus, said he wished he owned better hardware so he could examine the data as a whole and that so far he had only been able to look at it in parts:

r/

r/datasets·u/Stuck_In_the_Matrix

Full Reddit Submission Corpus now available (2006 thru August 2015)

00

Open on Reddit

The second bill is the index, and it is the one nobody quotes. A file is not a dataset until something can query it. On the Hacker News thread, a commenter asked directly whether there was any cheap way to search, noting that he already owned a large technical archive that was useless without search and that he did not want to grep everything. That is the real cost structure of a bulk corpus: download, then storage, then an index, then the operational work of keeping all three alive.

The per-community splits published on Academic Torrents exist to cut the first bill, and they are genuinely useful, but they carry their own limit that people discover late. As one commenter on a 2025 r/datasets thread put it while recommending exactly that route, the dump torrents cover only a limited number of subreddits. Coverage of the top N communities is not coverage.

If your comparison is against a managed API rather than against your own collection code, the price per call is the other half of the sum and the pricing breakdown covers it. The point of this section is narrower: the free file is not free, and the two costs it hides are the ones that arrive after you have already committed.

Is it legal to build your own Reddit dataset?

Collecting publicly visible content through the data API under Reddit's own terms is the documented route, and it is the one every measurement in this post used. The bulk archives sit on different footing, and you do not have to take our word for that because the people who maintain them say so in public.

The Reddit Data API Terms govern programmatic collection, and the API documentation is the reference for what each route returns. That is the surface you are working against when you build the live version, and it is a surface with a stated contract.

The archive position is more complicated and is worth quoting rather than paraphrasing. The person who maintains the per-community dump splits wrote this on a 2023 thread about exactly that question:

"I agree that reddit doesn't really have any legal methods of stopping people from just having the data. It's entirely possible they will try to sue people who make money off the data, especially lots of money. But they really don't care about research projects."

"That said, it is definitely unauthorized. Pushshift did not have permission from reddit to collect the data. Many, many other research projects have used it anyway, but it's still unauthorized." On a 2023 thread specifically about the legality of using the data dumps going forward, the person who maintains the per-community splits wrote that Reddit does not really have any legal methods of stopping people from simply having the data, that they probably do not care about research projects, and then added, plainly, that it is definitely unauthorised, that the archive did not have permission from Reddit to collect the data, and that many research projects have used it anyway. Another commenter in the same thread made the copyright argument in the opposite direction, that the content belongs to the people who wrote it rather than to Reddit.

Both of those views can be held by reasonable people and neither of them is legal advice, including when we repeat them. What matters for a build decision is the asymmetry: one route has a published contract you can read and comply with, and the other has a maintainer describing it as unauthorised. If your dataset is going into anything commercial, that asymmetry is the whole decision. We cover it at length in the legal read on collecting Reddit data and in what commercial use actually requires.

There is a third consideration that is not legal and matters anyway. Our churn pass found 25 of 1,000 year-old posts whose author is no longer returned. A live pipeline honours those deletions automatically because it re-reads. A stored corpus honours them only if you build something that does, and building that something is a choice you make once and then live with.

How big a sample do you need before the numbers mean anything?

Decide the window first, then page until you have covered it, and publish the window next to every figure. Sample size is the wrong question and it is the one everybody asks.

Our own freshness pass is the demonstration. Every one of the ten calls returned exactly 100 records, an identical sample size by every conventional measure, and those ten identical samples covered periods ranging from 0.47 hours to 2,692.91 hours. Comparing r/AskReddit and r/redditdev on those two pages is comparing 28 minutes against 112 days. No statistical test rescues that, and no amount of extra rows fixes it, because the defect is in the sampling frame rather than in the count.

The habit that fixes it is small. Every table you publish carries the window and the read date, and every chart caption says the same. Our tables carry asOf and sampleN for exactly this reason, and the methodology field on each one names what would falsify the result. That last part is the discipline that costs the most and pays back the most: if you cannot write down what result would prove you wrong, you have not made a measurement, you have made an assertion with a number in it.

There is a second sampling trap specific to this platform and it is worth naming because it is invisible. Automated accounts post on fixed schedules, so their contributions land in the same hour bucket every day. Any activity histogram built without filtering them describes a scheduler rather than a community. Our companion work on what the API tells you about an audience measured that directly and found the apparent peak posting hour of a large community moving by twelve hours once automation was filtered out. If your dataset is about human behaviour, the author column needs a filter before the timestamp column means anything.

For the sizing half of the question, the ranked list of communities by API-measured size is the place to start, and the subreddit analytics comparison covers what you can buy instead of measuring yourself.

What breaks first when you build this yourself?

The refresh loop, because it is invisible when it is missing. Everything else in a Reddit collection pipeline fails loudly and therefore gets fixed in the first week: a broken cursor stops the row count growing, a schema mismatch fails on insert, a rate limit returns a status code you can read. The refresh loop fails silently, and a silent failure in a data pipeline is the expensive kind.

Ranked by how long it takes to notice:

• Missing cursor. Minutes. The row count stops growing and you see it on the first run.
• Schema mismatch. Minutes. The insert fails with a type error.
• Rate limit. Hours. An HTTP status code you can read and back off from.
• Missing refresh loop. Never, until somebody compares your numbers to the site.
• Pinned input window. Never, because the job keeps succeeding against a source that stopped moving.

Here is the same list as prose. A missing cursor breaks immediately and obviously, because your job re-collects the same first page every run and the row count stops growing. A schema mismatch breaks on the first insert. A rate limit breaks with an HTTP status you can read. All three of those get fixed because they announce themselves.

A missing refresh loop announces nothing. The rows are there, the counts look right, the dashboard renders, and the engagement columns are simply a snapshot of the moment each row happened to be collected. In our measurement, 496 of 1,000 rows were already wrong 11.4 minutes in. After a day, a naive corpus of new posts is a corpus of first-observation values wearing the label "score". Nobody notices until somebody compares your numbers to the site.

The second failure by time-to-notice is the pinned window. A collection job that hardcodes a start date, or that reads the newest available file rather than asserting the file is new, will keep running successfully forever against an input that stopped moving. The producer looks healthy, the consumer looks healthy, and the dataset stops growing at a date nobody chose. The defence is one line: make an empty or unchanged input a hard error rather than a quiet no-op, and print the input you actually read on every run.

Third is the silent partial. A collection loop that swallows exceptions to keep going will happily record a run as complete when three of ten communities returned nothing. Count what you scanned against what you meant to scan, print both, and fail when they differ. That is the same discipline as the sample-window rule, applied to your own job rather than to your analysis.

The practitioners on the archive threads have their own version of this list, and it is mostly about the import rather than the collection. One described the design as stream everything or gather every N minutes and store everything in a database, then noted that importing from the bulk archive would be the painful half because the dataset is enormous. That split, easy live collection and painful historical import, matches what we see, and it is a good argument for doing the live half first and deciding later whether you actually need the history.

How do you check whether your own dataset has gone stale?

Run three assertions on a schedule and make each one fatal. Not a warning, not a dashboard tile, fatal, because a stale dataset looks exactly like a healthy one from the outside.

The first assertion is on the newest record. Read the maximum created_utc in your store, compare it to now, and fail if the gap exceeds what your collection cadence should permit. Our live pass gives you a calibration point: a median newest record 12.03 minutes old across ten communities, so a store whose newest row is a day old is not lagging, it is broken. Run the assertion per community rather than globally, because one dead community hides inside a healthy aggregate indefinitely.

The second assertion is on mutable fields. Take a random hundred records you hold, re-read them through the bulk id endpoint, and compare. A pipeline in good health will show some drift, because that is the nature of the data. A pipeline whose refresh loop has silently stopped will show drift that grows monotonically every time you run the check, and that pattern is the signal. Zero drift on a sample of new posts is also a finding, and not a good one: our measurement says roughly half of a fresh cohort should move inside twelve minutes, so a clean zero means the check is not checking.

The third assertion is on coverage. Count the communities you collected in the last window against the communities you meant to collect, and print both numbers. A count with no denominator cannot be wrong, which is what makes it useless. This is the same rule as the sampling one and it is broken the same way, by reporting a total that sounds large instead of a fraction that would look small.

For the freshness of the sources themselves rather than of your store, the archive release feed is public and readable in one request, which is how we measured the cadence in this post. The release feed tells you the newest month available, and the self-hosted archive project documents the import path. If you run a hybrid, checking that feed monthly is a five-second job that stops your historical base drifting a year behind without anyone noticing.

The failure mode this whole section exists to prevent has a shape, and researchers hit it constantly. Somebody builds a corpus, publishes work from it, and the corpus keeps being cited long after its last row. The static datasets that rank for this term are a museum of exactly that: a GitHub corpus from 2016, a catalog entry wrapping a fixed record count, a Kaggle listing with no refresh path. None of them is dishonest. Each one simply stopped, and nothing on the page says when.

The people who used the archive when it did update were clear about what they lost when it stopped:

Adarsh Mathew

@adarshmathew92

Looks like the Pushshift archive is dead. Enabler of so much interesting research, including mine. Thanks for everything, @jasonbaumgartne. Your work was invaluable, even if some entities chose to exploit it.

And the description of what the thing actually was, from someone watching it end, is the most precise sentence anyone has written about this whole subject:

Reddit Lies

@reddit_lies

Reddit just terminated Pushshift's access to Reddit's API. Pushshift was a constantly updating archive of Reddit comments and posts. This intentional obfuscation of data will have dire consequences for the platform. Watchdog remains unaffected. https://t.co/RzE7yjrfDT

The word that matters there is "constantly updating". What ended was not the file. It was the refresh.

Verdict

Build the live one, and keep a copy of the archive if your questions have a past tense. That is the recommendation the measurements support, and the reason is a gap of three orders of magnitude on freshness against a single genuine advantage the archive holds on deleted records. Both halves of that sentence were measured on the same day, and both are reported below with the sample sizes and the conditions that would falsify them.

The measurement is unambiguous on freshness. One call per community across ten communities on 2026-09-02 returned a newest record with a median age of 12.03 minutes, and the freshest was 21 seconds old. The freshest public monthly dump that day could contain nothing later than 2026-07-31, which is 32.73 days back, because those releases publish a median of 9 days after their covered month ends across the 15 consecutive releases we read. That is a ratio of 3,917 to one on the median, and it is a property of a monthly cadence rather than a failure of anyone's engineering.

The measurement is equally unambiguous on why that gap compounds. 496 of 1,000 records changed score or comment count in 11.4 minutes, 443 on score and 116 on comments, with a single post moving 458 points. A row you exported yesterday is not a slightly older version of the truth about a new post, it is a different number wearing the same column name. Re-read at 38.8 minutes rather than 11.4, the figure rose to 565 of 1,000 and the largest single move to plus 1,523 points, so the decay is monotone rather than a quirk of one short window.

Against that, the archive holds one thing the live read cannot. 25 of 1,000 posts from the past year no longer name an author through the API, concentrated at 9 of 100 in r/technology. If you need what the record said before somebody removed it, the file is the only place it exists, and that fact carries obligations as well as capabilities.

The build itself is smaller than its reputation. 100 records per call, 22 fields per post record and 70 per comment, 1,620.8 bytes per record, 20.71 records per second paging sequentially from one client, a cursor that walked five pages deep with 500 unique ids and no duplicates. Two loops, one for collection and one for refresh, an append-only observations table, and an as-of stamp on every mutable field. That is the entire design, and none of it is exotic.

You can run every measurement in this post against the documented endpoints on any plan, or sign up and reproduce the whole pass in an afternoon. If you do it on a different day and the median newest record comes back materially older than twelve minutes, that is a finding worth having and this post would need updating. The parts most likely to move first are the archive's newest release, which will advance by a month while the cadence stays the same, and the latency figures, which were one client on one afternoon and were never a benchmark. Everything else is arithmetic on 3,000 records we read out loud, and it is written down so somebody can check it. For the wider view of what the data API returns beyond posts, the data API overview, the user endpoint and the search tutorial are the next three stops, and PRAW remains the reference for how the same response shapes look from a Python client.