Reddit for AI Training Data: Arctic Shift, APIs, and the Clean Pipeline

Reddit is one of the most useful public corpora for training and grounding language models, because it is full of real questions, real answers, and real argument across tens of thousands of communities. The problem is access. The free index that researchers leaned on for a decade is gone, the official API is gated and capped, and the developers searching for "Arctic Shift" in 2026 land on a GitHub README and a pile of forum threads with no pipeline guide anywhere. This post is that guide. It covers what Arctic Shift actually gives you, where it stops, the REST API path for current data, how to format everything as JSONL for an LLM, and how to keep the whole corpus clean and deduplicated.

Not affiliated with Reddit Inc. redditapis.com is an independent third-party REST proxy for Reddit's API. This guide is vendor-neutral: it names the free bulk archives, shows the open-source tooling, and points you at the live API path only where bulk dumps cannot reach.

---

TL;DR: No single source gives you a complete, current Reddit training set. Arctic Shift is the maintained Pushshift successor for bulk history, roughly 2.5 billion items through February 2026 as ~261 GB of Parquet on Hugging Face, updated monthly, with a 4 to 6 week lag and no real-time path. Pushshift itself is dead for developers; PullPush is a partial successor with a 1 to 2 month recency gap. For data from the last 30 days you walk a paginated REST search or listings endpoint against live Reddit data, write each page to JSONL, and deduplicate on the shared Reddit post id. The clean pipeline is: Parquet for history, a live API for recent data, JSONL for the model, and an id set for dedup.

---

What you will build:

• A mental map of every Reddit data source in 2026 and the exact slice of time each one covers
• A paginated pull of current Reddit data, written straight to JSONL
• A pre-training corpus and an instruction-tuning dataset from the same raw posts
• A dedup layer that joins bulk Parquet history to live API pulls without double-counting

The pipeline this post builds: bulk history plus live current data, filtered, deduplicated, and written as one JSONL corpus

The confusion at the tool layer is the whole reason this guide exists. Two separate threads on the same question landed in r/DataHoarder within days of each other, and the question is the market pain in one line:

r/DataHoarder·u/pinkysooperfly

Pullpush vs Arctic Shift vs Pushshift Dumps?

00

Open on Reddit

---

What Is Arctic Shift (and What It Is Not)?

Arctic Shift is a community-maintained Reddit archive that picked up where Pushshift left off after Reddit's 2023 API changes broke the original index. It re-processes the recoverable Reddit corpus and republishes it as downloadable bulk dumps, with a free search and download front end at arctic-shift.photon-reddit.com and an open-source project on GitHub. As of early 2026 it covers roughly 2.5 billion posts and comments through February 2026, distributed as compressed Parquet on Hugging Face at around 261 GB. What it is not is a real-time API. The archive lags the live site by weeks, so it is built for historical and bulk research, not for pulling what was posted yesterday.

The Arctic Shift footprint in numbers, from the 2026 community summary

Those figures are not marketing. They come from a current community summary of Reddit data options on OpenData StackExchange, which describes Arctic Shift as the Pushshift successor on Hugging Face with a 2.5 billion item archive through February 2026 at roughly 261 GB of Parquet. A separate self-hosting project documented on Hacker News put its own figure at 2.38 billion posts and confirmed the monthly dump rhythm as the way archives stay current. The exact count drifts as new months land, but the order of magnitude is stable: billions of items, hundreds of gigabytes, refreshed monthly.

The access path is straightforward. The Hugging Face mirror lets you query the Parquet directly with DuckDB without downloading the whole thing, which is the cheapest way to pull a filtered slice. Here is a filtered read that pulls high-score machine learning comments without touching local storage:

import duckdb

# Stream a filtered slice straight from the Hugging Face Parquet mirror.
# Replace the dataset path with the current Arctic Shift HF dataset.
rows = duckdb.sql("""
    SELECT id, subreddit, author, body, score, created_utc
    FROM read_parquet('hf://datasets/<arctic-shift-dataset>/comments/**/*.parquet')
    WHERE subreddit = 'MachineLearning'
      AND score >= 5
      AND body NOT IN ('[deleted]', '[removed]')
    LIMIT 5000
""").df()

print(f"Pulled {len(rows)} filtered comments without a full download")

The columnar Parquet layout means filtering on subreddit, score, or date range reads only the relevant column chunks rather than scanning the full file, which is what makes a multi-billion-item corpus queryable on a laptop. DuckDB's documentation covers the remote-Parquet patterns directly. What you get back is the same field set most Pushshift-era scripts expected: id, subreddit, author, body, score, created_utc. That schema consistency is the reason Arctic Shift is a drop-in mental model for anyone who built on Pushshift before.

The hard limit is freshness, and it is worth stating plainly before you design around it. The most recent monthly dump is already a few weeks old when it lands, so the newest data Arctic Shift can give you is on the order of a month behind. For a longitudinal study of years of discourse that is irrelevant. For an AI pipeline that needs the last 30 days of community sentiment it is a wall, and it is exactly the wall the REST API section solves. If you are sizing the storage and throughput tradeoffs of working at this scale, the Reddit scraping benchmarks on throughput and error rates put real numbers on what bulk versus live access costs you.

The Arctic Shift tradeoff: deep history and zero cost on one side, a multi-week freshness wall on the other

The freshness ceiling was acknowledged by the maintainers of the tools built on top of the dumps. One self-hosting project on Hacker News framed it as a future feature rather than a solved problem, planning to import the monthly Arctic Shift dumps so local archives could update monthly at best. Monthly is the ceiling, not a temporary state. Any design that assumes Arctic Shift can supply recent data will hit this, so the right framing is to treat the dumps as your deep backfill and source recency elsewhere.

How current each source is: only a live API reaches the last few weeks, and every bulk archive trails it by a month or more

---

Why Pushshift Died (and What PullPush Is)

Pushshift died for public developers in May 2023, when Reddit revoked its API access as part of the same pricing change that shut down most third-party Reddit apps. Pushshift had been the free, Reddit-wide index behind more than 1,700 scholarly papers, and when Reddit cut off its feed from the firehose the data simply stopped updating. A restricted moderation-only program returned later, but it is gated to verified subreddit moderators and is useless for research or training. PullPush (pullpush.io) is the community-run successor that rebuilt an API on the recoverable archive, and it works for a defined historical window but carries a real recency gap that you have to design around.

The Pushshift arc: a decade as the standard, then a hard cutoff in May 2023 that forced the whole ecosystem to fork

The cutoff was a public event. Reddit announced it would begin charging for API access in April 2023, and the pricing that followed killed the economics that let Pushshift run for free. The community consensus on what replaced it is settled. As one widely-upvoted Hacker News comment put it, Arctic Shift is the project that picks up where Pushshift left off when the API changes killed it. That is the clean one-line summary: Pushshift is the dead standard, Arctic Shift is the maintained bulk successor.

PullPush is the third name in the confusion, and it deserves an honest read rather than a dismissal. It exposes a Pushshift-style REST interface and covers a usable slice of the corpus, which is why it still shows up in peer-reviewed work. A 2026 study in PubMed Central collected its Reddit data through the PullPush API for a February to March 2025 window, so the tool genuinely works for defined historical periods. The catch is recency. Developers report that PullPush stops returning results for posts within the last one to two months, a gap confirmed repeatedly in r/redditdev. So PullPush is reliable for older data and unreliable for anything recent, which is the same shape of limit Arctic Shift has, just at a different boundary.

The dependency pain is not historical cleanup either; it is ongoing. A mid-2026 outage notice on r/pushshift confirms developers are still hitting walls with these community tools in real time:

r/pushshift·u/Pushshift-Support

OUTAGE: Pushshift API and data

00

Open on Reddit

That ongoing fragility is the engine behind the search-volume surge for "arctic shift reddit" and "pullpush api." People keep arriving at these tools, keep hitting their limits, and keep looking for the working stack. The honest map below assigns each tool to the job it actually does well, so you stop trying to make one source cover all three.

Each source maps to one job: torrent dumps for offline depth, Arctic Shift Parquet for maintained bulk, PullPush for older REST history

The reasonable read in 2026 is to stop hunting for the single tool that replaces Pushshift, because none of them do. Pushshift combined Reddit-wide search, full history, bulk export, and real-time ingestion in one free service, and the surviving options each cover a fragment. The deeper comparison of every surviving bulk option, with rate limits and reliability records, lives in the best Pushshift alternatives breakdown; for the bigger picture on why Reddit closed the free era, the Reddit data API access guide covers the official side.

---

How Arctic Shift and PullPush Compare in 2026

Both Arctic Shift and PullPush rebuilt from the recoverable archive after Pushshift lost its feed, but they solve different parts of the problem and have different ceilings. The choice between them is not a preference question; it is a question of what your pipeline actually needs.

	Arctic Shift	PullPush
Format	Compressed Parquet on Hugging Face	REST API (Pushshift-compatible)
Coverage	~2.5B items through Feb 2026	Historical window, 1 to 2 month recency gap
Access	Free download, DuckDB-queryable	Free REST, no auth required
Best for	Bulk historical corpus	Pushshift-compatible scripts and older data
Weakness	No REST API, 4 to 6 week dump lag	Recency gap, community-run reliability

The practical read: if you need bulk history in Parquet form, use Arctic Shift. If you have existing scripts that call a Pushshift-style REST interface, PullPush is the drop-in. For data within the last 30 days, neither covers it, which is the gap the live REST API section addresses below. Neither tool provides a created_utc filter that reaches the current week, so anything time-sensitive routes through the live API path.

---

The REST API Path for Current Reddit Data

The REST API path is how you get the last 30 to 60 days of Reddit data that no bulk dump covers yet. Where Arctic Shift and PullPush both lag by weeks, a paginated search or listings call against live Reddit data returns content from this morning. The pattern is the same one Reddit's own listings use: you send an authenticated GET, read the page of posts plus an after cursor, pass that cursor back to fetch the next page, and write each batch to a JSONL file as you go. A maintained REST proxy such as https://api.redditapis.com wraps this with a single bearer token and a clean IP pool, so you skip the OAuth app-registration queue and the datacenter IP filtering that breaks raw calls from a server. The full endpoint set is documented at docs.redditapis.com.

The core loop is a search call that walks the cursor. Here it is end to end, writing every page straight to JSONL:

import os
import json
import time
import requests

API_KEY = os.environ["REDDIT_APIS_KEY"]
BASE = "https://api.redditapis.com"
HEADERS = {"Authorization": f"Bearer {API_KEY}"}

def pull_to_jsonl(query, out_path, subreddit=None, max_pages=10, pause=0.5):
    """Paginate the search endpoint and append each post as one JSON line.
    Walks the `after` cursor until it runs out or max_pages is reached."""
    after = None
    written = 0
    with open(out_path, "a", encoding="utf-8") as fh:
        for _ in range(max_pages):
            params = {"q": query, "sort": "new", "limit": 100}
            if subreddit:
                params["subreddit"] = subreddit
            if after:
                params["after"] = after
            resp = requests.get(
                f"{BASE}/api/reddit/search",
                params=params, headers=HEADERS, timeout=60,
            )
            resp.raise_for_status()
            data = resp.json()
            for post in data["posts"]:
                fh.write(json.dumps(post, ensure_ascii=False) + "\n")
                written += 1
            after = data.get("after")
            if not after:
                break
            time.sleep(pause)
    return written

count = pull_to_jsonl("local llm fine tuning", "reddit_raw.jsonl",
                      subreddit="LocalLLaMA", max_pages=5)
print(f"Wrote {count} posts to reddit_raw.jsonl")

The response shape is a top-level posts array plus an after cursor. Each post object carries id, name, title, author, permalink, url, text, subreddit, upvotes, comments, upvote_ratio, over_18, created_utc, and more. Reddit caps any single search query at roughly 1,000 results across ten pages, so do not chase max_pages past 10 on one keyword; instead vary the query or scope it to different subreddits to widen coverage. The same after-cursor mechanics and the field-name mapping are covered in depth in the Reddit search API tutorial.

The current-data loop: page on the after cursor, append to JSONL, and track the max created_utc so the next run starts where this one stopped

For a subreddit-wide pull rather than a keyword search, the listings endpoint is the better tool. It returns recent posts for a subreddit and pages the same way, which is what you want when you are building a corpus of an entire community rather than matching a term:

import os
import json
import requests

API_KEY = os.environ["REDDIT_APIS_KEY"]
BASE = "https://api.redditapis.com"
HEADERS = {"Authorization": f"Bearer {API_KEY}"}

def recent_posts(subreddit, limit=100):
    resp = requests.get(
        f"{BASE}/api/reddit/posts",
        params={"subreddit": subreddit, "sort": "new", "limit": limit},
        headers=HEADERS, timeout=60,
    )
    resp.raise_for_status()
    return resp.json()["posts"]

posts = recent_posts("MachineLearning")
newest = max(p["created_utc"] for p in posts) if posts else 0
print(f"{len(posts)} recent posts, newest created_utc = {newest}")

The piece that makes this repeatable is incremental dedup. Store the maximum created_utc you saw on each run, persist it, and use it as the lower bound on the next run so you never re-fetch a window you already have. That single number turns a one-shot scrape into a daily refresh that only ever pulls new content. The clean-IP routing behind a managed endpoint matters here too, because the native path returns 403 from most cloud hosts no matter how correct the code is; the REST versus PRAW comparison and the PRAW versus a hosted REST API breakdown cover that tradeoff, and the authentication and OAuth guide covers the token mechanics. If you are pulling at real volume, the rate limits reference shows where pacing matters. You can get REST API access at redditapis.com/signup and run the loop above against live data in a couple of minutes; the no-PRAW Python tutorial covers the surrounding read and write calls.

---

Schema Alignment: Mapping Arctic Shift Fields to the REST API

The field names differ between the Arctic Shift Parquet dump and a REST API response, and normalizing them before the dedup pass saves debugging time downstream. The key field mappings to handle:

• score in Parquet corresponds to upvotes in the REST response
• body in Parquet corresponds to text in the REST response
• created_utc is consistent across both sources
• id is the shared dedup key and is consistent in both

Add a normalization step before any merge operation so downstream code sees a single field shape:

def normalize(post, source="rest"):
    """Normalize a post dict to a common schema regardless of source."""
    if source == "parquet":
        return {
            "id": post["id"],
            "subreddit": post["subreddit"],
            "title": post.get("title", ""),
            "text": post.get("body", ""),
            "upvotes": post.get("score", 0),
            "created_utc": post["created_utc"],
        }
    return post  # REST responses already use the standard field names

Applying this before any filter or dedup pass keeps the rest of the pipeline source-agnostic: your filtering code references upvotes and text regardless of whether a given post came from Parquet or the live API. The full field reference for the REST endpoint is at docs.redditapis.com.

---

Formatting Reddit Data as JSONL for LLM Training

JSONL is the standard on-disk shape for LLM training data because it streams: one JSON object per line, read incrementally, no need to load the whole file into memory. Formatting raw Reddit posts into a training corpus is mostly a filter-and-shape job. For a pre-training corpus you concatenate the post title and body into one text document, drop deleted and removed bodies, and apply a score threshold to cut noise. For instruction tuning you build pairs: a post title or question as the prompt, a high-scoring comment as the completion, written as a {"prompt": ..., "completion": ...} object per line. The same raw pull feeds both shapes; what differs is how you assemble each line.

Start with the pre-training shape, because it is the simpler of the two. Read the raw JSONL, filter, and emit a clean text field:

import json

def build_pretrain_corpus(in_path, out_path, min_score=5, min_len=50):
    kept = 0
    with open(in_path, encoding="utf-8") as src, \
         open(out_path, "w", encoding="utf-8") as dst:
        for line in src:
            post = json.loads(line)
            body = (post.get("text") or "").strip()
            if body in ("", "[deleted]", "[removed]"):
                continue
            if post.get("upvotes", 0) < min_score:
                continue
            text = f"{post['title']}\n\n{body}".strip()
            if len(text) < min_len:
                continue
            dst.write(json.dumps({"text": text}, ensure_ascii=False) + "\n")
            kept += 1
    return kept

print(build_pretrain_corpus("reddit_raw.jsonl", "reddit_pretrain.jsonl"))

The filters are doing the real work. A minimum score drops the long tail of unseen posts that add noise without signal, the deleted and removed check strips placeholder bodies that would otherwise teach the model the string [removed], and a minimum length cut removes one-word posts. Add language detection with a library such as langdetect if your corpus needs to stay in one language, since Reddit is multilingual and a mixed corpus dilutes a single-language model. The field names follow the managed endpoint, where score is upvotes and the body is text; if you are reading Arctic Shift Parquet instead, those are score and body, so normalize to one internal shape before this step.

Keep the fields that carry signal or enable joins; drop placeholder bodies and low-score noise before they reach the model

Instruction tuning is where Reddit's structure pays off, because the title-and-top-comment shape is a natural question-and-answer pair. Pull the comment tree for a matched post by permalink, take a high-scoring top-level comment as the response, and wrap the pair:

import os
import json
import requests

API_KEY = os.environ["REDDIT_APIS_KEY"]
BASE = "https://api.redditapis.com"
HEADERS = {"Authorization": f"Bearer {API_KEY}"}

def make_pair(post, min_comment_score=10):
    """Build one instruction-response pair from a post and its top comment."""
    tree = requests.get(
        f"{BASE}/api/reddit/comments",
        params={"permalink": post["permalink"]},
        headers=HEADERS, timeout=60,
    ).json()
    comments = tree.get("comments", [])
    top = max(comments, key=lambda c: c.get("upvotes", 0), default=None)
    if not top or top.get("upvotes", 0) < min_comment_score:
        return None
    return {"prompt": post["title"], "completion": top["text"]}

# write pairs for a list of posts already pulled
with open("reddit_sft.jsonl", "w", encoding="utf-8") as fh:
    for post in posts:
        pair = make_pair(post)
        if pair:
            fh.write(json.dumps(pair, ensure_ascii=False) + "\n")

The comment endpoint shape is documented at docs.redditapis.com; fetching a known thread by permalink is cheaper and more complete than a second comment search. One quality note worth holding onto: the subreddit you pull from matters more than the volume you pull. High-score posts from focused technical communities produce far better fine-tuning signal than a general pull from a default front-page subreddit, where the top comments are jokes rather than answers.

Signal density by community: focused technical subs make strong instruction pairs, broad general subs make weak ones

The practical takeaway is to build a subreddit allow-list before you pull, not after. Decide which communities match your model's domain, scope your pulls to them, and you cut the filtering burden downstream because the raw data is already higher quality. Finding and vetting those communities at scale is its own task; the find-subreddits API guide covers discovering them programmatically. If your pipeline also needs to write back to Reddit, for example to collect responses to a prompt you post, the DM-via-API guide and the broader MCP servers roundup cover the agent-side surface.

The last formatting concern is dedup across sources, which is the join between your bulk history and your live pulls. Because Arctic Shift Parquet and the live API share the same Reddit post id namespace, the id is your key:

import json

def dedup_sources(historical_ids_path, live_jsonl, out_path):
    seen = set()
    with open(historical_ids_path, encoding="utf-8") as fh:
        for line in fh:
            seen.add(line.strip())  # one Reddit post id per line from Parquet
    written = 0
    with open(live_jsonl, encoding="utf-8") as src, \
         open(out_path, "w", encoding="utf-8") as dst:
        for line in src:
            post = json.loads(line)
            if post["id"] in seen:
                continue
            seen.add(post["id"])
            dst.write(line)
            written += 1
    return written

print(dedup_sources("historical_ids.txt", "reddit_raw.jsonl", "reddit_deduped.jsonl"))

Load every id from the historical Parquet into a set once, then stream the live JSONL past it, skipping ids you already hold and adding the rest. The set lookup is constant time, so this scales to tens of millions of ids on a normal machine. That single dedup pass is what lets you merge a multi-billion-item backfill with a daily live refresh and end up with one corpus that counts each post exactly once.

---

Choosing Subreddits for High-Quality Training Data

The subreddit you pull from is the biggest single quality lever in the pipeline. Focused technical communities produce precise, informative answers; broad default subs produce memes and jokes. Build your allow-list before you pull rather than filtering after, and the raw data arrives already higher quality.

Communities that produce strong instruction-tuning signal for developer-facing models:

• r/LocalLLaMA: practitioners discussing open-source LLM fine-tuning, quantization, and deployment; high signal density for applied ML questions
• r/MachineLearning: academic and applied ML discussion, links to papers, experimental results
• r/learnprogramming: clear question-and-answer structure, beginner-to-intermediate technical range
• r/Python: library usage questions, debugging discussions, idiomatic code reviews
• r/datascience: data wrangling, modeling decisions, career-adjacent technical content
• r/learnmachinelearning: structured beginner questions with community-corrected answers

Communities to exclude from instruction-tuning corpora (fine for pre-training volume but weak for pairs):

• Humor and entertainment subs where top comments are jokes rather than answers
• Default front-page subs where upvote mechanics reward broad appeal over accuracy
• Political subs where the discourse structure is adversarial rather than informative

For pre-training corpora the allow-list can be wider, since you are collecting text rather than question-answer structure. For instruction tuning it should be narrow: ten high-quality communities outperform a hundred general ones. The find-subreddits API guide covers programmatic discovery if you need to build the allow-list from a domain description rather than a hand-curated list.

---

Filtering and Cleaning the Raw Corpus

Cleaning the raw pull before it reaches the model is cheaper than correcting a trained model after the fact. The filters that matter most, applied in this order, with each pass running as a streaming read over the JSONL rather than loading everything into memory:

1. Drop deleted and removed bodies: filter body not in ('[deleted]', '[removed]') before any other step; these placeholder strings add noise with no signal
2. Apply a minimum score threshold: upvotes >= 5 on posts and upvotes >= 10 on comments removes the long tail of unseen content; raise to >= 20 for instruction-tuning pairs where quality matters more than volume
3. Enforce a minimum length: drop posts where len(title + body) < 50; one-word posts and link posts without a body produce weak training examples
4. Run language detection: Reddit is multilingual; if your model is English-only, filter with langdetect or fastText before writing to the corpus
5. Strip or anonymize usernames: replace with a [USER] token in any corpus you plan to publish or release; usernames are personal data in most privacy frameworks
6. Deduplicate on id: run the id-set dedup pass at the end so exact duplicates from overlapping time windows never land in the same file

Apply each filter as a streaming pass over the JSONL rather than loading everything into memory first. A line-by-line generator keeps the pipeline memory-stable at any corpus size.

---

Legal and Responsible-Use Framing

Using public Reddit data for AI is governed by Reddit's terms and any license you hold, and the line that matters is between accessing individual public posts through a published API and bulk crawling the corpus for a commercial training set. Reddit now licenses its data commercially to AI companies, but that is the company monetizing its own corpus, a separate activity from how you read public posts programmatically. Reading public content through an API that respects rate limits and terms is not the same as bulk crawling in breach of those terms. This is an engineering explainer, not legal advice; for large-scale training use, review the current terms and consult counsel.

The commercial licensing context is worth understanding because it explains the access landscape you are working in. Reddit disclosed approximately $203 million in total data licensing agreements covering 2024, including a deal reported at around $60 million a year with Google and around $70 million a year with OpenAI. Those are the deals that priced Reddit's corpus as a commercial asset and ended the free-archive era. The shift from those flat annual fees toward usage-based and dynamic pricing is the subject of the parent guide on Reddit's usage-based AI data licensing, which is the policy complement to this engineering walkthrough.

Researchers noticed the shift well before the licensing deals were public. As Ethan Mollick, a Wharton professor who tracks AI adoption, noted when studying the impact of generative AI on online communities:

Ethan Mollick

@emollick

ChatGPT is undermining question-and-answer communities, like StackOverflow, but not ones that require human interaction, like Reddit… yet. (Both are sources for training data) The paper also discusses how to revitalize user communities in the age of AI. https://t.co/gTvSolK8fO h… Show more

The commercial deals that reset the landscape: Reddit's corpus repriced as a licensed asset, reported in 2024 figures

What this means in practice for a builder is a short list of constraints:

• Keep to public subreddits and skip private, restricted, or quarantined ones
• Filter out deleted and removed content rather than trying to recover it, both because it is low-quality and because users removed it deliberately
• Avoid tying usernames to real-world identities in any published dataset
• Respect the rate_limit of whatever path you use; hammering a service hard enough to degrade it is the behavior that gets access revoked for everyone

None of this is onerous, and most of it overlaps with simply building a clean corpus: deleted bodies and private-sub content are noise you would filter anyway.

The managed REST route helps on the compliance surface as well as the engineering one. A path that holds Reddit access and exposes it over a single token with per-call pricing keeps you inside published limits by design and gives you a billing line you can point at, which matters more as licensing tightens. For where that sits against the official API and a bulk crawl on cost, the pricing-versus-Apify comparison and the ranked alternatives guide lay out the three routes. Commercial managed scrapers such as Apify sit at a different point on that curve for teams that want infrastructure handled.

---

Running the Pipeline Incrementally

A one-shot pull gets you the initial corpus; an incremental daily refresh keeps it current without re-fetching history. The two state values to persist between runs are last_utc (the maximum created_utc seen from the live API) and seen_ids (the set of id values already in the corpus). With those, each run costs exactly one new time window.

The incremental loop at a high level:

1. Load seen_ids from the id file (one Reddit post id per line)
2. Load last_utc from the state file (a Unix timestamp integer)
3. Walk the REST search endpoint with a time filter starting from last_utc
4. For each page: skip posts whose id is in seen_ids, write the rest to JSONL, add their id to seen_ids, update last_utc to the maximum created_utc seen
5. After the pull: write the updated seen_ids and last_utc back to disk
6. Schedule the loop as a daily cron or a GitHub Actions workflow so it runs without manual intervention

The state files stay small: even a corpus of 10 million posts needs only a 200 to 300 MB id file, which fits comfortably in memory for the set lookup. The rate limits reference covers the pacing parameters to set so the loop respects per-minute and per-hour limits without manual tuning.

---

Putting the Pipeline Together

The clean Reddit training pipeline in 2026 is three sources joined by one key. Arctic Shift Parquet supplies the bulk history: billions of items through the latest monthly dump, free and DuckDB-queryable. A paginated REST API supplies the recent window the dumps cannot reach, written page by page to JSONL. The Reddit post id deduplicates across both so each post lands in the corpus once. From there, the formatting layer shapes the deduplicated stream into either a pre-training text corpus or instruction-response pairs, with score, length, and language filters applied before anything reaches the model. That is the whole machine, and each part covers exactly the gap the others leave.

The complete loop: bulk plus live, joined on id, filtered, and formatted into the training shape your model needs

The reason to build it this way rather than reaching for one source is that the developers asking "what is the working stack in 2026" on r/redditdev keep getting fragments. There is no single hosted endpoint that hands you a complete, current, training-ready Reddit corpus, and pretending otherwise is how people end up with a dataset that is either stale by a month or missing a decade of history. The split design is the answer: deep history is a solved problem with free Parquet, recency is a solved problem with a live API, and the join is a ten-line dedup pass. For a deeper read on the broader tool landscape this sits in, the GummySearch alternatives breakdown by data depth maps the consumer end of the same space.

For a hands-on look at wiring Reddit data into an AI workflow end to end, this walkthrough of an AI-assisted Reddit scraping pipeline lines up closely with the split-source approach above:

A community snapshot of how routine this has become, with practitioners now treating Reddit as a default training source, frames why a clean access path keeps getting more valuable rather than less:

bayes

@bayeslord

training on reddit data. call that midtraining

---

The Bottom Line

Reddit is one of the best public corpora for training and grounding language models, and assembling a clean version of it in 2026 comes down to using the right source for each slice of time. Arctic Shift is the maintained Pushshift successor for bulk history, roughly 2.5 billion items through February 2026 as ~261 GB of Parquet, free and queryable but weeks behind the live site. Pushshift itself is gone for developers, and PullPush covers older data with a recency gap. For the last 30 to 60 days you walk a paginated REST search or listings endpoint against live Reddit data, write each page to JSONL, and deduplicate on the shared post id. Format the deduplicated stream into a pre-training corpus or instruction pairs, filter by score, length, and language, and scope your pulls to focused communities so the raw data is high quality before it reaches the model. The pipeline is Parquet for history, an API for recency, an id set for the join, and JSONL for the model. Grab a free key and pull current Reddit data into JSONL.