Moving a ScraperAPI Reddit job to a REST call.
Most Reddit jobs on a generic scraper are one of four page types: a subreddit listing, a search results page, a thread, or a user page. Each one maps to a single endpoint here, which is what makes the switch small. A listing becomes GET /api/reddit/posts with a subreddit and a sort, a search page becomes GET /api/reddit/search with a q parameter, a thread becomes GET /api/reddit/post/ followed by the post id, and a user page becomes GET /api/reddit/user/ followed by the name.
1. Swap the fetch
Replace the call that sent a Reddit URL through the scraper with a GET to api.redditapis.com and an Authorization header carrying your key as a bearer token. The rest of your HTTP client, timeouts included, stays as it is.
2. Delete the parser
The response is already JSON with named fields: title, text, author, subreddit, upvotes, comments, created_utc and permalink for a post. Map those names onto whatever your storage expects and remove the selectors, which are the part that used to break.
3. Switch pagination to the cursor
Listings return an after value. Pass it back unchanged on the next request and stop when it comes back null, instead of building the next page URL yourself. It is an opaque cursor, so do not construct one from a Reddit fullname.
4. Keep the retry wrapper, but loosen it
Blocks and retries against Reddit are now handled on our side, so your client only needs ordinary handling: back off on a 503, which means upstream capacity is briefly busy, and on a 429, which can also mean a third deep comment search started while 2 are still running on your account or Reddit throttling the account session behind an authenticated read (home feed, private listings). Every 429 carries a Retry-After header to wait out. Reddit throttling a write comes back as a 5xx error (usually a 502) instead, not a 429, with code REDDIT_RATELIMIT when you create posts too fast. Top up when a 402 says the balance is empty.