Reddit JSON requests can return HTTP 502 or an HTML page with no usable data.
The scraper now tries the public Reddit RSS feed after a failed JSON request.
It reuses the working format for later requests in the same run.
- Failed requests use a new proxy connection, a bounded delay, and at most three attempts.
- Each request has a 75-second total deadline.
- Retry delays support both forms of the
Retry-After header.
- RSS pagination preserves the subreddit, post order, and top period.
- RSS records include
data_source: "reddit_rss". Scores and comment counts are null because Reddit omits them from RSS.
- RSS comments have
null scores and depths. Reddit can limit the number of comments in a feed.
- A comment failure preserves the post and any comments already collected. Other posts continue, and the run reports the incomplete input.
- The run summary includes request paths, status codes, retry counts, and RSS use. Proxy credentials stay out of these records.
The scraper still reports a failed run when all access attempts fail. A missing subreddit does not count as an empty success.