Substack Scraper & Medium Scraper - Newsletter & Blog Posts

What does Substack Scraper & Medium Scraper do?

It returns the posts of any newsletter or blog from its public feed (RSS, Atom or JSON Feed): Substack newsletters (also on custom domains), Medium authors, publications and tags, Ghost, Beehiiv, WordPress, Blogger, Hashnode or any site with a feed. Paste newsletter URLs, blog home pages, feed URLs or Medium handles; get one clean row per post. No start fee, no login: $1 per 1,000 posts, so small jobs and daily newsletter monitoring stay cheap.

• 📰 Feeds, not page scraping: feeds are published by the platforms for syndication and readers. No login, no browser, no cookies. The feed is found automatically from a home page (<link rel="alternate">, then the usual /feed, /rss, /atom.xml paths)
• 🔗 Any input: https://www.lennysnewsletter.com, https://name.substack.com, substack:name, https://medium.com/@user, @user, https://medium.com/better-programming, https://medium.com/tag/python, https://www.platformer.news, https://wordpress.org/news/feed/. The same feed pasted twice is read once, and a post that two inputs lead to is returned (and charged) once
• 🧾 Clean post rows: title, subtitle, url, author, publishedAt (ISO, UTC), excerpt, text (plain text), tags, imageUrl, wordCount, readingTimeMinutes, platform, source. Podcast episodes also get audioUrl
• 💎 Substack extras: paid or free (audience), the full post's word count (postWordCount), likes, restacks, and posts older than the feed's latest 20 from the newsletter's public archive list
• 🔒 Paywalls respected, and marked: posts for paid subscribers come back with their public preview and paywalled: true. The Actor never logs in and never tries to get around a paywall
• 📄 Optional full text for posts whose feed carries only a summary (Medium member snippets, summary-only blogs): the post page is read and its public text returned. Charged only when the page had more text than the feed
• 🎯 Filters before charging: published since (2026-09-01 or 7 days), keywords (any of), excluded words, authors. Filtered-out posts are free
• 🆕 Only new posts since the last run: for daily or hourly monitoring of many newsletters. Posts already seen are not output and not charged
• 💵 $1 per 1,000 posts, full text from the page $1 per 1,000. No start fee

Where the data comes from, and the limits

Platform	Feed the Actor reads	What the feed contains
Substack (name.substack.com or a custom domain)	/feed + the newsletter's public archive list (/api/v1/archive, the list its own /archive page shows)	Latest 20 posts in the feed with full text for free posts and a preview for paid posts; the archive adds older posts (title, subtitle, date, excerpt), paid/free status, word count and likes
Medium (author, publication, tag)	https://medium.com/feed/@user, /feed/<publication>, /feed/tag/<tag>	Latest 10 stories; full text for most stories, a snippet for many member-only stories
Ghost, Beehiiv, WordPress, Blogger, Hashnode, any blog	The feed the site advertises, or /feed, /rss, /atom.xml…	Usually the latest 10–50 posts; full text or a summary depending on the site's settings; members-only posts often come without text

What this means in practice:

• Feeds carry recent posts only. For a newsletter's full history use the Substack archive option (on by default) or schedule the Actor with Only new posts so every post is captured as it comes out.
• Paid posts stay paid. A paid Substack post returns the preview the newsletter publishes in its feed; a Ghost or Medium members-only post returns its summary or public opening. Such rows say paywalled: true and textComplete: false, with the reason in paywallNote.
• Full text is read from the public page and cut where the site's paywall notice starts ("This post is for paying subscribers only", Medium's "Create an account to read the full story"). A page that shows fewer words than its own "7 min read" label is treated as partial.

Who is it for?

• Newsletter and media analysts: track what a set of newsletters publish, how often, how long, and which posts are paid
• Marketers and PR: monitor mentions of your brand, competitors or topics across Substack, Medium and industry blogs
• AI and RAG builders: clean plain text of posts for summaries, embeddings and daily digests
• Researchers: build datasets of posts per author, publication or Medium tag
• Readers and teams: one feed of new posts from many sources into Slack, Sheets or email

How much does it cost?

Price per 1,000 events by Apify plan:

Event	Free	Starter	Scale	Business and higher
Post	$1.00	$1.00	$0.90	$0.80
Full text from the post page (optional)	$1.00	$1.00	$0.90	$0.80

No start fee. Not charged: posts removed by your filters, posts already seen with Only new posts, sources where no feed could be found or the website could not be reached, invalid input lines (including private addresses such as localhost), posts that an earlier input of the same run already returned, full-text reads of paywalled posts, and full-text reads where the page had no more text than the feed. A paywalled post itself (its row with the public preview) is charged as a normal post.

For comparison (checked September 2026, Apify Store, Free plan prices): moving_beacon-owner1/medium-scraper charges $0.01 per result; automation-lab/substack-scraper $0.00115 per post ($0.0023 with content) plus $0.005 per run; automation-lab/medium-scraper $0.0023 per article plus $0.005 per run; fatihtahta/substack-scraper $0.0025 per record; easyapi/substack-posts-scraper $0.00499 per item plus $0.09 per run; automation-lab/rss-feed-reader $0.00115 per item plus $0.035 per run; technicaldost/rss-feed-scraper $0.008 per item. A few RSS readers are cheaper per item (scrapesignal_labs/universal-feed-reader $0.0001) but read plain feeds only, without Substack/Medium handling, paywall status or full text.

Control your cost

• What is charged: each post returned, and (optional) each full text read from a post page when it added text.
• Max posts per source (default 50) and Max posts for the whole run (default 1,000: at most $1 on the Free plan) cap the charge.
• Filters are applied before charging, so "posts about AI agents in the last 7 days" costs only for those posts.
• Only new posts makes schedules cheap: after the first run you pay only for posts published since the previous run.
• At the start, the run logs its worst case (e.g. Plan: 40 posts × $0.001 + 40 full texts fetched from post pages × $0.001 = at most $0.08) and warns when that is more than your maximum charge per run.
• When the maximum charge per run is reached, the run stops, keeps everything found so far, and says so. The SUMMARY record then has status: "LIMIT_REACHED" and notProcessed (up to 100 sources not processed).

Worked example (real run, 30 September 2026): 8 sources (Lenny's Newsletter, Astral Codex Ten, Medium tag "python", The Neuron on Beehiiv, Platformer on Ghost, WordPress News, Julia Evans' blog, and example.com, which has no feed), 5 posts per source, full text on: 35 posts from 7 sources in 6.4 seconds; 4 Medium stories got their full text from the page; 5 paywalled posts were returned with their public part (not charged for full text); example.com got a free error row. Cost on the Free plan: 35 × $0.001 + 4 × $0.001 = $0.039.

How to use it

1. Enter your newsletters, blogs or feeds, one per line.
2. Optional: set Max posts per source, Posts published since, keywords, excluded words or authors.
3. Optional: turn on Fetch full text of truncated posts and/or Only new posts since the last run (then schedule the Actor).
4. Click Start. The Posts table is in the Output tab; export as CSV, Excel or JSON, or call the Actor from the API.

Input example

{
    "sources": [
        "https://www.lennysnewsletter.com",
        "substack:astralcodexten",
        "https://medium.com/tag/python",
        "https://www.theneurondaily.com",
        "https://www.platformer.news"
    ],
    "maxPostsPerSource": 5,
    "fetchFullText": true
}

New posts about AI from a list of newsletters, every morning:

{ "sources": ["https://www.platformer.news", "https://www.theneurondaily.com", "@karpathy"], "keywords": ["AI", "agents"], "onlyNewPosts": true, "monitorName": "ai-newsletters" }

Output example: post (Substack, free)

{
    "type": "post",
    "input": "https://www.lennysnewsletter.com",
    "source": "Lenny's Newsletter",
    "sourceUrl": "https://www.lennysnewsletter.com",
    "feedUrl": "https://www.lennysnewsletter.com/feed",
    "platform": "substack",
    "title": "All of the Lenny & Friends Summit talks are now online!",
    "subtitle": "Plus, some reflections and takeaways from the day",
    "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
    "author": "Lenny Rachitsky",
    "publishedAt": "2026-09-29T13:15:57.000Z",
    "excerpt": "Plus, some reflections and takeaways from the day",
    "text": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice. …",
    "wordCount": 1275,
    "readingTimeMinutes": 5,
    "postWordCount": 1276,
    "textComplete": true,
    "textSource": "feed",
    "paywalled": false,
    "audience": "everyone",
    "likes": 250,
    "restacks": 3,
    "substackPostType": "newsletter",
    "imageUrl": "https://substack-post-media.s3.amazonaws.com/public/images/9fed347c-0e02-4190-a590-1236719b99de_9213x6142.jpeg",
    "charged": true
}

Output example: paywalled post (Ghost, members only)

{
    "type": "post",
    "source": "Platformer",
    "platform": "ghost",
    "title": "Following: OpenAI taps the brakes",
    "url": "https://www.platformer.news/open-ai-model-release-canceled/",
    "author": "Ella Markianos",
    "publishedAt": "2026-09-29T00:16:11.000Z",
    "text": "On the eve of the company's developer conference, it cancels a new model over safety fears",
    "wordCount": 16,
    "textComplete": false,
    "truncatedReason": "the feed has only a summary of this post",
    "paywalled": true,
    "paywallNote": "the post page shows a paywall (\"This post is for paying subscribers\"); only the public part is returned",
    "fullTextCharged": false,
    "charged": true
}

paywalled is true when the feed or the page shows a paywall, false when Substack marks the post as free for everyone, and null when unknown. textSource is feed, page (full text read from the post page) or archive (an older Substack post: excerpt only).

Errors (not charged)

{ "type": "source", "input": "https://example.com", "success": false, "errorType": "not_found", "error": "no RSS, Atom or JSON feed found: HTTP 404 (not found) — paste the feed URL if you know it (not charged)", "feedsTried": ["https://example.com/", "https://example.com/feed", "https://example.com/rss"], "charged": false }

errorType is invalid_input (a line that is not a URL or handle, or a private/local address), not_found (no feed found at that address), unreachable (the domain does not exist, the SSL certificate is invalid or expired, the site did not answer, or it redirects to a private address), robots_disallowed (the site's robots.txt disallows the feed, so it is skipped), blocked (the site refuses automated visitors: this Actor identifies itself and does not bypass bot protection), too_large (a feed over 20 MB), network/timeout.

The SUMMARY record (key-value store) has status (SUCCESS, PARTIAL_RESULTS, FAILED, NO_RESULTS, LIMIT_REACHED), totals (postsOutput, fullTextsCharged, paywalledPosts, postsFilteredOut) and perSource (feed URL found, platform, posts in the feed, archive posts, matched, returned, a note such as "no new posts since the last run").

Integrations

• Schedules: run hourly or daily with Only new posts for a new-posts feed across many newsletters.
• API and webhooks: start runs and read the dataset from any language; a webhook tells you when a run finishes.
• Exports: CSV, Excel, JSON, XML and HTML from the dataset.
• Apify integrations: send results to Google Sheets, Slack, Zapier, Make, n8n and others from the Integrations tab.

Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/substack-medium-posts-scraper) to Claude, Cursor or any MCP client, then ask e.g. "What did Lenny's Newsletter and Platformer publish this week? Summarize the free posts and list the paid ones."

{ "sources": ["https://www.lennysnewsletter.com", "https://www.platformer.news"], "sinceDate": "7 days" }

Failed items are not charged and carry an errorType.

Tips and limitations

• Recent posts only for most platforms (Medium 10, Substack 20 plus the archive, most blogs 10–50). Use Only new posts on a schedule to build a complete history going forward.
• Substack profile links (substack.com/@name) are guessed as name.substack.com; paste the newsletter's own URL when it differs.
• Medium member-only stories usually come as a snippet; with full text on you get the public opening and paywalled: true.
• Word counts count words of the text returned; postWordCount is Substack's own count for the whole post (also for paid posts).
• Only new posts remembers post URLs per feed for each monitor name. Use a different monitor name for a different list.
• Requests to one site are spaced by Pause between requests (500 ms by default), and rate-limited answers are retried after the time the site asks for.

Terms and responsible use

• The Actor reads public feeds (published for syndication and feed readers) and, only when you turn on full text, the public post page. It never logs in, never uses cookies or subscriptions, and never bypasses a paywall: paid content is not returned.
• It checks each site's robots.txt before reading a feed, the Substack archive list or a post page, and skips what robots.txt disallows (September 2026: Substack and Medium allow their feeds for all crawlers; Medium's robots.txt blocks some AI-training crawlers, which this Actor is not).
• Posts are the authors' work. Use the data for monitoring, research, analysis and personal reading; link to the original post and credit the author if you show excerpts, and do not republish full paid or copyrighted texts. Medium publishes its content license at medium.com/license.xml.
• Only public author names are returned; the feed owner's e-mail address (present in some feeds) is never output.

FAQ

Does the Actor collect usage data? The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

Can it get paid Substack posts? No. Paid posts return the public preview that the newsletter itself puts in its feed, marked paywalled: true.

Why only 10 posts for a Medium author? Medium's feed carries the latest 10 stories. Schedule the Actor with Only new posts to keep every new story.

Does it work with custom-domain Substacks? Yes (e.g. https://www.lennysnewsletter.com): the feed says it is Substack, and the archive extras are read from the same domain.

What about Beehiiv newsletters? Paste the newsletter's web address; its feed (rss.beehiiv.com) is found from the home page when the newsletter has RSS enabled.

What if I leave the list empty or mistype the date? Nothing is read and nothing is charged: you get one free invalid_input row saying what to fix, and the run ends normally (not as a failed run). A single bad line in a longer list gets its own free row and the other sources still run.

What about private addresses such as localhost or 192.168.x.x? They are never requested: the line gets a free invalid_input row, and a site that redirects to a private address gets a free unreachable row.

What if the same post shows up under two inputs? For example a Medium handle and the publication's own domain, or a blog and a copy of its feed. The post is returned and charged once; the second input's perSource entry says how many duplicates were left out (SUMMARY.postsDuplicate).

Does it handle old feeds in other encodings? Yes: ISO-8859-1, Windows-1252, GBK, Shift_JIS and other encodings declared in the HTTP header or the XML line are decoded correctly. Feeds larger than 20 MB are refused with a free too_large row.

Can I run it on a schedule? Yes. Turn on Only new posts since the last run, set a monitor name, and create a schedule in Apify Console.

Found this useful? A short review on the Store helps other users find it.

Related Actors

More tools by TidyTools that work well with this one:

• Website Markdown Crawler: crawl a whole site or docs portal into clean Markdown for LLMs and RAG
• Website Contact Extractor: find company emails, phones and social profiles on websites
• Email List Cleaner: clean an email list with MX, disposable and typo checks
• Website Change Monitor: get an alert when a page, a price or a text changes