# Substack Scraper (public newsletter posts) (`lowlanddata/substack-scraper`) Actor Extract posts from public Substack newsletters as clean JSON: title, date, author byline, likes, comments, word count and tags. Keyword search, date window and optional full text of free posts. No login, no API key. Pay per result, built for monitoring and AI pipelines. - **URL**: https://apify.com/lowlanddata/substack-scraper.md - **Developed by:** [Lowland Data](https://apify.com/lowlanddata) (community) - **Categories:** Social media, AI, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $2.40 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Substack Scraper (public newsletter posts) Substack hosts tens of thousands of newsletters, and the archive of any of them is a list of posts you can only read page by page. Watching a publication over time, comparing several, or pulling every post that mentions a topic needs those posts as rows, with the likes and comment counts attached. Give it publication names (or links) and every post comes back as one item: title, subtitle, date, the author byline, audience (free or paid), likes, comment count, word count, tags, cover image and a short preview. Switch on `includeBody` and free posts also come with their full text. No login, no API key, no Substack account. It reads the public archive that every Substack publication serves to any visitor. The publication's name and the author names printed on each post stay on the row as its byline; commenters, subscribers, readers, author handles, bios and avatars are never collected. ### Quick start (30 seconds) 1. Put one or more publications into **publications**, e.g. `astralcodexten` or `https://www.lennysnewsletter.com, https://substack.com/@lenny`. 2. Click **Start**. That's the whole minimum setup. 3. When the run finishes, open the dataset's **Overview** tab for a clean table, or **Export** it as CSV/Excel/JSON. Optional knobs: a keyword to search for inside the publications, newest-first or most-popular-first, a date window, free or paid posts only, and the full text of free posts. Any input works on a daily Schedule. ### What you can build with it - **Watch a publication for new posts.** Put it in **publications** and set `postedAfter` to yesterday. The run stops as soon as it reaches older posts, so a daily Schedule only reads what is new. - **Find out what a topic does on Substack.** Put a term in **searchQuery** across a list of publications and compare `reactionCount`, `commentCount` and `restackCount` to see which posts and which publications the topic travels in. - **Build a reading corpus.** Turn on **includeBody** and every free post arrives as plain text next to its date, authors and tags, ready to embed, summarise or search. - **Feed an AI agent clean data.** Every field is structured and predictable, so an assistant or pipeline can read a publication's last 100 posts directly. ### What you get Each post is one dataset item: ```json { "id": 208730073, "title": "Announcing Lennyโ€™s Jobs: The best place in the world to find, vet, and land your dream job", "subtitle": "Where product managers, engineers, designers, and growth/marketing professionals discover high-quality open roles at tech companies", "slug": "announcing-lennys-jobs-the-best-place", "url": "https://www.lennysnewsletter.com/p/announcing-lennys-jobs-the-best-place", "canonicalUrl": "https://www.lennysnewsletter.com/p/announcing-lennys-jobs-the-best-place", "publicationName": "Lenny's Newsletter", "publicationUrl": "https://www.lennysnewsletter.com", "authors": ["Lenny Rachitsky"], "postDate": "2026-08-18T15:40:06.921Z", "audience": "everyone", "type": "newsletter", "section": null, "reactionCount": 389, "commentCount": 18, "restackCount": 10, "wordCount": 933, "coverImage": "https://substackcdn.com/image/fetch/$s_!PFZn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f696879-6374-42d6-a171-47933f6eb126_1456x970.png", "tags": ["Career"], "excerpt": "๐Ÿ‘‹ Hey there, Iโ€™m Lenny. Each week, I share deeply researched product, growth, and career advice. For more: Lennyโ€™s Jobs | Lennyโ€™s Podcast | How I AI | Lennybot | Become an AI-Native Builder and some other favorite AI/PM courses" } ``` With `includeBody` on, free posts carry one more field, `bodyText`, with the full text. Field notes, so you know exactly what you are buying: - `id` is Substack's numeric post id, unique across publications; `slug` is the post's name inside its publication. `url` is the human link on the publication's own address, which is its custom domain when it has one. - `authors` are the names printed in the post's byline, guest authors included. Names only: no handles, profile links, avatars or bios. - `audience` is `everyone` for free posts and `only_paid` for posts behind the paywall. A paid post comes back with all its metadata and the short public preview in `excerpt`, never its text. `bodyText` is null for paid posts whatever `includeBody` says. - `wordCount` is the length of the whole post as Substack reports it, also for paid posts. - `excerpt` is the short preview Substack shows for every post, about 200 characters. For many publications it is a standard intro line that repeats across posts. - `reactionCount`, `commentCount` and `restackCount` are Substack's figures at scrape time. The comments themselves are not collected. - `postDate` is the publication time in UTC. - `coverImage` is the link as Substack serves it at run time; download it if you need to keep it. - `section` is filled only for publications that sort their posts into sections. ### How much does it cost to scrape Substack? $3.00 per 1,000 posts delivered, pay-as-you-go, no subscription, and you pay per result rather than per hour. A run that finds nothing still costs the $0.005 actor start. In plain dollars: - **100 posts โ‰ˆ $0.30**: a daily multi-publication watch. - **500 posts โ‰ˆ $1.50**: a solid archive of a publication. - **A 10,000-post crawl โ‰ˆ $30**: a large set of publications, deduplicated. The price is all-inclusive: your runs' platform usage is covered by it, with no separate compute or proxy charges. Measured on the platform: a 25-post run finishes in about 5 seconds and a 300-post run in about 15 seconds. With `includeBody` on, plan for about two seconds per free post, since each text is a separate request. The actor reads Substack directly and only falls back to the Apify proxy if a request is blocked, so there is nothing to configure. Free-plan runs are limited to a sample of 5 items, enough to evaluate the output format against your real publications. ### Not technical? Let your AI assistant set it up Copy this into ChatGPT, Claude or any AI assistant, fill in the one line, and follow the conversation: ```text Help me set up the "Substack Scraper" actor on Apify (https://apify.com/lowlanddata/substack-scraper). Guide me one step at a time. What I want to watch: [E.G. "new posts in astralcodexten and lenny that mention 'AI'"] Guide me to: 1. Propose my input values: publications (names or links, comma-separated), an optional searchQuery for a keyword inside the publications, postedAfter for a date window, audience (all, free or paid), includeBody if I want the full text of free posts, and maxItems. 2. Check that each publication exists by opening its address in my browser (name.substack.com, or its own domain). If it shows the newsletter's home page, the actor can read it. 3. Create a free Apify account (apify.com), open the actor page, paste the values into the Input form, and start a run. 4. Set up a daily Schedule in the Apify Console with the same input, plus an email or Slack integration so new posts reach me automatically. 5. Show me how to export results as CSV/Excel, or read them from the API if I code. 6. If the results are what I wanted, remind me at the end to leave a quick rating on the actor page, and to report anything broken or missing on its Issues tab. ``` ### Input | Field | Description | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `publications` | Publication names, addresses, custom domains or `substack.com/@author` links, comma- or space-separated, e.g. `astralcodexten, https://www.lennysnewsletter.com`. Required. | | `searchQuery` | Only posts that match this term, searched inside each publication and listed by relevance. Optional. | | `sort` | `new` (default) walks newest-first, `top` lists the most popular first. Ignored while a search term is set. | | `audience` | `all` (default), `free` or `paid`. Paid posts come back as metadata and preview only. | | `postedAfter` | Only posts published on or after this date (YYYY-MM-DD); a newest-first walk stops at older history. | | `postedBefore` | Only posts published on or before this date (YYYY-MM-DD). | | `includeBody` | Add `bodyText`, the full text of each free post. Off by default; needs one extra request per post. | | `maxItems` | Stop after this many posts in total (default 500). | | `proxyConfiguration` | Optional. Direct by default; the Apify proxy joins on its own if blocked. Set one to force a proxy. | Publications are read in the order you list them, and `maxItems` counts across all of them. An `@author` link is followed to the publication that author runs, and a link to a single post reads the publication it belongs to. A name that is not a Substack publication, or no longer exists, is not charged: the rest of the run is delivered, with an honest note in the run status. A name that cannot be recognised at all, such as `localhost`, is refused up front with the offending names in the message. ### Use it from your code Run the actor and get posts straight back with one HTTP call (fine for scoped runs up to ~5 minutes): ```bash curl "https://api.apify.com/v2/acts/lowlanddata~substack-scraper/run-sync-get-dataset-items?token=" \ -X POST -H "Content-Type: application/json" \ -d '{"publications": "astralcodexten", "maxItems": 100}' ``` Node.js: ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: '' }); const run = await client.actor('lowlanddata/substack-scraper').call({ publications: 'astralcodexten, https://www.lennysnewsletter.com', searchQuery: 'AI', maxItems: 200, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); ``` Python: ```python from apify_client import ApifyClient client = ApifyClient("") run = client.actor("lowlanddata/substack-scraper").call( run_input={"publications": "astralcodexten", "maxItems": 100}) items = client.dataset(run["defaultDatasetId"]).list_items().items ``` Schedules, webhooks and the Make/Zapier/n8n integrations all work out of the box: this is a standard Apify actor. ### Use it with AI agents (MCP) Claude, Cursor and other MCP-capable agents can run this scraper as a tool through Apify's hosted MCP server: the agent fills in the publications itself, starts the run and reads the results: no glue code. Claude Code: ```bash claude mcp add apify --transport http "https://mcp.apify.com?actors=lowlanddata/substack-scraper" ``` Cursor or Claude Desktop (add a custom connector / MCP server with this URL): ```text https://mcp.apify.com?actors=lowlanddata/substack-scraper ``` Sign in with your Apify account when prompted. Runs are billed to it. Setup details per client: [Apify MCP docs](https://docs.apify.com/platform/integrations/mcp). Prompts that work once connected: - "Get the last 50 posts from the Substack publication astralcodexten and summarize what it covered this month." - "Search the Substack publications noahpinion and lenny for posts about pricing and list them with their like counts." - "Compare the average likes per post of these three Substack publications over their last 30 posts each." ### What about paid posts? Substack lets authors put posts behind a paywall. The actor never opens that text. A paid post still comes back as a row (title, subtitle, date, authors, likes, comment count, word count, tags, cover image) plus the short public preview Substack shows to everyone, so a publication's row count matches what its archive lists. Set `audience` to `free` for free posts only, or `paid` to see only what sits behind the paywall. With `includeBody` on, only free posts get their text. ### Is it legal to scrape Substack? A Substack publication is a broadcast: its archive and its free posts are open to anyone without logging in, and that is the only surface this scraper reads. It does not log in, subscribe, read comments or notes, or touch subscriber and reader data, and requests are paced at about one a second. The publication's name and the author names printed on each post stay on the row as its byline; everything beyond the post itself (commenters, subscribers, author handles, bios, avatars, the author's other publications) is never read, so none of it can reach your dataset. One honest limit: titles, previews and post texts are the authors' own words, delivered as-is. If an author names a person or publishes contact details in a post, that text is not rewritten. The guarantee covers the data fields, not what authors choose to publish. This is not legal advice. Substack's terms of use contain a clause on automated access, so check them against your own use. Whether your use of a publication's posts is allowed under those terms, copyright and your jurisdiction's rules is yours to assess. ### Is there a Substack API alternative? Substack publishes no official API for reading a publication's posts. This actor is the practical alternative: the same archive as structured JSON through one HTTP call (`run-sync-get-dataset-items`), on a schedule, or as an MCP tool for AI agents, with no account to create. ### Does Substack block scrapers? Substack serves each publication's archive openly and does not disallow its archive and post endpoints in robots.txt. This actor stays inside that welcome: it waits about a second between requests, goes direct, and keeps the load negligible. It does not read the paths robots.txt asks crawlers to leave alone (comments, notes, embeds, subscribe pages) and does not try to get around the paywall. No CAPTCHA fights, no bot-wall cat-and-mouse, which is also why runs are reliable enough for daily schedules. ### How do I monitor a Substack publication? Put the publication in `publications`, add a `searchQuery` if you only care about a topic, set `postedAfter` to yesterday's date, and add a daily Schedule in the Apify Console with an email/Slack integration on the runs. The dataset then contains only the new posts, each with its likes and comments at that moment. The AI-assistant prompt above walks a non-technical user through exactly this setup. ### FAQ **Which publications can it read?** Any public Substack publication, on a `substack.com` address or on its own domain. Name it (`astralcodexten`), link it, or give an author's `substack.com/@handle` link. **Can I get the text of paid posts?** No. Paid posts come back with metadata and the public preview only. The text is collected for free posts, and only when `includeBody` is on. **Why does a name come back as "not a Substack publication"?** The name does not exist on Substack, the publication was deleted, or the address belongs to a site that is not on Substack. Open `name.substack.com` in a browser: if it shows no newsletter there, the actor cannot read it either. It costs nothing and the rest of the run is delivered. **Can I search a publication by keyword?** Yes, set `searchQuery` and only matching posts are returned, most relevant first. A search always lists by relevance, so the `sort` option is ignored and a date window filters the results instead of stopping the walk early. **Can I get only the newest posts?** Yes, set `postedAfter` to a date. On a plain newest-first run each publication's walk stops as soon as it reaches older posts, so a daily watch collects exactly the new posts and nothing else. **How fresh is the data?** Live at run time: every run reads the archive directly. For continuous freshness, schedule the actor. **Can I export to Excel or CSV?** Yes, every dataset exports as CSV, Excel, JSON or XML from the Apify Console or API. **Does it work with Make, Zapier or n8n?** Yes, it is a standard Apify actor; all platform integrations, webhooks and schedules apply. **Do I need a Substack account or API key?** Neither. The public archive needs no login, and that is the only surface this actor reads. **Does it collect commenters, subscribers or author profiles?** No. You get the publication's name and the author names printed on the post as its byline, and the post itself. Comments, commenter names, subscriber and reader data, author handles, bios and avatars are never read. **How do I scrape several publications in one run?** List them comma- or space-separated in `publications`: `"astralcodexten, lenny, noahpinion"`. Links and `@author` links are accepted too. `maxItems` counts across all of them. **What are `reactionCount` and `restackCount`?** Likes and restacks (shares inside Substack) at the moment of the run. Run the actor daily on the same publications and diff the snapshots to see which posts travel. **Can I download the cover images?** The rows carry links to them as Substack served them at run time. Download what you need to keep, since the links are not a permanent archive. **What happens on a large archive?** The actor pages through the archive 50 posts at a time until `maxItems`, your date window or the first post of the publication. Measured on the platform: a publication with a 1,500-post archive came back whole in about two and a half minutes, using under 100 MB of memory. **How much does Substack scraping cost here?** $3.00 per 1,000 posts, metered on delivery. A 100-post daily watch is roughly $0.30; a failed or empty run costs the $0.005 actor start and nothing per result. **Can I try it before spending anything?** Yes, a free-plan run returns up to 5 real posts from your publications, enough to check the fields against your pipeline. **Can an AI agent fetch Substack posts through this?** Yes, connect Apify's hosted MCP server and agents like Claude or Cursor can pick publications, run the actor and read the posts themselves. Setup is in the MCP section above. ### Related scrapers More clean-output data sources from the same maker: - [Bluesky Scraper](https://apify.com/lowlanddata/bluesky-scraper): public posts from the open social network. - [Telegram Channel Scraper](https://apify.com/lowlanddata/telegram-channel-scraper): posts from public Telegram channels. - [Kleinanzeigen.de Scraper](https://apify.com/lowlanddata/kleinanzeigen-scraper): Germany's biggest classifieds site. ### Troubleshooting The actor fails fast with the reason in the run's status message: - **"not a Substack publication, nothing charged: ..."**: check the names, and open `name.substack.com` in a browser to confirm the newsletter exists. The rest of the run is delivered. - **"Not a Substack publication name, link or @handle: ..."**: a name in `publications` is not valid. Use a name like `astralcodexten`, a link like `https://www.lennysnewsletter.com`, or an author link like `https://substack.com/@lenny`. - **"could not be read (blocked or unreachable): ..."**: those publications failed after retries; the rest of the run is delivered normally. Try again later, or set a proxy. - **"N free posts had no body to read"**: the post was listed but its text was gone by the time it was fetched; the row keeps its metadata and `bodyText` is null. - **Fewer items than requested on a free plan**: the 5-item free sample cap; run on a paid Apify plan for full results. ### Support Found an issue or missing a field you need? The fastest route is the actor's **Issues tab** - it is monitored and reports get fixed. For anything private, email **contact@lowlanddata.com**. This actor is actively maintained. Working well for you? A **rating on this page** takes ten seconds and helps other users find a reliable option. It is also the clearest signal of what we should build next. # Actor input Schema ## `publications` (type: `string`): One or more Substack publications, separated by commas or spaces. Use the name ("astralcodexten"), the address ("astralcodexten.substack.com"), a custom domain ("https://www.lennysnewsletter.com") or an author link ("https://substack.com/@lenny"). A link to a single post works too: the whole publication is read, not only that post. ## `searchQuery` (type: `string`): Optional. Only return posts that match this term, using the publication's own archive search. Results come most relevant first, not by date, and the sort option below is ignored while a search is set. Leave empty to read the whole archive. ## `sort` (type: `string`): new walks the archive newest-first. top lists the most popular posts first, which is a quick way to the best of a publication. A search term always lists by relevance. ## `audience` (type: `string`): Keep all posts, only the free ones, or only the paid ones. Paid posts always come back as metadata and the public preview; their text is never read. ## `postedAfter` (type: `string`): Only include posts published on or after this date. A newest-first walk stops as soon as it reaches older posts. Format: YYYY-MM-DD (for example 2026-09-08) or a full ISO timestamp such as 2026-09-08T14:30:00Z. Slash dates like 08/09/2026 are refused on purpose: they are read as September in Europe and August in the United States, and this actor will not guess which you meant. ## `postedBefore` (type: `string`): Only include posts published on or before this date. Format: YYYY-MM-DD (for example 2026-09-08) or a full ISO timestamp such as 2026-09-08T14:30:00Z. Slash dates like 08/09/2026 are refused on purpose: they are read as September in Europe and August in the United States, and this actor will not guess which you meant. ## `includeBody` (type: `boolean`): Adds a bodyText field with the full text of each free post. It needs one extra request per post, so a run takes noticeably longer. Paid posts never get a body, whatever this setting says. ## `maxItems` (type: `integer`): Stop after this many posts in total. Free-plan runs are capped at 5 items. The form starts at 25 for a cheap first run - that is the form's prefill, not a cap on your plan; clear it to use the default of 500. ## `proxyConfiguration` (type: `object`): Optional. By default the actor reads Substack directly, which is fastest, and switches to the Apify proxy by itself if Substack blocks the request. Turn on Apify Proxy, or list your own proxy URLs, to route every request through a proxy. ## `reportFailures` (type: `boolean`): When a run fails, send the developer the actor name and build, an error category, the error message (truncated) and a one-way hash of your input - so a bug can be fixed without anyone asking you to reproduce it. Your input itself, your results and your identity are never sent, and successful runs send nothing at all. Switch this off and the actor behaves exactly the same. ## Actor input object example ```json { "publications": "astralcodexten", "sort": "new", "audience": "all", "includeBody": false, "maxItems": 25, "proxyConfiguration": { "useApifyProxy": false }, "reportFailures": true } ``` # Actor output Schema ## `posts` (type: `string`): Scraped Substack posts, one JSON object per post. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "publications": "astralcodexten", "maxItems": 25, "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("lowlanddata/substack-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "publications": "astralcodexten", "maxItems": 25, "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("lowlanddata/substack-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "publications": "astralcodexten", "maxItems": 25, "proxyConfiguration": { "useApifyProxy": false } }' | apify call lowlanddata/substack-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,lowlanddata/substack-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/MJZBIUFLmRg0cA1ua/builds/jjbuNc6mK9Ob9MsO4/openapi.json