# Substack Scraper – Newsletters, Creators, Posts & Leads (`haketa/substack-scraper`) Actor Scrape Substack: discover top newsletters by category with author, subscriber count and contact, or pull any publication's posts with full content, reactions, comments and podcast data. Search by category or publication URL. Export to JSON, CSV or Excel. - **URL**: https://apify.com/haketa/substack-scraper.md - **Developed by:** [Haketa](https://apify.com/haketa) (community) - **Categories:** Lead generation, Social media, News - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.75 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README

Substack Scraper πŸ“°

Platform What Leads No code Export

Discover Substack newsletters and creators β€” and scrape their posts.

Find the **top newsletters in any category** (with author, **subscriber count** and contact), or pull **any publication's posts** with full content, reactions, comments and podcast data. A complete toolkit for **creator lead generation, content research and AI pipelines** β€” no code required. --- ### ⚑ At a glance | | | |---|---| | πŸ”Ž **Discover** | Top newsletters by category β€” Technology, Finance, Crypto… | | πŸ‘€ **Creator leads** | Author, subscriber count, tagline, contact, monetization | | πŸ“„ **Full posts** | Title, full content, reactions, comments, word count | | 🎧 **Podcasts** | Episode audio URL and duration | | πŸ’΅ **Free vs paid** | Audience filter and paywall flag | | πŸ“¦ **Bulk** | Many categories or publications at once | | ⚑ **Fast** | Pure-HTTP β€” no browser | | πŸ’Ύ **Export** | Excel, CSV, JSON or API | --- ### πŸ“Œ Why this Actor? Most Substack scrapers only pull posts. This one does that **and** turns Substack's category leaderboards into a **creator lead list**: - πŸ”Ž **Newsletter discovery** β€” browse any category and get the top publications ranked, each with the **author**, **subscriber count**, tagline, and whether they run paid subscriptions or a podcast. - πŸ‘€ **Creator intelligence** β€” perfect for sponsorships, PR outreach, influencer research and competitive analysis. - πŸ“„ **Full post scraping** β€” title, subtitle, full plain-text content, reactions, comments, word count and cover image. - 🎧 **Podcast support** β€” audio URL and duration for podcast posts. - πŸ’΅ **Free / paid aware** β€” filter by audience and see which posts are paywalled. --- ### ✨ What you get (data fields) #### πŸ‘€ Publication (creator lead) | Field | Description | |-------|-------------| | `name` | Newsletter name | | `author` | Creator / author name | | `subscribers` | Subscriber count (e.g. `773,000`) | | `tagline` | Publication tagline | | `contactEmail` | Contact email, when published | | `category` | Discovery category | | `categoryRank` | Rank within the category | | `isMonetizing` | Has paid subscriptions | | `hasPodcast` | Runs a podcast | | `authorBestsellerTier` | Substack bestseller tier | | `url` / `subdomain` | Publication links | #### πŸ“„ Post | Field | Description | |-------|-------------| | `title` / `subtitle` | Post title & subtitle | | `content` | Full post text (plain) | | `contentHtml` | Raw HTML *(optional)* | | `author` | Post byline | | `postDate` | Publish date | | `audience` | `free` / `paid` | | `reactionCount` | Likes / reactions | | `commentCount` | Number of comments | | `wordcount` | Word count | | `podcastUrl` / `podcastDurationSeconds` | Podcast media | | `publication` | Source newsletter | | `url` | Post URL | --- ### πŸš€ Quick start **Find newsletters in a category:** 1. Under **Discover newsletters by category**, pick e.g. *Finance*. 2. Click **Start** β†’ get a ranked list of top finance newsletters with authors and subscriber counts. **Scrape a publication's posts:** 1. Under **Publication URLs**, add e.g. `astralcodexten` or `https://www.honest-broker.com`. 2. Click **Start** β†’ get their posts with full content. > πŸ’‘ **Tip:** Leave everything blank to get a sample of top Technology newsletters. --- ### 🎯 Use cases | Use case | What it delivers | |----------|------------------| | 🧲 **Creator lead generation** | Ranked newsletters by category with author, subscribers and contact β€” for sponsorships and outreach. | | πŸ“Š **Market & competitor research** | See who leads a niche, how big they are, and what they publish. | | πŸŽ™οΈ **PR & influencer outreach** | Build targeted lists of writers and podcasters to pitch. | | πŸ€– **AI / RAG pipelines** | Feed full post text into LLMs, embeddings or a vector DB. | | πŸ“° **Content monitoring** | Track a publication's posts, reactions and comments over time. | | πŸ’΅ **Monetization analysis** | See which creators run paid subscriptions and podcasts. | --- ### βš™οΈ Input reference | Input | Type | Default | Description | |-------|------|---------|-------------| | **Discover newsletters by category** | list | β€” | Categories to find top newsletters in. | | **Publication URLs** | list | β€” | Subdomains, URLs or custom domains to scrape posts from. | | **Post URLs** | list | β€” | Individual post URLs. | | **Scrape posts from publications** | boolean | `true` | Off = publication profile only (lead). | | **Include full post content** | boolean | `true` | Fetch each post's full text. | | **Post audience** | select | All | All / Free only / Paid only. | | **Max posts per publication** | number | `20` | `0` = all. | | **Max publications (discovery)** | number | `50` | Cap for category discovery. | | **Only posts since (date)** | string | β€” | e.g. `2026-01-01`. | | **Proxy / concurrency** | β€” | Datacenter / 5 | Sensible defaults. | #### πŸ“₯ Example β€” discover finance creators ```json { "discoverCategories": ["153"], "maxPublications": 50, "scrapePosts": false } ```` #### πŸ“₯ Example β€” scrape a publication's posts ```json { "publications": ["astralcodexten", "https://www.honest-broker.com"], "maxPostsPerPublication": 30, "includePostContent": true, "audience": "free" } ``` *** ### πŸ“€ Example output **Creator lead:** ```json { "recordType": "publication", "name": "The Honest Broker", "author": "Ted Gioia", "subscribers": "296,000", "tagline": "Essays on music, books, media & culture", "isMonetizing": true, "hasPodcast": false, "category": "Culture", "categoryRank": 3, "url": "https://www.honest-broker.com" } ``` **Post:** ```json { "recordType": "post", "publication": "Astral Codex Ten", "title": "The Hugging Face Incident", "postDate": "2026-07-24T...", "audience": "free", "reactionCount": 416, "commentCount": 382, "wordcount": 1325, "content": "…full post text…", "url": "https://www.astralcodexten.com/p/the-hugging-face-incident" } ``` *** ### πŸ’‘ Tips & best practices - βœ… **Discover first, scrape later** β€” find the newsletters in a niche, then feed the best ones back in to scrape their posts. - βœ… **Turn off content** for a fast metadata-only list; turn it on for full text. - βœ… **Filter by audience** to focus on free (public) posts. - βœ… **Sort by subscribers** to find the biggest creators in a category. - βœ… **Use `since date`** for incremental content runs. *** ### ❓ FAQ **Can I find newsletters by topic?** Yes β€” pick one or more categories and get the top publications ranked, with author and subscriber count. **Can I get subscriber counts?** Yes, when the publication exposes them (shown as e.g. `296,000`). **Can I get full post content?** Yes for public (free) posts. Paid posts return the public preview only. **Do I need to code?** No. Pick categories or paste publication URLs, click Start, export. **In what format is the data?** Structured JSON by default, exportable to CSV, Excel, HTML or via API. *** ### πŸ“Š Output & integrations Results are stored in a standard Apify dataset β€” preview as a **table**, export to **JSON, CSV, Excel, HTML**, pull via the **API**, or push to **Google Sheets, Make, Zapier** and more. *** ### βš–οΈ Legal & responsible use This Actor reads **publicly available** Substack information for legitimate research, discovery and monitoring. - Post content remains the property of its authors β€” respect copyright and terms of service. - Any personal data (e.g. contact emails) requires a lawful basis for processing and outreach β€” ensure you have one and honour opt-out requests. - You are responsible for how you use the collected data. *** ### πŸ›Ÿ Support Found a bug, need an extra field, or want a tweak? **Open an issue** on the Actor's Issues tab.

Happy discovering! πŸ“°

# Actor input Schema ## `discoverCategories` (type: `array`): Pick one or more categories to find the top newsletters in them β€” each result includes the author, subscriber count and contact. Great for creator lead generation and research. ## `publications` (type: `array`): Substack publications to scrape β€” a subdomain (e.g. 'astralcodexten'), a full URL (e.g. 'https://newsletter.pragmaticengineer.com'), or a custom domain. Their posts are collected. ## `postUrls` (type: `array`): Optional. Individual Substack post URLs (…/p/…) to fetch directly. ## `scrapePosts` (type: `boolean`): When on, each publication's posts are collected. Turn off to get only the publication's profile (name, author, subscriber count, contact) as a lead. ## `includePostContent` (type: `boolean`): When on, each post's full text is fetched. Turn off for a faster, lighter list of post metadata only. ## `includeHtml` (type: `boolean`): Also include the original HTML of each post's content. ## `audience` (type: `string`): Filter posts by audience. ## `maxPostsPerPublication` (type: `integer`): Maximum number of posts to collect from each publication. Set 0 for all. ## `maxPublications` (type: `integer`): When discovering by category, the maximum number of publications to return in total. ## `sinceDate` (type: `string`): Optional. Only collect posts published on or after this date, e.g. 2026-01-01. ## `proxyConfiguration` (type: `object`): Proxy settings. Datacenter proxies (the default) are more than enough. ## `maxConcurrency` (type: `integer`): How many publications/posts to process in parallel. Recommended: 3–8. ## Actor input object example ```json { "discoverCategories": [], "publications": [ "astralcodexten", "https://www.honest-broker.com" ], "postUrls": [], "scrapePosts": true, "includePostContent": true, "includeHtml": false, "audience": "all", "maxPostsPerPublication": 20, "maxPublications": 50, "proxyConfiguration": { "useApifyProxy": true }, "maxConcurrency": 5 } ``` # Actor output Schema ## `datasetItems` (type: `string`): Download everything collected in this run (JSON, CSV, Excel). ## `runUrl` (type: `string`): Open this run in the Apify Console. ## `recordType` (type: `string`): publication or post ## `name` (type: `string`): Newsletter name (publication records) ## `title` (type: `string`): Post title (post records) ## `author` (type: `string`): Author / creator ## `subscribers` (type: `string`): Subscriber count (publication records) ## `contactEmail` (type: `string`): Publication contact email, when public ## `category` (type: `string`): Discovery category ## `tagline` (type: `string`): Publication tagline ## `publication` (type: `string`): Source newsletter (post records) ## `postDate` (type: `string`): Post publish date ## `audience` (type: `string`): free / paid ## `reactionCount` (type: `string`): Number of reactions/likes ## `commentCount` (type: `string`): Number of comments ## `wordcount` (type: `string`): Post word count ## `content` (type: `string`): Full post content (plain text) ## `subscribersLabel` (type: `string`): Subscriber count label ## `isMonetizing` (type: `string`): Whether the publication has paid subscriptions ## `url` (type: `string`): Publication or post URL ## `scrapedAt` (type: `string`): ISO 8601 timestamp # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "discoverCategories": [], "publications": [], "postUrls": [], "proxyConfiguration": { "useApifyProxy": true } }; // Run the Actor and wait for it to finish const run = await client.actor("haketa/substack-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "discoverCategories": [], "publications": [], "postUrls": [], "proxyConfiguration": { "useApifyProxy": True }, } # Run the Actor and wait for it to finish run = client.actor("haketa/substack-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "discoverCategories": [], "publications": [], "postUrls": [], "proxyConfiguration": { "useApifyProxy": true } }' | apify call haketa/substack-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=haketa/substack-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Substack Scraper – Newsletters, Creators, Posts & Leads", "description": "Scrape Substack: discover top newsletters by category with author, subscriber count and contact, or pull any publication's posts with full content, reactions, comments and podcast data. Search by category or publication URL. Export to JSON, CSV or Excel.", "version": "1.0", "x-build-id": "B0AZkyKWUYfnWml0A" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/haketa~substack-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-haketa-substack-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/haketa~substack-scraper/runs": { "post": { "operationId": "runs-sync-haketa-substack-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/haketa~substack-scraper/run-sync": { "post": { "operationId": "run-sync-haketa-substack-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "discoverCategories": { "title": "Discover newsletters by category", "type": "array", "description": "Pick one or more categories to find the top newsletters in them β€” each result includes the author, subscriber count and contact. Great for creator lead generation and research.", "items": { "type": "string", "enum": [ "4", "62", "153", "118", "134", "103", "96", "76739", "355", "34", "11", "109", "18", "114", "223", "339", "94", "13645", "15414", "49692" ], "enumTitles": [ "Technology", "Business", "Finance", "Crypto", "Science", "News", "Culture", "U.S. Politics", "Health & Wellness", "Education", "Music", "Travel", "History", "Philosophy", "Faith & Spirituality", "Literature", "Sports", "Food & Drink", "Climate & Environment", "Humor" ] } }, "publications": { "title": "Publication URLs", "type": "array", "description": "Substack publications to scrape β€” a subdomain (e.g. 'astralcodexten'), a full URL (e.g. 'https://newsletter.pragmaticengineer.com'), or a custom domain. Their posts are collected.", "items": { "type": "string" } }, "postUrls": { "title": "Post URLs", "type": "array", "description": "Optional. Individual Substack post URLs (…/p/…) to fetch directly.", "items": { "type": "string" } }, "scrapePosts": { "title": "Scrape posts from publications", "type": "boolean", "description": "When on, each publication's posts are collected. Turn off to get only the publication's profile (name, author, subscriber count, contact) as a lead.", "default": true }, "includePostContent": { "title": "Include full post content", "type": "boolean", "description": "When on, each post's full text is fetched. Turn off for a faster, lighter list of post metadata only.", "default": true }, "includeHtml": { "title": "Include raw HTML content", "type": "boolean", "description": "Also include the original HTML of each post's content.", "default": false }, "audience": { "title": "Post audience", "enum": [ "all", "free", "paid" ], "type": "string", "description": "Filter posts by audience.", "default": "all" }, "maxPostsPerPublication": { "title": "Max posts per publication", "minimum": 0, "type": "integer", "description": "Maximum number of posts to collect from each publication. Set 0 for all.", "default": 20 }, "maxPublications": { "title": "Max publications (discovery)", "minimum": 0, "type": "integer", "description": "When discovering by category, the maximum number of publications to return in total.", "default": 50 }, "sinceDate": { "title": "Only posts since (date)", "type": "string", "description": "Optional. Only collect posts published on or after this date, e.g. 2026-01-01." }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Proxy settings. Datacenter proxies (the default) are more than enough.", "default": { "useApifyProxy": true } }, "maxConcurrency": { "title": "Max concurrent requests", "minimum": 1, "maximum": 12, "type": "integer", "description": "How many publications/posts to process in parallel. Recommended: 3–8.", "default": 5 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```