Substack Scraper: Posts, Comments & Authors
Pricing
from $3.00 / 1,000 posts
Scrape any Substack publication: post archives, article text, comments, author profiles and subscriber signals. Search across newsletters and export structured data for research, monitoring and AI datasets. No browser. Output to CSV, JSON or Excel.
You can access the Substack Scraper: Posts, Comments & Authors programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.
{ "openapi": "3.0.1", "info": { "version": "1.2", "x-build-id": "oZuta5b55paEoXFxH" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/doggo~substack-scraper-posts-comments-authors/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-doggo-substack-scraper-posts-comments-authors", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/doggo~substack-scraper-posts-comments-authors/runs": { "post": { "operationId": "runs-sync-doggo-substack-scraper-posts-comments-authors", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/doggo~substack-scraper-posts-comments-authors/run-sync": { "post": { "operationId": "run-sync-doggo-substack-scraper-posts-comments-authors", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "Substack URLs to scrape: publication homepages (subdomain or custom domain), individual post URLs, author profiles (https://substack.com/@handle), and open.substack.com share links. Leave empty only if you enable Discovery mode below.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "mode": { "title": "Scrape mode", "enum": [ "posts", "publication", "author" ], "type": "string", "description": "What to scrape for each publication URL. 'posts' = publication metadata + all posts in the archive. 'publication' = publication metadata only. Post and author URLs are always scraped as posts/authors regardless of this setting ('author' is accepted as an alias of 'publication').", "default": "posts" }, "maxPostsPerPublication": { "title": "Max posts per publication", "minimum": 0, "type": "integer", "description": "Limit the number of posts scraped per publication (lower = cheaper runs). Set 0 to scrape the ENTIRE archive — large publications can have thousands of posts. Leaving the field empty uses the default of 50.", "default": 50 }, "includeContent": { "title": "Include full post content", "type": "boolean", "description": "Fetch each post's full HTML + plain text body. Adds one request per post but gives you the actual article content.", "default": true }, "includeComments": { "title": "Include comments", "type": "boolean", "description": "Fetch the full comment thread for each post (adds one request per post). Note: every comment becomes a separate result in the dataset, and popular posts can have hundreds of comments.", "default": false }, "onlyFreePosts": { "title": "Only free posts", "type": "boolean", "description": "Skip paid/subscriber-only posts in publication archives (their bodies can't be fully scraped anyway). Post URLs given directly in Start URLs are always scraped.", "default": false }, "searchQuery": { "title": "Search query (optional)", "type": "string", "description": "Filter each publication archive by a search term. Only posts matching the term are scraped." }, "discoveryMode": { "title": "Discovery mode", "enum": [ "none", "leaderboard", "search" ], "type": "string", "description": "Auto-discover publications without providing URLs — seeds from 5 curated top publications and expands through each publication's /recommendations page until the limit below is hit. 'Search' additionally filters every discovered publication's archive by the Discovery query. Off by default: a discovery run scrapes MANY publications and produces a correspondingly large (and more expensive) dataset.", "default": "none" }, "discoveryQuery": { "title": "Discovery query", "type": "string", "description": "Used with 'Search' Discovery mode: keyword used to filter the archive of each discovered publication. Ignored otherwise." }, "maxPublicationsToDiscover": { "title": "Max publications to discover", "minimum": 1, "maximum": 500, "type": "integer", "description": "Cap on how many publications Discovery mode will feed into the scraper (the 5 seed publications count toward it; Start URLs do not). Lower = cheaper runs.", "default": 25 }, "maxConcurrency": { "title": "Max concurrency", "minimum": 1, "maximum": 25, "type": "integer", "description": "Parallel requests. Higher = faster runs; proxy rotation is handled automatically, but very aggressive settings can hit Substack rate limits.", "default": 5 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } }}OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.
OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.
By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.
You can download the OpenAPI definitions for Substack Scraper: Posts, Comments & Authors from the options below:
If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.
You can also check out our other API clients: