Substack Leaderboard Scraper

Scrapes the full publication object for every newsletter on a Substack category leaderboard (e.g. Technology → paid), in leaderboard rank order.

How it works

1. Each leaderboard URL is parsed into a category slug and a ranking type — https://substack.com/leaderboard/technology/paid → technology, paid.
2. The slug is resolved to a numeric category id via GET https://substack.com/api/v1/categories (with a bundled fallback map).
3. Publications are read from the public leaderboard API, paginated 25 at a time: GET https://substack.com/api/v1/category/public/{categoryId}/{type}?page=N&limit=25. This endpoint already returns the complete publication object (plans, theme, appTheme, contributors, logoPalette, subscriber ranking details, …) — no per-publication request is needed.
4. Every publication is pushed to the dataset, enriched with leaderboardUrl, leaderboardRank, and scrapedAt.

Networking

Built on the requested stack:

• requests — primary HTTP client.
• curl_cffi — automatic fallback with a real Chrome TLS fingerprint, used whenever requests returns a block/challenge (401/403/407/429/503 or a challenge page) or raises.
• regex — leaderboard-URL parsing and embedded-JSON (window._preloads) extraction.
• parsel — lightweight HTML parsing (page title / validation).

Apify Proxy is used by default; a pool of pre-rotated proxy URLs is cycled across requests.

Input

Field	Type	Description
leaderboardUrls	string[]	One or more https://substack.com/leaderboard/<category>/<paid|free> URLs. Required.
maxItems	integer	Max publications to scrape in total. 0 = unlimited. Default 50.
proxyConfiguration	object	Proxy settings. Default: Apify Proxy.

{
  "leaderboardUrls": ["https://substack.com/leaderboard/technology/paid"],
  "maxItems": 50,
  "proxyConfiguration": { "useApifyProxy": true }
}

Output

One dataset item per publication — the full Substack publication object plus:

• leaderboardUrl — the source leaderboard URL.
• leaderboardRank — 1-based rank on that leaderboard.
• scrapedAt — ISO-8601 timestamp (...Z).

Run locally

pip install -r requirements.txt
apify run       # requires the Apify CLI

📬 Contact & Support

Have a question, need a custom feature, or want to request data removal? Email us at scraperhubapi@gmail.com — we're happy to help.