Substack Leaderboard Scraper avatar

Substack Leaderboard Scraper

Pricing

from $1.99 / 1,000 results

Go to Apify Store
Substack Leaderboard Scraper

Substack Leaderboard Scraper

Substack Leaderboard Scraper Substack Leaderboard scraper extracts publicly available ranking data, newsletter names, authors, categories, subscriber indicators, and leaderboard metrics 🏆📰 Perfect for market research, creator discovery, trend analysis, and competitive intelligence.

Pricing

from $1.99 / 1,000 results

Rating

0.0

(0)

Developer

Scrapers Hub

Scrapers Hub

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

2 days ago

Last modified

Share

Scrapes the full publication object for every newsletter on a Substack category leaderboard (e.g. Technology → paid), in leaderboard rank order.

How it works

  1. Each leaderboard URL is parsed into a category slug and a ranking typehttps://substack.com/leaderboard/technology/paidtechnology, paid.
  2. The slug is resolved to a numeric category id via GET https://substack.com/api/v1/categories (with a bundled fallback map).
  3. Publications are read from the public leaderboard API, paginated 25 at a time: GET https://substack.com/api/v1/category/public/{categoryId}/{type}?page=N&limit=25. This endpoint already returns the complete publication object (plans, theme, appTheme, contributors, logoPalette, subscriber ranking details, …) — no per-publication request is needed.
  4. Every publication is pushed to the dataset, enriched with leaderboardUrl, leaderboardRank, and scrapedAt.

Networking

Built on the requested stack:

  • requests — primary HTTP client.
  • curl_cffi — automatic fallback with a real Chrome TLS fingerprint, used whenever requests returns a block/challenge (401/403/407/429/503 or a challenge page) or raises.
  • regex — leaderboard-URL parsing and embedded-JSON (window._preloads) extraction.
  • parsel — lightweight HTML parsing (page title / validation).

Apify Proxy is used by default; a pool of pre-rotated proxy URLs is cycled across requests.

Input

FieldTypeDescription
leaderboardUrlsstring[]One or more https://substack.com/leaderboard/<category>/<paid|free> URLs. Required.
maxItemsintegerMax publications to scrape in total. 0 = unlimited. Default 50.
proxyConfigurationobjectProxy settings. Default: Apify Proxy.
{
"leaderboardUrls": ["https://substack.com/leaderboard/technology/paid"],
"maxItems": 50,
"proxyConfiguration": { "useApifyProxy": true }
}

Output

One dataset item per publication — the full Substack publication object plus:

  • leaderboardUrl — the source leaderboard URL.
  • leaderboardRank — 1-based rank on that leaderboard.
  • scrapedAt — ISO-8601 timestamp (...Z).

Run locally

pip install -r requirements.txt
apify run # requires the Apify CLI