Substack Recommendations Network Scraper

Extract public Substack newsletter recommendations into a clean, exportable graph.

The Substack Recommendations Network Scraper turns every public recommendation into one normalized source-to-target edge. Start from a .substack.com publication or a Substack-powered custom domain, then optionally follow recommended newsletters with strict depth, publication, and edge limits.

No Substack login or API key is required. The Actor uses lightweight HTTP extraction rather than a browser, supports scheduling, and exports results in JSON, CSV, Excel, XML, or through the Apify API.

What does Substack Recommendations Network Scraper do?

This Actor opens each publication's public /recommendations page and extracts the newsletter relationships visible there.

It can:

• 🕸️ Map directed newsletter-to-newsletter recommendation edges.
• 📰 Capture source and recommended publication names and URLs.
• ✍️ Include target authors, publication descriptions, and public recommendation blurbs.
• 🔁 Follow recommendations breadth-first for a bounded network crawl.
• 🧹 Normalize tracking-heavy URLs and deduplicate repeated directed edges.
• 🗓️ Run on a schedule to monitor network changes over time.

The output is graph-ready: one row equals one edge, so it can be loaded directly into spreadsheets, SQL, NetworkX, Neo4j, Gephi, or a BI tool.

Who is this Substack network scraper for?

Newsletter growth and partnership teams

• Find newsletters already adjacent to your category.
• Build cross-promotion and guest-collaboration prospect lists.
• Compare which publications are repeatedly recommended across a niche.

Sponsorship and media buyers

• Map clusters around newsletters that already reach a target audience.
• Discover adjacent publications for sponsorship outreach.
• Add recommendation context to an existing newsletter media list.

Creator agencies and newsletter operators

• Track changes in client and competitor recommendation networks.
• Identify bridge publications that connect multiple newsletter communities.
• Export targets for a partnership CRM or outreach workflow.

Researchers and data teams

• Build reproducible directed graphs from public relationship data.
• Measure degree, centrality, clusters, and changes between scheduled snapshots.
• Preserve source-page and scrape-time provenance for every edge.

Why use this Actor?

• ✅ Graph-native rows: no nested recommendation arrays to unpack.
• ✅ Controlled recursion: maxDepth, maxPublications, and maxItems prevent runaway crawls.
• ✅ Custom-domain support: use publications such as newsletter.pragmaticengineer.com and www.lennysnewsletter.com.
• ✅ No browser overhead: anonymous HTTP extraction keeps runs fast and inexpensive.
• ✅ Stable edge IDs: deterministic IDs simplify snapshot comparison and database upserts.
• ✅ Traceable data: each row includes its source recommendations page and scrape timestamp.
• ✅ Apify integrations: schedule runs, trigger webhooks, and connect datasets to thousands of tools.

Choose this Actor when recommendation relationships are the product. Use the general Substack Scraper when you need post archives, article bodies, or comments instead.

What Substack recommendation data can you extract?

Category	Fields	Why it matters
Edge identity	edgeId, depth	Deduplicate relationships and analyze traversal levels.
Source node	sourcePublicationName, sourcePublicationUrl	Identify the recommending newsletter.
Target node	targetPublicationName, targetPublicationUrl	Identify the recommended newsletter.
Creator context	targetAuthor, targetDescription	Qualify partnership or sponsorship prospects.
Recommendation context	recommendationBlurb, recommendationAttribution	Preserve the public endorsement when Substack displays one.
Provenance	sourceRecommendationsUrl, scrapedAt	Audit and compare scheduled snapshots.

Optional fields are omitted when Substack does not publish them or when includePublicationMetadata is disabled. The Actor does not fabricate missing values.

How much does it cost to map a Substack recommendation network?

This Actor uses pay-per-event pricing. The current price is a $0.005 run-start fee plus a charge for each unique recommendation edge saved.

Apify plan	Price per edge
Free	$0.000036218
Starter / Bronze	$0.000031494
Scale / Silver	$0.000024565
Business / Gold	$0.000018896
Platinum	$0.000012597
Diamond	$0.000010000

Real source volume varies by publication. Calculate a run as the $0.005 start fee plus saved edges multiplied by the plan's per-edge price. The Apify Free plan's $5 monthly credits can cover roughly 137,900 edges at the Free-tier rate after the start fee.

You are charged only for saved unique edges, subject to the one-time run-start event.

How to scrape Substack recommendations

1. Open Substack Recommendations Network Scraper on Apify.
2. Add one or more publication homepages or /recommendations URLs.
3. Keep maxDepth at 0 for only those publications.
4. Set maxDepth to 1 or more to follow recommended newsletters.
5. Keep maxPublications and maxItems conservative for the first run.
6. Click Start.
7. Preview the Recommendation edges dataset view.
8. Export the dataset or connect it to your downstream workflow.

A fast single-publication input:

{
  "publicationUrls": [
    { "url": "https://newsletter.pragmaticengineer.com" }
  ],
  "maxDepth": 0,
  "maxPublications": 1,
  "maxItems": 25,
  "includePublicationMetadata": true
}

A bounded recursive graph input:

{
  "publicationUrls": [
    { "url": "https://newsletter.pragmaticengineer.com" },
    { "url": "https://www.lennysnewsletter.com" }
  ],
  "maxDepth": 2,
  "maxPublications": 100,
  "maxItems": 1000,
  "includePublicationMetadata": true
}

Input parameters

Parameter	Type	Default	Description
publicationUrls	array	Pragmatic Engineer URL	Publication homepages or /recommendations pages. Required.
maxDepth	integer	0	Recommendation hops to follow, from 0 to 3.
maxPublications	integer	10	Maximum recommendation pages fetched, including seeds.
maxItems	integer	100	Maximum unique directed edges saved.
includePublicationMetadata	boolean	true	Include author, description, blurb, and attribution when available.
proxyConfiguration	object	direct connection	Optional Apify Proxy or custom proxy settings.

Publication URLs must point to a homepage or /recommendations page. Post URLs such as /p/some-article fail closed instead of silently scraping the wrong entity.

How recursive traversal works

Traversal is breadth-first.

• Depth 0 processes only seed publications.
• Depth 1 also processes publications found on seed recommendation pages.
• Depth 2 follows one more hop from those publications.
• Depth 3 is the hard safety ceiling.

Every discovered publication is queued once. Every directed edge is emitted once per run. Traversal stops as soon as any relevant global limit is reached.

A source recommending a target is different from the target recommending the source, so those two directions receive different edgeId values.

Output example

{
  "edgeId": "d0f3f1f0de7e24b88583702a",
  "depth": 0,
  "sourcePublicationName": "The Pragmatic Engineer",
  "sourcePublicationUrl": "https://newsletter.pragmaticengineer.com",
  "targetPublicationName": "Big Technology",
  "targetPublicationUrl": "https://www.bigtechnology.com",
  "targetAuthor": "Alex Kantrowitz",
  "targetDescription": "A newsletter about big tech and society by independent journalist Alex Kantrowitz.",
  "sourceRecommendationsUrl": "https://newsletter.pragmaticengineer.com/recommendations",
  "scrapedAt": "2026-07-21T06:59:21.115Z"
}

When a card includes a written endorsement, the same row can also contain recommendationBlurb and recommendationAttribution.

URL normalization and deduplication

The Actor removes fragments and campaign parameters such as utm_source and utm_campaign from publication identity.

These inputs resolve to the same source:

• https://newsletter.pragmaticengineer.com
• https://newsletter.pragmaticengineer.com/
• https://newsletter.pragmaticengineer.com/recommendations?utm_source=test

The normalized source and target URLs generate a deterministic edgeId. This makes repeated scheduled datasets easier to diff or upsert.

Tips for best results

• 🎯 Start with maxDepth: 0 to inspect a seed publication before expanding.
• 📉 Keep maxPublications low when testing a new network.
• 🧭 Use multiple curated seeds when you want coverage across separate newsletter communities.
• 📝 Leave publication metadata enabled for partnership and sponsorship research.
• ⚡ Disable metadata only when you need a compact source-target edge list.
• 🗓️ Schedule consistent inputs and compare edgeId sets between runs.
• 🔌 Configure a proxy only if direct requests become rate-limited in a larger recurring crawl.
• 🚦Treat missing public recommendation pages as unavailable data, not as evidence that a newsletter recommends nobody privately.

Monitor recommendation network changes

For recurring monitoring:

1. Run the same seed list and limits on a fixed schedule.
2. Store each dataset or merge rows into a database keyed by edgeId.
3. Compare current edge IDs with the previous snapshot.
4. Send new or removed edges to Slack, email, or a review queue.
5. Use scrapedAt and sourceRecommendationsUrl as provenance.

This workflow can reveal newly visible partnerships, changed positioning, or newsletter-network expansion without scraping posts or subscriber-only content.

Integrations and workflow ideas

Substack graph → Google Sheets

Export source, target, author, and blurb columns for a partnership prospecting sheet. Deduplicate future runs by edgeId.

Substack graph → Neo4j or NetworkX

Treat sourcePublicationUrl and targetPublicationUrl as nodes and each dataset row as a directed relationship. Calculate in-degree, out-degree, PageRank, and communities.

Scheduled run → Slack or Discord

Use a webhook or Make scenario to announce new recommendation edges for monitored competitors or clients.

Substack graph → CRM

Create or enrich newsletter organization records, then attach the recommending source as relationship context for outreach.

Dataset → Make or Zapier

Trigger qualification, enrichment, routing, and human review only for new edge IDs.

Use the Apify API with JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/substack-recommendations-network-scraper').call({
    publicationUrls: [{ url: 'https://newsletter.pragmaticengineer.com' }],
    maxDepth: 1,
    maxPublications: 10,
    maxItems: 100,
    includePublicationMetadata: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Use the Apify API with Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("automation-lab/substack-recommendations-network-scraper").call(run_input={
    "publicationUrls": [{"url": "https://newsletter.pragmaticengineer.com"}],
    "maxDepth": 1,
    "maxPublications": 10,
    "maxItems": 100,
    "includePublicationMetadata": True,
})

items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

Use the Apify API with cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~substack-recommendations-network-scraper/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "publicationUrls": [{"url": "https://newsletter.pragmaticengineer.com"}],
    "maxDepth": 0,
    "maxPublications": 1,
    "maxItems": 25,
    "includePublicationMetadata": true
  }'

Fetch results from the run's defaultDatasetId, or use an Apify client to wait for completion and list items.

Use with AI agents via MCP

Substack Recommendations Network Scraper can be used by AI assistants through the Apify Model Context Protocol integration.

Claude Code

$claude mcp add --transport http apify "https://mcp.apify.com?tools=automation-lab/substack-recommendations-network-scraper"

Claude Desktop, Cursor, or VS Code

Add this hosted server to the client's MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/substack-recommendations-network-scraper"
    }
  }
}

Authenticate with your Apify account when prompted.

Example prompts — try asking your MCP-enabled assistant:

• “Use automation-lab/substack-recommendations-network-scraper to map one hop from The Pragmatic Engineer and return the most frequently connected newsletters.”
• “Extract Lenny's Newsletter recommendations with author and blurb metadata, then format partnership prospects as a table.”
• “Build a bounded Substack recommendation graph from these three publication URLs and explain which newsletters bridge the clusters.”

Is it legal to scrape Substack recommendations?

Scraping publicly available web data is generally legal in many jurisdictions, but the outcome depends on how data is collected and used.

This Actor is designed for public recommendation pages. It does not bypass login controls, paywalls, or private publication access. You are responsible for following applicable laws, Substack's terms, contractual duties, privacy rules, and intellectual-property requirements.

Collect only the data you need, avoid harmful high-frequency use, and apply appropriate retention and access controls. Do not use the output for spam, harassment, discrimination, or other unlawful purposes. Seek professional legal advice for regulated or high-risk use cases.

Limitations

• Only publicly displayed outbound recommendations are extracted.
• Private, paid, subscriber-only, and inbound recommendation data are out of scope.
• Some publications have no public recommendations page.
• Authors, descriptions, blurbs, and attribution are optional because publishers may omit them.
• Custom domains that are no longer backed by Substack may fail validation during extraction.
• A depth-limited graph is a bounded sample, not a complete map of all Substack publications.
• Source markup can change; review scheduled outputs and report repeatable parsing issues through the Actor's Issues tab.

Frequently asked questions

Does this scrape Substack posts or comments?

No. It specializes in public recommendation relationships. Use Substack Scraper for posts and comments.

Do I need a Substack account or API key?

No. The Actor reads anonymous public recommendation pages.

How fast is a run?

A single publication usually finishes in seconds. Recursive duration depends on maxPublications, network availability, retries, and any configured proxy.

Why did my recursive run stop before visiting every discovered newsletter?

The Actor stops when it reaches maxDepth, maxPublications, or maxItems. Increase only the limit that matches your intended graph scope.

Why are some author or blurb fields absent?

Substack does not display those fields on every recommendation card. Missing optional fields are omitted rather than filled with guessed values.

Why is my dataset empty?

Confirm the URL is a public Substack publication homepage or /recommendations page. A valid publication may genuinely display no outbound recommendations. If every requested page is unavailable, the run fails with a clear error instead of reporting a false successful extraction.

Can I use custom publication domains?

Yes. Substack-powered custom domains are supported and normalized to their HTTPS origin.

How do I compare snapshots?

Schedule the same input and compare deterministic edgeId values. IDs present only in the new dataset are additions; IDs present only in the old dataset are removals.

How is this different from a general Substack scraper?

General scrapers typically return posts or publication records with nested recommendations. This Actor emits one graph-ready row per directed relationship and supports bounded traversal.

Other Substack scrapers and tools

• Substack Scraper — extract posts, article metadata, publication details, and comments.
• Substack People Scraper — discover public Substack profiles and publications by keyword.
• Substack Leaderboard Scraper — export category rankings and rising publications.
• Substack Comments Scraper — collect public discussion threads from posts.
• Substack Notes Search & Author Monitor — search and monitor public Substack Notes activity.

Use these Actors together to separate discovery, ranking, recommendation relationships, post content, and audience conversation into clean datasets.

Support

If you find a reproducible issue:

1. Open the Actor's Issues tab.
2. Share the input URL and bounded settings used.
3. Include the failed run link when possible.
4. Remove secrets or private data before posting.

For extraction-quality reports, describe which public recommendation card is visible on the source page but missing or malformed in the dataset.