Website to Markdown — LLM & RAG Content Exporter

Crawl any website and get one clean Markdown document per page — ready to chunk into a vector database, feed to a RAG pipeline, fine-tune on, or migrate into a docs system.

Why this exporter

The big content crawlers are powerful — and unpredictable. Their issue tabs are full of the same complaint: "I set a 60-second timeout and the crawler ran 36 minutes." Usage-based pricing plus browser rendering means you find out what a run cost after it's done.

This actor is the opposite trade:

• Hard caps, flat pricing. maxPages is a hard stop. 1 result = 1 page. A 500-page cap can never cost more than 500 results. There is no compute-unit roulette.
• No browser. Pure HTTP + HTML parsing. Runs are fast and cheap; static and server-rendered sites (docs, blogs, marketing sites, knowledge bases — the 90% case for RAG) come out clean.
• Boilerplate stripped. Nav, headers, footers, cookie banners, share buttons, sidebars removed. Main content auto-detected (<main>, <article>, common content containers) before conversion.
• Sitemap-seeded. Reads sitemap.xml so orphan pages the link crawl would miss still get exported.
• Thin pages are free. Pages under minTextLength characters (tag pages, pagination shells) are skipped and never billed.
• Robots.txt respected. Always.

Honest limitation: this actor does not execute JavaScript. Client-side-rendered SPAs will come out empty — use a browser-based crawler for those. That trade-off is exactly why this one is 5-10x cheaper and never hangs.

Input

Field	Type	Default	Notes
startUrls	array	—	Crawl stays on the same host.
maxPages	integer	100	Hard cap on exported pages.
maxDepth	integer	3	Link depth from start page.
useSitemaps	boolean	true	Seed the crawl from sitemap.xml.
includeUrlGlobs	array	[]	e.g. https://docs.example.com/guide/*
excludeUrlGlobs	array	[]	e.g. */tags/*, */login*
minTextLength	integer	200	Skip (and don't bill) pages with less content.
proxyConfiguration	object	Apify Proxy

{
    "startUrls": [{ "url": "https://docs.example.com" }],
    "maxPages": 500,
    "includeUrlGlobs": ["https://docs.example.com/*"],
    "excludeUrlGlobs": ["*/changelog/*"]
}

Output schema

One dataset item per page:

{
    "url": "https://docs.example.com/guide/getting-started",
    "canonicalUrl": "https://docs.example.com/guide/getting-started",
    "title": "Getting started",
    "description": "Install and configure in five minutes.",
    "lang": "en",
    "depth": 1,
    "markdown": "# Getting started\n\nInstall the CLI...",
    "wordCount": 812,
    "fetchedAt": "2026-07-04T10:00:00.000Z"
}

Pricing

Pay per result. 1 result = 1 exported page. Thin/skipped pages cost nothing.

Pages	Cost at $1.00 / 1,000
500 (typical docs site)	$0.50
5,000	$5.00
50,000	$50.00

Your maximum spend is always maxPages — known before you press Start.

Using from API / MCP / AI agents

curl -X POST "https://api.apify.com/v2/acts/<ACTOR_ID>/run-sync-get-dataset-items?token=<TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{ "startUrls": [{ "url": "https://docs.example.com" }], "maxPages": 50 }'

Pairs naturally with the Apify MCP server: agents can pull a site into context as Markdown with a single tool call.

FAQ

Tables? Converted as readable text in v0.1; GFM table output is on the roadmap (see changelog policy below).

Why did I get fewer pages than maxPages? The site is smaller than the cap, pages were thinner than minTextLength, or robots.txt disallowed sections. All three are correct behavior — you were not billed for them.

How do I know a run succeeded? Every run writes a RUN_SUMMARY record to the key-value store: pages exported, thin pages skipped, and any failed URLs with their error. 404s and other HTTP errors are never scraped as content — they're retried and reported, not billed.

Changelog

• 0.1.0 (2026-07) — Initial release: main-content detection, boilerplate stripping, Markdown conversion, sitemap seeding, include/exclude globs, thin-page filter, hard page cap, per-run RUN_SUMMARY with error reporting.

---

Maintained by Sturdy Data — boring, reliable scrapers. Report issues in the Issues tab; we respond within 24 hours.