Website to Markdown Crawler – Clean RAG Content

Convert public web pages into clean Markdown for RAG, LLM applications, semantic search, audits, and content analysis. Each successful page includes readable content plus SEO metadata, headings, canonical URL, links, JSON-LD, crawl depth, and timestamps.

Start in under a minute

{
  "startUrls": [{ "url": "https://example.com" }],
  "maxPages": 10,
  "maxDepth": 1,
  "sameDomainOnly": true,
  "useSitemaps": true,
  "respectRobotsTxt": true
}

Start with one page for a quick test, then raise maxPages and maxDepth. Use include/exclude patterns and custom CSS removal selectors to control what enters the dataset.

Example result shape

{
  "url": "https://example.com/",
  "finalUrl": "https://example.com/",
  "statusCode": 200,
  "title": "Example Domain",
  "description": null,
  "canonicalUrl": null,
  "language": "en",
  "headings": ["Example Domain"],
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
  "textLength": 125,
  "links": ["https://www.iana.org/domains/example"],
  "jsonLd": [],
  "depth": 0,
  "scrapedAt": "2026-08-09T10:37:49.000Z"
}

Why use this Actor

• Readable Markdown instead of raw HTML.
• Main-content preference with scripts, forms, navigation, footers, and common noise removed.
• Sitemap discovery and robots.txt support.
• Same-domain control, crawl depth, and hard page caps.
• Include/exclude URL fragments and custom removal selectors.
• Built-in SSRF protection for localhost, private IPs, metadata hosts, and credentialed URLs.
• One paid result only for each successful HTML page; skipped and failed pages are summarized in OUTPUT.

Best for

• Preparing small and medium websites for RAG or semantic search.
• Creating Markdown corpora for LLM analysis.
• Extracting documentation, blogs, help centers, and public knowledge bases.
• SEO and content inventory workflows.

Pricing and limitations

Current launch pricing starts at $1 per 1,000 successful Markdown page results; the Pricing tab shows every billable event and the exact estimate.

This is an HTTP crawler. It does not execute client-side JavaScript or bypass login, paywalls, CAPTCHAs, robots rules, or anti-bot controls. Dynamic single-page applications may return incomplete content. Use only where you have a lawful basis and respect site terms, copyright, privacy, and crawl-rate expectations.