Markdown Crawler API — Crawl a Whole Website to Markdown

Point it at a website and get one clean Markdown document per page — title, address, depth, headings, text and word count — ready for a vector database, a RAG pipeline or an LLM context window. It follows same-site links breadth first (or reads the sitemap), stays within your page and depth limits, respects robots.txt and its crawl delay, and skips images, PDFs and scripts. Navigation, headers, footers and sidebars are dropped with the Readability algorithm, so you store content, not chrome.

Try it in 1 minute: press Start with the prefilled input (the Apify platform docs, 50 pages, depth 3). Within a minute you get up to 50 records, each with url, depth, title, markdown, text, wordCount and headings. Price: $1.00 per 1,000 pages; pages that fail or are blocked by robots.txt are free.

Use it to:

• 🧠 Build a RAG knowledge base — documentation sites, help centres and blogs turned into Markdown chunks with headings for splitting.
• 🤖 Give an agent a whole site — crawl once, hand the Markdown to the model, answer questions about the product.
• 🔍 Audit content — every page's title, description, language, word count and link count in one table.
• 🔁 Keep a mirror fresh — schedule a daily crawl and re-embed only the pages whose text changed.
• 🗺️ Map a site — depth, parentUrl and linksCount show how pages connect.

How it works

1. Each start URL is fetched over plain HTTP with browser-like headers (30-second timeout, three attempts; a network error is retried through the Apify proxy). robots.txt is read once per site: disallowed pages are skipped and the Crawl-delay is honoured (up to 10 s).
2. Every <a href> on the page that stays on the same website (or matches your Include only patterns and none of your Exclude patterns) is queued, breadth first, until Max pages addresses have been taken or Max depth is reached. Hashes and (by default) query strings are ignored, so a page is fetched once. With Seed from sitemap the site's sitemap pages are queued too.
3. Each page's main content is isolated with Mozilla Readability and converted to Markdown (headings, lists, links, images and tables kept; addresses made absolute). When no main content is detected, the whole body minus navigation and footer is used — a page is never silently empty.
4. E-mail addresses and phone numbers in the text are removed. Every page is one record with a status.

No browser is started, which keeps the crawl fast and cheap. Sites that render only in JavaScript come back with thin pages (readable: false, few words); a browser-based version is not included in this release.

Input

Field	Meaning	Default
startUrls	Where to begin, one per line	(required)
maxPages	Pages fetched per start URL	50
maxDepth	Links away from the start URL	3
includeGlobs / excludeGlobs	Address patterns to follow / never follow (** anything, * anything but /)	none
useSitemap	Also queue the site's sitemap pages	false
respectRobotsTxt	Skip disallowed pages, honour Crawl-delay	true
sameDomainOnly	Stay on the start URL's host (www. ignored)	true
stripQuery	Treat /page?x=1 and /page as one page	true
ignoreExtensions	File types never fetched	images, PDF, archives, media, CSS, JS, …
mainContentOnly	Readability main content vs. whole page	true
includeLinks	Deliver each page's links (up to 500)	false
maxTextLength	Cut text and Markdown after N characters	0 (no limit)
maxConcurrency	Pages fetched in parallel	5
proxyConfiguration	Proxy for retries after a network error	Apify proxy

Output

Field	What it is
url, finalUrl, canonicalUrl	the page as queued, after redirects, and the canonical address it declares
depth, parentUrl	how far from the start URL, and on which page the link was found (sitemap for sitemap pages)
title, description, language	page metadata as the page declares it
markdown, text	the content as Markdown (title as first heading) and as plain text
wordCount, headings, readable	size, heading outline, and whether a main content block was found
linksCount, links	unique links on the page, and the links themselves with includeLinks
input, status, error, httpStatus, scrapedAt	the start URL, OK or a reason code, and when

Statuses: OK (charged) — NOT_FOUND, NOT_HTML, TOO_LARGE, ROBOTS_DISALLOWED, INVALID_INPUT, FAILED (free, with error saying why). Pages that robots.txt disallows are skipped without a record, except a start page, which gets a ROBOTS_DISALLOWED record so you know why the run is empty.

Example

{
  "input": "https://docs.example.com/guide",
  "url": "https://docs.example.com/guide/getting-started",
  "finalUrl": "https://docs.example.com/guide/getting-started",
  "depth": 1,
  "parentUrl": "https://docs.example.com/guide",
  "title": "Getting started",
  "description": "Install the CLI and run your first job.",
  "language": "en",
  "canonicalUrl": "https://docs.example.com/guide/getting-started",
  "markdown": "# Getting started\n\nInstall the CLI with your package manager …\n\n## Your first job\n\n1. Create a project\n2. Run `example start`\n\n| Flag | Meaning |\n| --- | --- |\n| `--watch` | rerun on change |",
  "text": "Install the CLI with your package manager …",
  "wordCount": 412,
  "readable": true,
  "headings": [{ "level": 1, "text": "Getting started" }, { "level": 2, "text": "Your first job" }],
  "linksCount": 38,
  "httpStatus": 200,
  "status": "OK",
  "scrapedAt": "2026-10-19T12:00:00.000Z"
}

Pricing

You pay per page delivered ($1.00 per 1,000). Pages that fail, are not HTML or are blocked by robots.txt are free. Max pages bounds each crawl, and the run's Maximum charge caps the cost: the run stops cleanly when it is reached and tells you so.

FAQ

Is this an official API? No, it's an unofficial API built on public pages.

How do I crawl only one section of a site? Put a pattern in Include only, e.g. https://docs.example.com/guide/**. Only addresses matching it are followed; the start URL is always fetched.

Why are there fewer pages than Max pages? The site has fewer reachable pages within Max depth, links were excluded by your patterns or by robots.txt, or they point to files (PDF, images) that are skipped. linksCount on each record shows what was found.

Why does a page have only a few words? It is built in JavaScript and ships an empty shell; a plain fetch sees nothing to convert. Such pages come back with readable: false. A headless-browser mode is planned.

Does it run JavaScript or log in? No. It reads what a logged-out visitor without JavaScript sees and never bypasses captchas, logins or bot walls.

Like it? A short review on the Store page helps other people find this Actor. Something missing or broken? Tell us on the Issues tab — we read every one.

Related Actors

• Sitemap Extractor & Scraper — every URL, date and image from a sitemap
• Website Tech Stack Detector — CMS, frameworks and analytics of any site
• URL to Markdown API — clean Markdown from any URL for LLMs
• Article Extractor API — title, author, date and full text of any article
• PDF to Markdown API — PDFs to Markdown with headings and tables