Website to Markdown Crawler

Crawl a whole website, docs site or blog from one start URL and get every page back as clean, LLM-ready Markdown and text, for $0.001 per page. Built for AI engineers, RAG pipelines and AI agents that need a complete, up-to-date copy of a site's content.

Quick start

1. Paste a site or section into Start URL, e.g. your docs home page. To crawl several sites in one run, add them under More start URLs (up to 20).
2. Set Max pages (you pay only for pages returned). Sitemaps and robots.txt are handled automatically; add Include/Exclude URL patterns if you want to narrow the crawl.
3. Click Start and watch pages stream into the Output tab. Download JSON, CSV or Excel, or read them over the API.

Sample output

{
  "ok": true,
  "url": "https://docs.example.com/guides/getting-started",
  "depth": 1,
  "source": "sitemap",
  "title": "Getting started",
  "description": "Install the widget SDK and send your first request.",
  "wordCount": 642,
  "content": "Getting started\n\nInstall the SDK with npm...",
  "markdown": "# Getting started\n\nInstall the SDK with `npm install`...",
  "linksFound": 38,
  "tdmReserved": false,
  "tdmPolicy": null,
  "crawledAt": "2026-10-10T09:00:00.000Z"
}

How much does it cost?

Event	Price
Page crawled	$0.001 per page on the free plan, as low as $0.0007 on higher Apify plans
Actor start (automatic)	$0.00005 per GB of run memory, once per run

• 100 pages: $0.10 ($0.07 on the top tiers), plus the start fee ($0.00005 at 1 GB).
• 1,000 pages: $1.00 ($0.70 on the top tiers), plus the start fee.

Platform usage is included in the price. You pay only for pages returned with content. Skipped, failed, blocked, empty and duplicate pages are free, and Max pages caps your spend. If you set a maximum total charge for the run, the crawler stops as soon as it is reached.

Pricing update on 8 November 2026. The prices above take effect on 8 November 2026 at 21:40 UTC (Apify's 14-day notice). Until then the previous pricing applies: $0.001 for every row in the results, including rows for skipped or failed pages, plus Apify platform usage. The Pricing tab always shows what applies today.

The AIDevs LLM data toolkit

Three tools that turn the web into clean Markdown for LLMs, RAG and agents:

Tool	Use it for	Price
Website to Markdown Crawler (this Actor)	Whole websites, docs sites and blogs, cheaply	$0.001 per page
Web Page to Markdown	Specific URLs, with richer cleaning, full metadata (author, date, image) and an optional AI summary	$0.004 per page
PDF to Markdown Converter	PDF, DOCX and HTML documents, including tables, page ranges and file uploads	$0.004 per document

A typical RAG pipeline crawls the docs with this Actor, then sends the PDFs it skipped (manuals, whitepapers) to PDF to Markdown Converter.

URL patterns

Patterns use glob syntax: * matches within one path segment, ** matches anything (including /), {a,b} matches either option. Patterns that start with / match the URL path, so they work on any domain. The start URL is always crawled; exclusions win over inclusions.

Goal	Pattern
Only the guides section	include https://docs.example.com/guides/**
Only blog posts one level deep	include /blog/*
Two subdomains	include https://{docs,help}.example.com/**
Skip tag and author archives	exclude **/tag/** and **/author/**
Skip paginated lists	exclude **?page=*
Skip login pages	exclude /login**

FAQ

Is it legal? It depends on the site and on what you do with the content; this isn't legal advice. The crawler reads public pages only: it never logs in or sends cookies, respects robots.txt by default (the rules for AIDevsBot, or for all crawlers when there are none), and stops at CAPTCHAs, login walls and block pages instead of getting around them. Each site's own terms of use apply, so check them before you crawl. Pages can contain personal data, for example names on a team page. You are the data controller for what you collect and need a lawful basis (under the GDPR or similar laws) to store or use it, so keep only what you need.

Can I republish the content? Not without permission. Page content stays under its owner's copyright, so you need the owner's permission to republish it.

What does tdmReserved mean? Converting sites to Markdown for AI or RAG use can count as text and data mining (TDM) under EU copyright law, and sites can reserve those rights in a machine-readable way (TDMRep: a tdm-reservation HTTP header or <meta> tag; the tag wins). Pages with tdmReserved: true have reserved those rights, so get the rightholder's permission before using them for AI; tdmPolicy links the site's terms when it names any. The crawler doesn't read /.well-known/tdmrep.json, so false only means the page itself carries no reservation.

How does the crawler identify itself? Requests identify as AIDevsBot; site owners can opt out with User-agent: AIDevsBot / Disallow: / in robots.txt.

Will it leave the site I gave it? Not by default: Same domain only keeps link-following on the start URL's domain (and the domain it redirects to). Sitemap pages are limited to the start URL's section, e.g. a start URL of https://example.com/docs/ only uses sitemap pages under /docs/. Use include patterns to narrow further.

What happens when a site blocks bots? The crawler stops when it sees a CAPTCHA or anti-bot challenge, when the start page is refused, or after three refused pages in a row. It says so in the status message, never tries to get around the block, and charges nothing for blocked pages. Login pages are skipped.

What if robots.txt can't be loaded? A missing robots.txt (HTTP 404 or another 4xx) means no restrictions. A robots.txt that needs a login or is forbidden (HTTP 401 or 403) counts as disallowing the whole site. If robots.txt can't be reached (HTTP 429, a server error, a network error or a timeout), the whole site counts as disallowed for that run, as RFC 9309 says for unreachable files. Pages on a disallowed site are skipped and never charged; if that's the start URL's site, the run stops right away with a status message.

Does it run JavaScript? No. It reads server-rendered HTML, which covers most docs sites and blogs. Pages that build their content in the browser come back as EMPTY_CONTENT rows and are not charged.

Can my AI agent call it? Yes, through the Apify API and integrations.

Wrong charge? Report it in the Issues tab; we reply within 14 days.

What happens to my data? AIDevs processes your input and results only to run the Actor. Nothing is kept or shared beyond your own Apify storages.

Limitations

• No JavaScript rendering; pages are fetched one at a time (polite, not the fastest).
• Up to 20 start URLs and 2,000 pages per run. Sitemap reading is capped at 20 sitemap files and 5x Max pages URLs.
• PDF, Office files, images and other non-HTML files are skipped. Convert documents with PDF to Markdown Converter.
• Crawl-delay in robots.txt is not used; the crawler never sends parallel requests.
• Public pages only: no logins or cookies.
• Requests identify as AIDevsBot, never as a browser, so a site that only lets browsers or known search engines in may block the crawl; the crawler then stops and says so.
• tdmReserved covers the TDMRep signals on each page (HTTP header and <meta> tag), not /.well-known/tdmrep.json or reservations written only in a site's terms.
• Uses Apify's standard datacenter proxies only; residential and custom proxies aren't supported.

Works with AI agents through Apify's MCP server.

More tools from AIDevs

• PDF to Markdown Converter: PDF, DOCX and HTML documents to clean Markdown for LLMs and RAG.
• Web Page to Markdown: any URL to clean Markdown, text and metadata, with an optional AI summary.
• Competitor Website Analyzer: competitive, SEO and sales briefs from any website.
• SaaS Competitor Battlecards: sales battlecards from public pricing and feature pages.
• PricePulse AI: cross-marketplace product and price intelligence.
• Google Maps Lead Engine: local business leads with AI scoring.
• Nordic Jobs Scraper: Swedish and Norwegian job postings from official open-data APIs.
• LinkedIn Jobs Scraper Pro: public LinkedIn job listings, no login or cookies.

---

Input reference

Field	Type	Default	Description
startUrl	string	(required)	The page to start crawling from.
startUrls	array	[]	More start URLs to crawl in the same run (up to 20 in total). Each is crawled in its own section; Max pages is shared by the whole run and a page is charged once even if two sites link to it.
maxPages	integer	25	Maximum number of pages to crawl (1-2,000).
maxDepth	integer	3	Link-hops from the start URL (0 = start page only). Sitemap pages count as one hop.
sameDomainOnly	boolean	true	Only follow links on the start URL's domain.
useSitemap	boolean	true	Also queue in-section pages from robots.txt sitemaps and /sitemap.xml.
includeUrlPatterns	array	[]	Only follow URLs matching one of these globs.
excludeUrlPatterns	array	[]	Never crawl URLs matching these globs.
respectRobotsTxt	boolean	true	Skip pages disallowed by robots.txt (the rules for AIDevsBot, else *). Turn off only for sites you own or have permission to crawl.
includeMarkdown	boolean	true	Also return a Markdown version of each page.
maxCharsPerPage	integer	0	Cap the text/Markdown length per page (0 = no limit).
proxyConfiguration	object	Apify Proxy	Proxy settings. Only Apify's standard (datacenter) proxies are used; residential groups and custom proxy URLs are ignored with a notice.

Example input

{
  "startUrl": "https://docs.example.com/",
  "maxPages": 200,
  "maxDepth": 4,
  "useSitemap": true,
  "includeUrlPatterns": ["https://docs.example.com/guides/**"],
  "excludeUrlPatterns": ["**/changelog/**"],
  "respectRobotsTxt": true
}

Example input: several sites in one run

{
  "startUrl": "https://docs.example.com/",
  "startUrls": [
    "https://help.example.com/",
    "https://blog.example.org/engineering/"
  ],
  "maxPages": 300,
  "maxDepth": 3
}

Each start URL is crawled in its own section, Max pages counts across all of them, and a page reached from two start URLs is returned and charged once. RUN_SUMMARY lists the stats per site.

Output reference

One dataset record per page:

Field	Description
url	The page URL.
depth	Link-hops from the start URL (start = 0, sitemap pages = 1).
source	How the page was found: start, link or sitemap.
title, description	Page title and meta description.
wordCount	Word count of the extracted text.
content	Clean plain text of the main content.
markdown	LLM-ready Markdown (when enabled).
linksFound	Number of links discovered on the page.
tdmReserved	true when the page reserves text-and-data-mining rights (TDMRep header or <meta> tag; see the FAQ).
tdmPolicy	The site's TDM policy URL when the page is reserved and names one, otherwise null.
crawledAt	ISO timestamp.

Pages that could not be used appear as free rows with ok: false, error and errorCode, for example HTTP_404, BLOCKED, LOGIN_REQUIRED or EMPTY_CONTENT. The key-value store record RUN_SUMMARY has the totals: pages crawled, skipped by robots.txt, duplicates, sitemap URLs found and why the crawl stopped.

API and integrations

curl "https://api.apify.com/v2/acts/entranced_gelato~website-to-markdown-crawler/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{ "startUrl": "https://docs.example.com", "maxPages": 50 }'

Several sites in one call:

curl "https://api.apify.com/v2/acts/entranced_gelato~website-to-markdown-crawler/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{ "startUrl": "https://docs.example.com", "startUrls": ["https://help.example.com"], "maxPages": 100 }'

from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("entranced_gelato/website-to-markdown-crawler").call(
    run_input={"startUrl": "https://docs.example.com", "maxPages": 100}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("ok"):
        print(item["url"], "->", len(item["markdown"] or ""), "chars")

LangChain: load crawled pages with ApifyWrapper.call_actor(...) and map markdown (or content) to Document.page_content, with url and title as metadata.

Make, Zapier and n8n: start a crawl from any flow with the Apify integration and route the Markdown into a vector database, a CMS or a knowledge base.

Not affiliated with, endorsed by or sponsored by LangChain, Make, Zapier or n8n.