Website to Markdown Crawler | LLM & RAG | $0.5 / 1K

Convert any website to clean Markdown, plain text and metadata, one row per page, ready for an LLM prompt, a RAG pipeline or a vector database. Point it at a docs site, a blog, a help center or a news section. It follows links, reads sitemap.xml and llms.txt if you ask it to, strips navigation, footers, cookie banners and ads, and keeps headings, lists, tables, links and fenced code blocks intact.

There is no browser behind it, so it is fast and cheap: $0.50 per 1,000 pages plus a $0.0015 start fee, billed only for pages that came back with real content. Pages that need JavaScript to show anything, pages blocked by a bot wall and empty pages are flagged and never billed.

What can you use this website crawler for?

• Website to Markdown for LLMs. Convert any site into Markdown you can paste into ChatGPT, Claude or Gemini, or save as a context file.
• Crawl a website for RAG. Get one clean document per page, with url, title, language and headings as metadata for chunking and embedding.
• Build a vector database knowledge base from a website. Feed the Markdown into Pinecone, Qdrant, Weaviate, pgvector or any other store.
• Docs to Markdown. Export a documentation site, API reference or help center into Markdown files for an AI assistant or a chatbot.
• Scrape a website for LLM training and fine-tuning data. Collect article text from blogs and news sections with author, publish date and language when the page exposes them.
• Crawl from llms.txt or sitemap.xml. Let the site tell you which pages matter instead of guessing with link depth.
• Monitor website content changes. Turn on content hashes and compare runs to see which pages changed.
• Website to Markdown API. Call it from your own code through the Apify API and get JSON rows back.

What data do you get?

One dataset row per crawled page:

Field	Description
url, loadedUrl, canonical	Requested URL, final URL after redirects, <link rel="canonical">
title, description, language	From <title>, meta description / Open Graph, <html lang>
markdown	The main content as GitHub-flavoured Markdown (headings, lists, links, tables, code blocks, images)
text	The same content as plain text
headings	Array of { level, text } for building chunks by section
wordCount	Words in the extracted content
depth, httpStatus, crawledAt	Crawl depth from the start URL, HTTP status, ISO timestamp
contentStatus, needsJavaScript, charged	ok, needs-javascript, blocked, thin, http-error; whether the page was billed
author, publishedTime, ogImage	When the page exposes them
contentHash	Optional SHA-256 prefix of the text, for change detection between crawls
links	Optional list of { url, text } for every link on the page
extractionMethod	How the main content was found (main, article, readability, body-fallback)

Input

Field	Default	What it does
startUrls	-	Where to start. Always crawled, even if they do not match the glob patterns.
maxPages	100	Cost cap. The crawl stops after this many billable pages.
maxDepth	3	Link hops from the start URLs. 0 = only the start URLs. Sitemap and llms.txt pages ignore depth.
crawlScope	same-hostname	same-hostname, same-domain (incl. subdomains) or start-url-path.
includeUrlGlobs / excludeUrlGlobs	empty	Glob patterns on the full URL, e.g. https://example.com/docs/**.
useSitemaps	false	Reads robots.txt Sitemap: lines and /sitemap.xml, including sitemap indexes and .gz files.
useLlmsTxt	false	Reads /llms.txt and queues the pages it links to.
mainContentOnly	true	Strip navigation, footers, banners and sidebars. Off = whole body.
includeImages	true	Keep  in the Markdown. Off saves tokens.
includeLinks	false	Add a links array to each row.
includeContentHash	false	Add contentHash for change detection.
includeSkippedPages	false	Also save free rows for skipped pages (JavaScript-only, blocked, thin, errors). Start URLs are always saved so you can see why they failed.
respectRobotsTxt	true	Skip URLs that robots.txt disallows for all crawlers.
maxConcurrency	5	Parallel requests (1-20). Backs off automatically on HTTP 429.
minWords	20	Pages with fewer extracted words count as empty.
removeSelectors	empty	Extra CSS selectors to delete before extraction.
proxyConfiguration	none	Optional. Without it, a 403/429 gets one retry through a free Apify datacenter proxy session.

Example input:

{
  "startUrls": [{ "url": "https://docs.python.org/3/tutorial/index.html" }],
  "maxPages": 25,
  "maxDepth": 2,
  "crawlScope": "start-url-path",
  "includeImages": false,
  "mainContentOnly": true
}

Output example

{
  "url": "https://react.dev/learn",
  "loadedUrl": "https://react.dev/learn",
  "depth": 0,
  "httpStatus": 200,
  "crawledAt": "2026-10-14T04:54:34.683Z",
  "canonical": "https://react.dev/learn",
  "title": "Quick Start – React",
  "description": "The library for web and native user interfaces",
  "language": "en",
  "markdown": "# Quick Start\n\nWelcome to the React documentation! This page will give you an introduction to 80% of the React concepts that you will use on a daily basis.\n\n### You will learn\n\n-   How to create and nest components\n-   How to add markup and styles\n...",
  "text": "Quick Start\n\nWelcome to the React documentation! ...",
  "headings": [{ "level": 1, "text": "Quick Start" }, { "level": 2, "text": "Creating and nesting components" }],
  "wordCount": 2195,
  "contentStatus": "ok",
  "needsJavaScript": false,
  "extractionMethod": "main>article",
  "charged": true
}

How to use the website crawler: website to Markdown in 3 steps

1. Add one or more Start URLs (for example https://docs.example.com/).
2. Pick a Crawl scope (same hostname, same domain, or only below the start URL's folder), set Max pages and Max depth. Optionally add include / exclude URL glob patterns and tick sitemap.xml or llms.txt discovery.
3. Run it and download the dataset as JSON, CSV or Excel, or read it through the API. Use the markdown field as your chunking input.

Pricing

Pay per event. No subscription, no compute bill.

Event	Price
Actor start (once per run)	$0.0015
Page crawled with real content	$0.0005 ($0.50 per 1,000 pages)

Example costs: 100 pages = $0.0515. 1,000 pages = $0.5015. 10,000 pages = $5.0015. Pages flagged as JavaScript-only, blocked, thin or failed are free and do not count against maxPages. Set maxPages (and Apify's own max-cost-per-run setting) to cap spend.

Integrations and scheduling

• Vector databases and RAG frameworks: the dataset is plain JSON, so it loads into LangChain, LlamaIndex or your own ingestion script. Apify also offers vector database integrations, for example Pinecone.
• Make, n8n, Zapier, Google Sheets: connect the Actor through Apify's integrations and push finished runs wherever you need them.
• Webhooks and API: trigger runs from code and get a webhook when a run finishes.
• Apify Schedules: re-crawl a docs site or blog daily or weekly so your knowledge base stays current.
• Incremental runs for changed pages: turn on includeContentHash, keep the previous run's hashes, and re-embed only the pages whose hash changed. The comparison is done by your workflow, the Actor provides the hash.
• AI agents: call it through the Apify MCP server, the input and output fields are documented in the schemas.

Known limits (please read)

• No JavaScript rendering. This Actor reads the HTML the server sends. Single-page apps that render everything in the browser come back as needs-javascript (and are not billed) instead of empty or junk Markdown. For those sites you need a browser-based crawler.
• Bot walls. Sites behind an active Cloudflare, Akamai or DataDome challenge may answer 403 or a "Just a moment" page. These are flagged blocked and not billed. A 403/429 is retried once through a free datacenter proxy session; some sites still refuse it.
• Main-content detection is heuristic. It looks for <main>, <article> and common content containers, then falls back to a readability algorithm. On index and listing pages (home pages, category pages) the "content" is mostly a list of links and teasers, and some template text can remain. Review a sample before indexing a whole site.
• No login, cookies or private pages. PDFs, images and other non-HTML files are skipped.
• Sitemap discovery reads up to 25 sitemap files per run.

FAQ

How do I convert a website to Markdown for an LLM with this website crawler?

Add the site's URL as a start URL, set Max pages, and run. Each page comes back with a markdown field you can paste into a prompt or save to a file. For one section of a site, use the crawl scope start-url-path. For a whole site, turn on useSitemaps and useLlmsTxt so the crawl is driven by the site's own page lists.

How do I crawl a website for RAG?

Use each row as one document: markdown as the content, url, title and language as metadata, and headings to split the text by section before embedding. Load the rows into your vector database with your own script or an Apify integration.

Does it work like Apify's Website Content Crawler?

It covers the HTML-only part of that job (Cheerio-style crawling, no browser) at a fraction of the cost. It does not render JavaScript, so it is not a replacement for browser mode.

What are alternatives to this Actor?

The most-used option on the Store is Website Content Crawler by Apify, which also offers browser-based crawling for JavaScript sites. Choose it when your target needs JavaScript rendering. Choose this Actor when the pages are server-rendered HTML (most docs, blogs and help centers) and you want the lowest cost per page. For a single query rather than a crawl, RAG Web Browser searches the web and returns Markdown for the top results.

Why use this instead of a hosted website-to-Markdown API or a converter extension?

A browser extension or one-page converter typically handles one page at a time. This Actor crawls many pages in one run, honours robots.txt, applies include and exclude patterns, and returns structured JSON with metadata, so you can run it on a schedule and feed a pipeline. Billing is per page with real content, so empty and blocked pages cost nothing.

Does it support llms.txt?

Yes, as a way to find pages: with useLlmsTxt on, it reads /llms.txt and queues the pages it links to. It does not generate an llms.txt file for your own site.

Can I use the output with LangChain, LlamaIndex or a vector DB?

Yes. Each row is one document: use markdown as the page content and url, title, language as metadata. headings helps with section-based chunking.

How do I detect changed pages?

Turn on includeContentHash and compare hashes between runs.

Why was a page not returned?

Run again with includeSkippedPages on and look at contentStatus: needs-javascript, blocked, thin (fewer than minWords words), http-error or not-html.

Is it polite?

It honours robots.txt by default, caps concurrency (default 5, max 20), and backs off automatically on 429 and 503 responses.

Legal

Crawl only content you are allowed to access and use. Public pages are fetched the way a normal browser would; you are responsible for complying with each site's terms of service, copyright and applicable data-protection law (for example GDPR) when you store or reuse the content.

Related Actors

Actor	What it does
Website Content Crawler by Apify	The most-used crawler for LLM data, with browser modes for JavaScript sites
RAG Web Browser	Web search plus Markdown of the top results, for AI agents
Pinecone Integration	Push crawled content into a Pinecone vector index
YouTube Transcript Scraper	Captions and full text of YouTube videos for LLM and RAG use
Contact Info Scraper	Emails, phone numbers and social links from company websites
Substack Scraper	Newsletter posts