Website Content Crawler for AI — Markdown, RAG Chunks & PDFs

✅ Checked daily. An automated test runs this Actor on the Apify cloud every day and checks that it still returns real data, so if the website changes and something breaks, it's caught the same day and fixed.

Crawl any website into clean Markdown or plain text ready for AI, RAG pipelines, vector databases, custom GPTs and LLM fine-tuning.

Why this crawler

• Fast and cheap by default. Pages are fetched over HTTP with real-browser TLS fingerprints. A stealth browser (Camoufox) starts only for the pages that need it, e.g. JavaScript apps or bot checks. You pay a fixed price per page, not for compute time.
• Clean content. Main-content extraction removes menus, headers, footers, cookie banners and ads. Headings, lists, tables and links are kept as Markdown. A "whole page" mode and your own CSS selectors to remove are also available.
• Finds every page. It reads robots.txt sitemaps (including sitemap indexes and .gz files) and /llms.txt as well as following links, and has include / exclude globs and scope modes.
• Documents too. It parses linked PDF, DOCX, PPTX, XLSX, CSV, TXT and MD files into Markdown in the same run.
• RAG-ready. It can split each page into heading-aware chunks with token counts (OpenAI cl100k) and a heading breadcrumb, plus a ready-to-import chunks.jsonl.
• Incremental re-crawls. Give the crawl a key. Later runs mark each page new, changed or unchanged, skip unchanged ones (no charge), and list pages that disappeared, so you can keep a knowledge base in sync cheaply.
• One-file exports. Each run also saves site.md (the whole site as one Markdown file) and llms.txt (a page index).
• Rich metadata. Title, description, author, published and modified dates, language, canonical URL, site name, image, keywords, headings outline, word and token counts, content hash, optional JSON-LD and link lists.

Input

Field	Default	Description
startUrls	–	Where to start (pages or direct document links)
crawlScope	path	path (same folder), host (whole site), domain (+ subdomains), single
maxPages, maxDepth	100, 20	Limits
includeGlobs, excludeGlobs	–	e.g. https://example.com/docs/**, **/tag/**
useSitemaps, useLlmsTxt, respectRobotsTxt	true	Discovery
renderingMode	auto	auto, http, browser
contentMode	article	article (main content) or full
outputFormats	["markdown"]	markdown, text, html
markdownLinks, markdownImages, removeSelectors	true, false, –	Markdown options
chunkSize, chunkOverlap	0 (off), 100	RAG chunks in tokens
parseDocuments	true	PDF / Office / text files
incrementalName, skipUnchanged	–, true	Change detection between runs
saveCombinedFiles	true	site.md, llms.txt, chunks.jsonl
maxConcurrency, browserConcurrency, browserWaitSecs, requestDelayMs	10, 2, 2, 0	Speed / politeness
proxyConfiguration	–	Optional; automatic proxy escalation is built in

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "crawlScope": "host",
  "maxPages": 2000,
  "chunkSize": 800,
  "incrementalName": "example-docs"
}

Output (one row per page or document)

url, title, description, author, published_at, modified_at, language, canonical_url, site_name, image, keywords, markdown (and/or text, html), headings, word_count, token_count, content_hash, change_status, chunks (index, heading_path, text, tokens), content_type (html / pdf / docx / …), fetched_with (http / browser), status_code, depth, referrer, discovered_via (start / link / sitemap / llms.txt), internal_links_count, external_links_count, optional links and json_ld, crawled_at.

Key-value store: site.md, llms.txt, chunks.jsonl, OUTPUT (run report: counts, failures, removed pages).

Example output

One item from a real run (long texts shortened):

{
  "url": "https://react.dev/learn",
  "requested_url": "https://react.dev/learn",
  "status_code": 200,
  "depth": 0,
  "referrer": "",
  "discovered_via": "start",
  "fetched_with": "http",
  "proxy": "direct",
  "content_type": "html",
  "title": "Quick Start – React",
  "description": "The library for web and native user interfaces",
  "author": "",
  "published_at": "",
  "modified_at": "",
  "language": "en",
  "canonical_url": "https://react.dev/learn",
  "site_name": "",
  "image": "https://react.dev/images/og/learn.png",
  "keywords": "",
  "markdown": "# Quick Start – React\n\nWelcome to the React documentation! This page will give you an introduction to 80% of the React concepts that you will use on a daily …",
  "headings": [
    {
      "level": 1,
      "text": "Quick Start – React"
    },
    {
      "level": 3,
      "text": "You will learn"
    },
    "…"
  ],
  "word_count": 1875,
  "token_count": 3339,
  "content_hash": "6a3fc0998df67f3a",
  "change_status": null,
  "internal_links_count": 62,
  "external_links_count": 23,
  "chunks": [
    {
      "index": 0,
      "heading_path": "Quick Start – React",
      "text": "# Quick Start – React\n\nWelcome to the React documentation! This page will give you an introduction to 80% of the React concepts that you will use on a daily …",
      "tokens": 410
    },
    {
      "index": 1,
      "heading_path": "Quick Start – React",
      "text": "\\</button> ); } export default function MyApp() { return ( \\<div> \\<h1>Welcome to my app\\</h1> \\<MyButton /> \\</div> ); } The `export default` keywords speci…",
      "tokens": 443
    },
    "…"
  ],
  "crawled_at": "2026-11-06T00:28:44+00:00"
}

How it compares

	This Actor	Typical alternative
Fields per result	29 (see Example output)	Varies by actor
Freshness	Read live from the source when the run starts	Varies
Reliability	Tested daily by an automated run on the Apify cloud (see the top of this page)	—
Price	$3.00 per 1,000 pages	$1.56–$8.88 per 1,000 results (comparable Actors on the Apify Store, 6 Nov 2026)

Pricing

Pay per result, no monthly fee:

• $3.00 per 1,000 pages: One crawled page or document saved to your dataset.
• $1.50 per 1,000 browser renders (add-on): Only for pages that needed a real browser (JavaScript apps, bot checks).
• $2.00 per 1,000 residential proxys (add-on): Only for pages that had to be fetched through a residential IP.
• $0.003 per run (start).

Bronze plans pay 5% less, Silver 10% less, Gold and above 15% less.

Examples: 1,000 pages = $3.00; 1,000 pages plus 1,000 browser renders = $4.50; 10,000 = $30.00.

Fixed prices, no compute surprises:

FAQ

How do I run it on a schedule? Create a task with your input and add a schedule in Apify Console (e.g. daily at 6 am). Each run writes a fresh dataset.

How do I get the data? Download the dataset as JSON, CSV, Excel, XML or HTML from the run page, or read it with the Apify API / Python and JavaScript clients. Integrations (Google Sheets, Make, Zapier, n8n, webhooks) work out of the box.

Do I need proxies or an account on the site? No. Proxies are handled inside the Actor and included in the price; no login is needed.

What if a run returns fewer results than expected? Check the run's key-value store OUTPUT record (when present) for a per-request report, then open an issue on the Issues tab with the run link and we'll look into it.

Is it legal? The Actor only collects publicly available data. You are responsible for using it in line with the website's terms, copyright and data-protection rules (GDPR, CCPA) — don't collect personal data you have no legal basis for.

Notes

• Only crawl sites you're allowed to. robots.txt is respected by default.
• Not affiliated with any of the sites you crawl.