Site Crawler — Website → Markdown Corpus for LLM & RAG

Point it at a docs site, blog, or knowledge base. Get back the whole thing as clean, LLM-ready Markdown — one tidy record per page. The crawler follows links for you, strips the navbars, ads, and cookie banners, and returns structured JSON (title, headings, main content, links, metadata, accurate token counts) for every page it visits. Built for RAG ingestion, AI agents, and anyone who needs a whole website's content — not just a single URL.

Turn "crawl this docs site into my vector DB" into a single Actor run. Clean Markdown, pre-counted tokens, ready to chunk and embed.

---

✨ What it does

• Whole-site crawling — give it a start URL (or a few); it follows links with configurable depth and a hard page cap.
• One clean record per page — Mozilla Readability strips nav, sidebars, ads, and footers so you keep just the article body.
• HTML → Markdown — high-fidelity conversion (GitHub-Flavored Markdown: tables, code blocks, lists, links) via Turndown.
• Structured JSON — url, title, description, siteName, lang, headings[], links[], wordCount, tokenCount, depth, fetchedAt.
• Accurate token counts — counted with the GPT/cl100k-family tokenizer so you know exactly how much context each page costs before embedding.
• Scope control — sameDomainOnly, includeGlobs, excludeGlobs, maxDepth, and maxPages keep the crawl exactly where you want it.
• Robust by design — one bad page never kills the run. Failed pages go to a separate failures dataset (and are never charged).
• Polite crawling — respects robots.txt by default, sends a real User-Agent, dedupes URLs, and retries transient errors.
• JS rendering when you need it — flip renderJs: true to render client-side sites with a headless browser (opt-in, higher compute).

---

🎯 Use cases

You want to…	This Actor gives you…
Ingest a whole docs site into RAG	Every page as clean Markdown chunks with token counts, ready to embed.
Build an LLM knowledge base from a wiki/blog	A complete Markdown corpus — no manual URL lists.
Keep a vector DB in sync with a website	Re-run to re-crawl; each page is a stable, diff-friendly record.
Give an AI agent a site's knowledge	Structured JSON your agent can reason over — no HTML noise.
Mirror docs to a repo/wiki	Publishable Markdown you can drop straight into version control.

---

📥 Input

Field	Type	Default	Description
startUrls	string[]	— (required)	One or more URLs to start crawling from.
maxPages	integer	50	Hard cap on pages crawled per run.
maxDepth	integer	2	Link-hops from the start URLs (0 = start URLs only).
sameDomainOnly	boolean	true	Only follow links on the same domain as each start URL.
includeGlobs	string[]	[]	Only crawl URLs matching these globs (e.g. https://docs.example.com/**).
excludeGlobs	string[]	[]	Skip URLs matching these globs (e.g. **/tag/**, **/*.pdf).
outputFormat	both | markdown | json	both	Include Markdown, JSON fields, or both.
onlyMainContent	boolean	true	Strip nav/ads/sidebars with Readability.
includeLinks	boolean	true	Include extracted absolute links + anchor text.
renderJs	boolean	false	Render JS-heavy sites with a headless browser (higher cost).
respectRobotsTxt	boolean	true	Skip URLs disallowed by the site's robots.txt.
maxConcurrency	integer	5	Max pages fetched in parallel.
maxTokens	integer	0	Truncate each page's Markdown to ~N tokens (0 = no limit).

Example input

{
  "startUrls": ["https://crawlee.dev/js/docs/quick-start"],
  "maxPages": 50,
  "maxDepth": 2,
  "sameDomainOnly": true,
  "includeGlobs": ["https://crawlee.dev/js/docs/**"],
  "excludeGlobs": ["**/api/**"],
  "onlyMainContent": true
}

---

📤 Output

One dataset item per successfully crawled page. Successful example:

{
  "url": "https://example.com/",
  "finalUrl": "https://example.com/",
  "statusCode": 200,
  "depth": 0,
  "title": "Example Domain",
  "description": null,
  "siteName": null,
  "lang": "en",
  "byline": null,
  "excerpt": "This domain is for use in documentation examples...",
  "headings": [],
  "wordCount": 17,
  "tokenCount": 29,
  "fetchedAt": "2026-07-19T02:17:19.954Z",
  "markdown": "This domain is for use in documentation examples without needing permission...",
  "links": [
    { "url": "https://iana.org/domains/example", "text": "Learn more" }
  ]
}

Failed pages are written to a separate failures dataset (never the default dataset, and never charged):

{
  "url": "https://example.com/broken-link",
  "finalUrl": "https://example.com/broken-link",
  "statusCode": 404,
  "error": "Request blocked - received 404 status code.",
  "fetchedAt": "2026-07-19T02:17:19.831Z"
}

---

🔌 Call it from code (Apify API)

curl -X POST "https://api.apify.com/v2/acts/YOUR_USERNAME~site-crawler/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://crawlee.dev/js/docs/quick-start"],"maxPages":25}'

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const { defaultDatasetId } = await client
    .actor('YOUR_USERNAME/site-crawler')
    .call({ startUrls: ['https://crawlee.dev/js/docs/quick-start'], maxPages: 25 });
const { items } = await client.dataset(defaultDatasetId).listItems();
console.log(`Crawled ${items.length} pages`);

---

💸 Pricing (Pay-Per-Event)

Event	Price
Actor start	$0.05 per run
Crawled page	$0.003 per successful page

• You only pay for pages that succeed — failed pages go to a separate dataset and are never charged.
• Set maxPages to cap your spend precisely: a 50-page crawl costs at most $0.05 + 50 × $0.003 = $0.20.
• 🎁 Free tier: free-plan users' platform usage is covered by Apify, so you can try it and run small crawls at no cost before scaling up.

---

⚖️ Acceptable use

This is a general-purpose crawling + format-conversion tool: you supply the start URLs and are responsible for having the right to crawl and use the content. By default the Actor respects robots.txt, stays on the same domain, and identifies itself with a descriptive User-Agent. It does not target any single platform's private API and does not harvest personal data as a feature. Please crawl responsibly and comply with each site's terms of service and applicable law.

---

🧱 Under the hood

Node.js · Crawlee (Cheerio + optional Playwright, enqueueLinks) · @mozilla/readability · Turndown (+ GFM) · gpt-tokenizer · Apify SDK. Stateless — nothing is stored between runs.