# Site Crawler: Website → Markdown Corpus for LLM/RAG (`boxbox10/site-crawler`) Actor Crawl a whole website or docs site and get one clean, LLM-ready Markdown + JSON record per page (title, headings, content, links, token count). Built for RAG ingestion and AI knowledge bases. - **URL**: https://apify.com/boxbox10/site-crawler.md - **Developed by:** [Marvin Eguilos](https://apify.com/boxbox10) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $3.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Site Crawler — Website → Markdown Corpus for LLM & RAG **Point it at a docs site, blog, or knowledge base. Get back the whole thing as clean, LLM-ready Markdown — one tidy record per page.** The crawler follows links for you, strips the navbars, ads, and cookie banners, and returns structured JSON (title, headings, main content, links, metadata, accurate token counts) for every page it visits. Built for RAG ingestion, AI agents, and anyone who needs a *whole website's content* — not just a single URL. > Turn "crawl this docs site into my vector DB" into a single Actor run. Clean Markdown, pre-counted tokens, ready to chunk and embed. --- ### ✨ What it does - **Whole-site crawling** — give it a start URL (or a few); it follows links with configurable depth and a hard page cap. - **One clean record per page** — Mozilla Readability strips nav, sidebars, ads, and footers so you keep just the article body. - **HTML → Markdown** — high-fidelity conversion (GitHub-Flavored Markdown: tables, code blocks, lists, links) via Turndown. - **Structured JSON** — `url`, `title`, `description`, `siteName`, `lang`, `headings[]`, `links[]`, `wordCount`, `tokenCount`, `depth`, `fetchedAt`. - **Accurate token counts** — counted with the GPT/`cl100k`-family tokenizer so you know exactly how much context each page costs *before* embedding. - **Scope control** — `sameDomainOnly`, `includeGlobs`, `excludeGlobs`, `maxDepth`, and `maxPages` keep the crawl exactly where you want it. - **Robust by design** — one bad page never kills the run. Failed pages go to a separate `failures` dataset (and are **never charged**). - **Polite crawling** — respects `robots.txt` by default, sends a real User-Agent, dedupes URLs, and retries transient errors. - **JS rendering when you need it** — flip `renderJs: true` to render client-side sites with a headless browser (opt-in, higher compute). --- ### 🎯 Use cases | You want to… | This Actor gives you… | |---|---| | **Ingest a whole docs site into RAG** | Every page as clean Markdown chunks with token counts, ready to embed. | | **Build an LLM knowledge base from a wiki/blog** | A complete Markdown corpus — no manual URL lists. | | **Keep a vector DB in sync with a website** | Re-run to re-crawl; each page is a stable, diff-friendly record. | | **Give an AI agent a site's knowledge** | Structured JSON your agent can reason over — no HTML noise. | | **Mirror docs to a repo/wiki** | Publishable Markdown you can drop straight into version control. | --- ### 📥 Input | Field | Type | Default | Description | |---|---|---|---| | `startUrls` | string[] | — **(required)** | One or more URLs to start crawling from. | | `maxPages` | integer | `50` | Hard cap on pages crawled per run. | | `maxDepth` | integer | `2` | Link-hops from the start URLs (0 = start URLs only). | | `sameDomainOnly` | boolean | `true` | Only follow links on the same domain as each start URL. | | `includeGlobs` | string[] | `[]` | Only crawl URLs matching these globs (e.g. `https://docs.example.com/**`). | | `excludeGlobs` | string[] | `[]` | Skip URLs matching these globs (e.g. `**/tag/**`, `**/*.pdf`). | | `outputFormat` | `both` \| `markdown` \| `json` | `both` | Include Markdown, JSON fields, or both. | | `onlyMainContent` | boolean | `true` | Strip nav/ads/sidebars with Readability. | | `includeLinks` | boolean | `true` | Include extracted absolute links + anchor text. | | `renderJs` | boolean | `false` | Render JS-heavy sites with a headless browser (higher cost). | | `respectRobotsTxt` | boolean | `true` | Skip URLs disallowed by the site's `robots.txt`. | | `maxConcurrency` | integer | `5` | Max pages fetched in parallel. | | `maxTokens` | integer | `0` | Truncate each page's Markdown to ~N tokens (`0` = no limit). | #### Example input ```json { "startUrls": ["https://crawlee.dev/js/docs/quick-start"], "maxPages": 50, "maxDepth": 2, "sameDomainOnly": true, "includeGlobs": ["https://crawlee.dev/js/docs/**"], "excludeGlobs": ["**/api/**"], "onlyMainContent": true } ```` *** ### 📤 Output One dataset item per successfully crawled page. Successful example: ```json { "url": "https://example.com/", "finalUrl": "https://example.com/", "statusCode": 200, "depth": 0, "title": "Example Domain", "description": null, "siteName": null, "lang": "en", "byline": null, "excerpt": "This domain is for use in documentation examples...", "headings": [], "wordCount": 17, "tokenCount": 29, "fetchedAt": "2026-07-19T02:17:19.954Z", "markdown": "This domain is for use in documentation examples without needing permission...", "links": [ { "url": "https://iana.org/domains/example", "text": "Learn more" } ] } ``` Failed pages are written to a **separate `failures` dataset** (never the default dataset, and **never charged**): ```json { "url": "https://example.com/broken-link", "finalUrl": "https://example.com/broken-link", "statusCode": 404, "error": "Request blocked - received 404 status code.", "fetchedAt": "2026-07-19T02:17:19.831Z" } ``` *** ### 🔌 Call it from code (Apify API) ```bash curl -X POST "https://api.apify.com/v2/acts/YOUR_USERNAME~site-crawler/run-sync-get-dataset-items?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"startUrls":["https://crawlee.dev/js/docs/quick-start"],"maxPages":25}' ``` ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_TOKEN' }); const { defaultDatasetId } = await client .actor('YOUR_USERNAME/site-crawler') .call({ startUrls: ['https://crawlee.dev/js/docs/quick-start'], maxPages: 25 }); const { items } = await client.dataset(defaultDatasetId).listItems(); console.log(`Crawled ${items.length} pages`); ``` *** ### 💸 Pricing (Pay-Per-Event) | Event | Price | |---|---| | Actor start | **$0.05** per run | | Crawled page | **$0.003** per successful page | - **You only pay for pages that succeed** — failed pages go to a separate dataset and are never charged. - Set `maxPages` to cap your spend precisely: a 50-page crawl costs at most **$0.05 + 50 × $0.003 = $0.20**. - **🎁 Free tier:** free-plan users' platform usage is covered by Apify, so you can try it and run small crawls at **no cost** before scaling up. *** ### ⚖️ Acceptable use This is a general-purpose crawling + format-conversion tool: **you supply the start URLs and are responsible for having the right to crawl and use the content.** By default the Actor respects `robots.txt`, stays on the same domain, and identifies itself with a descriptive User-Agent. It does not target any single platform's private API and does not harvest personal data as a feature. Please crawl responsibly and comply with each site's terms of service and applicable law. *** ### 🧱 Under the hood Node.js · [Crawlee](https://crawlee.dev) (Cheerio + optional Playwright, `enqueueLinks`) · [@mozilla/readability](https://github.com/mozilla/readability) · [Turndown](https://github.com/mixmark-io/turndown) (+ GFM) · [gpt-tokenizer](https://github.com/niieani/gpt-tokenizer) · Apify SDK. Stateless — nothing is stored between runs. # Actor input Schema ## `startUrls` (type: `array`): One or more URLs to start crawling from. The crawler follows links from these pages. Each successful page is one billable result. ## `maxPages` (type: `integer`): Hard cap on how many pages are crawled in a single run. The crawler stops enqueuing new links once this many pages have been reached. ## `maxDepth` (type: `integer`): How many link-hops away from the start URLs to follow (0 = only the start URLs, 1 = start URLs + pages they link to, etc.). ## `sameDomainOnly` (type: `boolean`): Only follow links that stay on the same (registrable) domain as each start URL. Strongly recommended to avoid crawling the whole internet. ## `includeGlobs` (type: `array`): Optional. Only crawl URLs matching at least one of these glob patterns (e.g. "https://docs.example.com/\*\*"). Leave empty to allow all (subject to sameDomainOnly). ## `excludeGlobs` (type: `array`): Optional. Skip URLs matching any of these glob patterns (e.g. "**/tag/**", "\*\*/\*.pdf"). Applied after includeGlobs. ## `outputFormat` (type: `string`): What to include per result: Markdown only, JSON fields only, or both. ## `onlyMainContent` (type: `boolean`): Use Readability to strip nav, ads, sidebars, and footers — keep just the article body. ## `includeLinks` (type: `boolean`): Include the list of extracted links (absolute URLs + anchor text) in each result. ## `renderJs` (type: `boolean`): Enable for JS-heavy / client-rendered sites. Uses a headless browser (higher compute cost). Leave off for most static docs/blogs. ## `respectRobotsTxt` (type: `boolean`): Skip URLs disallowed by the target site's robots.txt. You are responsible for having the right to crawl the site. ## `maxConcurrency` (type: `integer`): Maximum number of pages fetched in parallel. Lower this to be gentler on the target site. ## `maxTokens` (type: `integer`): Optional. Truncate each page's Markdown to approximately this many tokens (0 = no limit). Useful for fitting LLM context windows. ## Actor input object example ```json { "startUrls": [ "https://docs.example.com" ], "maxPages": 50, "maxDepth": 2, "sameDomainOnly": true, "includeGlobs": [ "https://docs.example.com/**" ], "excludeGlobs": [ "**/login/**", "**/*.pdf" ], "outputFormat": "both", "onlyMainContent": true, "includeLinks": true, "renderJs": false, "respectRobotsTxt": true, "maxConcurrency": 5, "maxTokens": 0 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://crawlee.dev/js/docs/quick-start" ] }; // Run the Actor and wait for it to finish const run = await client.actor("boxbox10/site-crawler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://crawlee.dev/js/docs/quick-start"] } # Run the Actor and wait for it to finish run = client.actor("boxbox10/site-crawler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://crawlee.dev/js/docs/quick-start" ] }' | apify call boxbox10/site-crawler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=boxbox10/site-crawler", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Site Crawler: Website → Markdown Corpus for LLM/RAG", "description": "Crawl a whole website or docs site and get one clean, LLM-ready Markdown + JSON record per page (title, headings, content, links, token count). Built for RAG ingestion and AI knowledge bases.", "version": "0.1", "x-build-id": "194TjUCC5Mw5bpeEE" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/boxbox10~site-crawler/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-boxbox10-site-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/boxbox10~site-crawler/runs": { "post": { "operationId": "runs-sync-boxbox10-site-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/boxbox10~site-crawler/run-sync": { "post": { "operationId": "run-sync-boxbox10-site-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "One or more URLs to start crawling from. The crawler follows links from these pages. Each successful page is one billable result.", "items": { "type": "string" } }, "maxPages": { "title": "Max pages", "minimum": 1, "maximum": 50000, "type": "integer", "description": "Hard cap on how many pages are crawled in a single run. The crawler stops enqueuing new links once this many pages have been reached.", "default": 50 }, "maxDepth": { "title": "Max crawl depth", "minimum": 0, "maximum": 20, "type": "integer", "description": "How many link-hops away from the start URLs to follow (0 = only the start URLs, 1 = start URLs + pages they link to, etc.).", "default": 2 }, "sameDomainOnly": { "title": "Same domain only", "type": "boolean", "description": "Only follow links that stay on the same (registrable) domain as each start URL. Strongly recommended to avoid crawling the whole internet.", "default": true }, "includeGlobs": { "title": "Include URL globs", "type": "array", "description": "Optional. Only crawl URLs matching at least one of these glob patterns (e.g. \"https://docs.example.com/**\"). Leave empty to allow all (subject to sameDomainOnly).", "items": { "type": "string" } }, "excludeGlobs": { "title": "Exclude URL globs", "type": "array", "description": "Optional. Skip URLs matching any of these glob patterns (e.g. \"**/tag/**\", \"**/*.pdf\"). Applied after includeGlobs.", "items": { "type": "string" } }, "outputFormat": { "title": "Output format", "enum": [ "both", "markdown", "json" ], "type": "string", "description": "What to include per result: Markdown only, JSON fields only, or both.", "default": "both" }, "onlyMainContent": { "title": "Only main content", "type": "boolean", "description": "Use Readability to strip nav, ads, sidebars, and footers — keep just the article body.", "default": true }, "includeLinks": { "title": "Include links", "type": "boolean", "description": "Include the list of extracted links (absolute URLs + anchor text) in each result.", "default": true }, "renderJs": { "title": "Render JavaScript (Playwright)", "type": "boolean", "description": "Enable for JS-heavy / client-rendered sites. Uses a headless browser (higher compute cost). Leave off for most static docs/blogs.", "default": false }, "respectRobotsTxt": { "title": "Respect robots.txt", "type": "boolean", "description": "Skip URLs disallowed by the target site's robots.txt. You are responsible for having the right to crawl the site.", "default": true }, "maxConcurrency": { "title": "Max concurrency", "minimum": 1, "maximum": 50, "type": "integer", "description": "Maximum number of pages fetched in parallel. Lower this to be gentler on the target site.", "default": 5 }, "maxTokens": { "title": "Max tokens per page", "minimum": 0, "maximum": 2000000, "type": "integer", "description": "Optional. Truncate each page's Markdown to approximately this many tokens (0 = no limit). Useful for fitting LLM context windows.", "default": 0 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```