# Website to Markdown for LLMs (`agentictools/website-to-markdown-llm`) Actor Crawl a site and export clean Markdown with token counts and chunks, ready for RAG. - **URL**: https://apify.com/agentictools/website-to-markdown-llm.md - **Developed by:** [Ken Agland](https://apify.com/agentictools) (community) - **Categories:** AI, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $0.30 / 1,000 page converteds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Website to Markdown for LLMs Crawl a site and export clean Markdown with token counts and chunks, ready for RAG. ### What it does - Crawls pages starting from your URLs, staying on the same domain by default. - Strips boilerplate (scripts, styles, nav, header, footer, sidebars) and keeps the main content. - Converts the main content to clean Markdown. - Estimates token count for each page. - Optionally splits each page into token-sized chunks for retrieval-augmented generation. No API key needed. ### Example input ```json { "startUrls": [ "https://example.com" ], "maxPages": 10, "maxDepth": 2, "sameDomainOnly": true, "includeChunks": true, "chunkSizeTokens": 800 } ```` ### Input | Field | Type | Default | Description | |---|---|---|---| | `startUrls` | array of strings | (required) | URLs to start crawling from. | | `maxPages` | integer | 10 | Stop after this many pages in total. | | `maxDepth` | integer | 2 | Link hops to follow from a start URL. Start URLs are depth 0. | | `sameDomainOnly` | boolean | true | Only follow links on the same domain as their start URL. | | `includeChunks` | boolean | false | Include the chunk array in each item. | | `chunkSizeTokens` | integer | 800 | Target chunk size in estimated tokens. | ### Output Each dataset item is one page: ```json { "url": "https://example.com/", "title": "Example Domain", "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...", "charCount": 210, "tokenEstimate": 53, "chunkCount": 1, "chunks": [ "# Example Domain\n\nThis domain is for use in illustrative examples..." ] } ``` The `chunks` field is present only when `includeChunks` is true. #### Run summary (OUTPUT) The run's default key-value store record `OUTPUT` holds an aggregate: ```json { "startUrls": [ "https://example.com/" ], "pagesCrawled": 1, "totalChars": 210, "totalTokens": 53, "totalChunks": 1, "includeChunks": true, "chunkSizeTokens": 800, "maxPages": 10, "maxDepth": 2, "sameDomainOnly": true } ``` ### How it works The Actor fetches each page's HTML with plain fetch (retrying on 429 and 5xx with backoff, failing fast on other 4xx). It parses the HTML with cheerio, removes boilerplate, and picks the main content region. It converts that content to Markdown with turndown, falling back to a cheerio-based builder (headings, paragraphs, list items) if turndown cannot run. Tokens are estimated as characters divided by 4. Crawling uses an in-memory breadth-first queue with a small concurrency, staying on the same domain and honoring `maxPages` and `maxDepth`. MIT licensed. # Actor input Schema ## `startUrls` (type: `array`): One or more URLs to start crawling from. The crawler follows links found on these pages. ## `maxPages` (type: `integer`): Stop after crawling this many pages in total. ## `maxDepth` (type: `integer`): How many link hops to follow from a start URL. Start URLs are depth 0. ## `sameDomainOnly` (type: `boolean`): Only follow links that stay on the same domain as their start URL. Turn off to follow external links too. ## `includeChunks` (type: `boolean`): Split each page's Markdown into token-sized chunks for retrieval-augmented generation and include them in the output. ## `chunkSizeTokens` (type: `integer`): Target size of each chunk in estimated tokens. Only used when chunks are included. ## Actor input object example ```json { "startUrls": [ "https://docs.apify.com" ], "maxPages": 10, "maxDepth": 2, "sameDomainOnly": true, "includeChunks": false, "chunkSizeTokens": 800 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://example.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("agentictools/website-to-markdown-llm").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://example.com"] } # Run the Actor and wait for it to finish run = client.actor("agentictools/website-to-markdown-llm").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://example.com" ] }' | apify call agentictools/website-to-markdown-llm --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=agentictools/website-to-markdown-llm", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Website to Markdown for LLMs", "description": "Crawl a site and export clean Markdown with token counts and chunks, ready for RAG.", "version": "0.1", "x-build-id": "tSKa2h8JOiqqXW3VG" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/agentictools~website-to-markdown-llm/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-agentictools-website-to-markdown-llm", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/agentictools~website-to-markdown-llm/runs": { "post": { "operationId": "runs-sync-agentictools-website-to-markdown-llm", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/agentictools~website-to-markdown-llm/run-sync": { "post": { "operationId": "run-sync-agentictools-website-to-markdown-llm", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "One or more URLs to start crawling from. The crawler follows links found on these pages.", "items": { "type": "string" } }, "maxPages": { "title": "Maximum pages", "minimum": 1, "maximum": 1000, "type": "integer", "description": "Stop after crawling this many pages in total.", "default": 10 }, "maxDepth": { "title": "Maximum depth", "minimum": 0, "maximum": 20, "type": "integer", "description": "How many link hops to follow from a start URL. Start URLs are depth 0.", "default": 2 }, "sameDomainOnly": { "title": "Stay on the same domain", "type": "boolean", "description": "Only follow links that stay on the same domain as their start URL. Turn off to follow external links too.", "default": true }, "includeChunks": { "title": "Include chunks", "type": "boolean", "description": "Split each page's Markdown into token-sized chunks for retrieval-augmented generation and include them in the output.", "default": false }, "chunkSizeTokens": { "title": "Chunk size in tokens", "minimum": 50, "maximum": 8000, "type": "integer", "description": "Target size of each chunk in estimated tokens. Only used when chunks are included.", "default": 800 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```