# Website to Markdown for RAG & LLMs (`hereditary_model/website-to-markdown`) Actor Crawls a website and converts every page into clean Markdown, ready for RAG pipelines, vector databases, and LLM context. Pay per page converted. - **URL**: https://apify.com/hereditary\_model/website-to-markdown.md - **Developed by:** [Aaron Marxsen](https://apify.com/hereditary_model) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Website to Markdown for RAG & LLMs Turn any website into clean, LLM-ready Markdown in one run. This Actor crawls a site, strips away navigation, footers, cookie banners, and scripts, and converts the real content of every page into Markdown โ€” the ideal format for RAG pipelines, vector databases, fine-tuning datasets, and LLM context windows. ### Why Markdown? LLMs work dramatically better with clean Markdown than with raw HTML: fewer tokens, preserved structure (headings, lists, code blocks, links), no markup noise. This Actor does the cleanup for you at crawl time. ### Features - ๐Ÿ•ท๏ธ Crawls an entire site or just a section (URL glob filters, e.g. `https://example.com/docs/**`) - ๐Ÿงน Removes navigation, headers, footers, sidebars, cookie banners, and scripts - ๐ŸŽฏ Prefers `
` / `
` content when the page provides it - ๐Ÿ“ GitHub-style Markdown: ATX headings, fenced code blocks - ๐Ÿšซ Skips near-empty pages so your dataset stays clean - ๐Ÿ’ธ Pay per page converted โ€” skipped pages cost nothing ### Input | Field | Type | Description | | --- | --- | --- | | `startUrls` | array | Where the crawl starts (stays on the same domain) | | `maxPages` | integer | Max pages to crawl (default 50) | | `includeUrlGlobs` | array | Only crawl URLs matching these globs (optional) | | `minTextLength` | integer | Skip pages with less extracted text than this (default 100) | | `proxyConfiguration` | object | Optional proxy settings | ### Output One dataset item per page: ```json { "url": "https://example.com/docs/getting-started", "title": "Getting Started โ€” Example", "description": "Learn how to ...", "markdown": "# Getting Started\n\nWelcome to ...", "charCount": 5321, "crawledAt": "2026-07-05T12:00:00.000Z" } ```` Export as JSON/CSV from the **Storage** tab, or stream straight into your pipeline via the [Apify API](https://docs.apify.com/api/v2) โ€” perfect for feeding LangChain, LlamaIndex, or a vector DB loader. ### Typical uses - Build a RAG knowledge base from your product docs - Keep an LLM assistant in sync with a changing website (run on a schedule) - Create clean text corpora for evaluation and fine-tuning - Migrate CMS content to Markdown # Actor input Schema ## `startUrls` (type: `array`): Website URL(s) where the crawl starts. The crawler stays on the same domain. ## `maxPages` (type: `integer`): Maximum number of pages to crawl and convert. Each converted page is one billable event. ## `includeUrlGlobs` (type: `array`): Only crawl URLs matching these glob patterns (e.g. https://example.com/docs/\*\*). Leave empty to crawl the whole domain. ## `minTextLength` (type: `integer`): Skip pages whose extracted text is shorter than this many characters (filters out empty or navigation-only pages). ## `proxyConfiguration` (type: `object`): Proxy settings used for crawling. Defaults to no proxy. ## Actor input object example ```json { "startUrls": [ { "url": "https://docs.apify.com/platform" } ], "maxPages": 50, "includeUrlGlobs": [], "minTextLength": 100 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://docs.apify.com/platform" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("hereditary_model/website-to-markdown").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://docs.apify.com/platform" }] } # Run the Actor and wait for it to finish run = client.actor("hereditary_model/website-to-markdown").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://docs.apify.com/platform" } ] }' | apify call hereditary_model/website-to-markdown --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=hereditary_model/website-to-markdown", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Website to Markdown for RAG & LLMs", "description": "Crawls a website and converts every page into clean Markdown, ready for RAG pipelines, vector databases, and LLM context. Pay per page converted.", "version": "0.1", "x-build-id": "7nMqC2NSnThr9LNNi" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/hereditary_model~website-to-markdown/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-hereditary_model-website-to-markdown", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/hereditary_model~website-to-markdown/runs": { "post": { "operationId": "runs-sync-hereditary_model-website-to-markdown", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/hereditary_model~website-to-markdown/run-sync": { "post": { "operationId": "run-sync-hereditary_model-website-to-markdown", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "Website URL(s) where the crawl starts. The crawler stays on the same domain.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxPages": { "title": "Max pages", "minimum": 1, "maximum": 100000, "type": "integer", "description": "Maximum number of pages to crawl and convert. Each converted page is one billable event.", "default": 50 }, "includeUrlGlobs": { "title": "Include URL globs", "type": "array", "description": "Only crawl URLs matching these glob patterns (e.g. https://example.com/docs/**). Leave empty to crawl the whole domain.", "default": [], "items": { "type": "string" } }, "minTextLength": { "title": "Minimum text length", "type": "integer", "description": "Skip pages whose extracted text is shorter than this many characters (filters out empty or navigation-only pages).", "default": 100 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Proxy settings used for crawling. Defaults to no proxy." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```