# Website to Markdown — LLM & RAG Content Exporter (`sturdydata/website-markdown-exporter`) Actor Crawl any website and get one clean Markdown document per page — ready for RAG pipelines, vector databases, LLM fine-tuning, or docs migration. Boilerplate (nav, footers, cookie banners) stripped, main content auto-detected, sitemap-seeded crawling, robots.txt respected. HARD page caps and flat p... - **URL**: https://apify.com/sturdydata/website-markdown-exporter.md - **Developed by:** [Sturdy Data](https://apify.com/sturdydata) (community) - **Categories:** Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Website to Markdown — LLM & RAG Content Exporter Crawl any website and get **one clean Markdown document per page** — ready to chunk into a vector database, feed to a RAG pipeline, fine-tune on, or migrate into a docs system. ### Why this exporter The big content crawlers are powerful — and unpredictable. Their issue tabs are full of the same complaint: *"I set a 60-second timeout and the crawler ran 36 minutes."* Usage-based pricing plus browser rendering means you find out what a run cost after it's done. This actor is the opposite trade: - **Hard caps, flat pricing.** `maxPages` is a hard stop. 1 result = 1 page. A 500-page cap can never cost more than 500 results. There is no compute-unit roulette. - **No browser.** Pure HTTP + HTML parsing. Runs are fast and cheap; static and server-rendered sites (docs, blogs, marketing sites, knowledge bases — the 90% case for RAG) come out clean. - **Boilerplate stripped.** Nav, headers, footers, cookie banners, share buttons, sidebars removed. Main content auto-detected (`
`, `
`, common content containers) before conversion. - **Sitemap-seeded.** Reads `sitemap.xml` so orphan pages the link crawl would miss still get exported. - **Thin pages are free.** Pages under `minTextLength` characters (tag pages, pagination shells) are skipped and never billed. - **Robots.txt respected.** Always. **Honest limitation:** this actor does not execute JavaScript. Client-side-rendered SPAs will come out empty — use a browser-based crawler for those. That trade-off is exactly why this one is 5-10x cheaper and never hangs. ### Input | Field | Type | Default | Notes | |---|---|---|---| | `startUrls` | array | — | Crawl stays on the same host. | | `maxPages` | integer | `100` | **Hard cap** on exported pages. | | `maxDepth` | integer | `3` | Link depth from start page. | | `useSitemaps` | boolean | `true` | Seed the crawl from `sitemap.xml`. | | `includeUrlGlobs` | array | `[]` | e.g. `https://docs.example.com/guide/*` | | `excludeUrlGlobs` | array | `[]` | e.g. `*/tags/*`, `*/login*` | | `minTextLength` | integer | `200` | Skip (and don't bill) pages with less content. | | `proxyConfiguration` | object | Apify Proxy | | ```json { "startUrls": [{ "url": "https://docs.example.com" }], "maxPages": 500, "includeUrlGlobs": ["https://docs.example.com/*"], "excludeUrlGlobs": ["*/changelog/*"] } ```` ### Output schema One dataset item per page: ```json { "url": "https://docs.example.com/guide/getting-started", "canonicalUrl": "https://docs.example.com/guide/getting-started", "title": "Getting started", "description": "Install and configure in five minutes.", "lang": "en", "depth": 1, "markdown": "# Getting started\n\nInstall the CLI...", "wordCount": 812, "fetchedAt": "2026-07-04T10:00:00.000Z" } ``` ### Pricing **Pay per result. 1 result = 1 exported page.** Thin/skipped pages cost nothing. | Pages | Cost at $1.00 / 1,000 | |---|---| | 500 (typical docs site) | $0.50 | | 5,000 | $5.00 | | 50,000 | $50.00 | Your maximum spend is always `maxPages` — known before you press Start. ### Using from API / MCP / AI agents ```bash curl -X POST "https://api.apify.com/v2/acts//run-sync-get-dataset-items?token=" \ -H "Content-Type: application/json" \ -d '{ "startUrls": [{ "url": "https://docs.example.com" }], "maxPages": 50 }' ``` Pairs naturally with the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp): agents can pull a site into context as Markdown with a single tool call. ### FAQ **Tables?** Converted as readable text in v0.1; GFM table output is on the roadmap (see changelog policy below). **Why did I get fewer pages than `maxPages`?** The site is smaller than the cap, pages were thinner than `minTextLength`, or robots.txt disallowed sections. All three are correct behavior — you were not billed for them. **How do I know a run succeeded?** Every run writes a `RUN_SUMMARY` record to the key-value store: pages exported, thin pages skipped, and any failed URLs with their error. 404s and other HTTP errors are never scraped as content — they're retried and reported, not billed. ### Changelog - **0.1.0** (2026-07) — Initial release: main-content detection, boilerplate stripping, Markdown conversion, sitemap seeding, include/exclude globs, thin-page filter, hard page cap, per-run `RUN_SUMMARY` with error reporting. *** Maintained by **Sturdy Data** — boring, reliable scrapers. Report issues in the Issues tab; we respond within 24 hours. # Actor input Schema ## `startUrls` (type: `array`): Where to start crawling. The crawler stays on the same host. ## `maxPages` (type: `integer`): The run exports at most this many pages and then stops. This is a HARD cap — the run can never cost more than maxPages results. ## `maxDepth` (type: `integer`): How many clicks away from the start page to follow links. ## `useSitemaps` (type: `boolean`): Also read the site's sitemap.xml to find pages the link crawl might miss. ## `includeUrlGlobs` (type: `array`): Only crawl URLs matching at least one pattern, e.g. https://docs.example.com/guide/\*. Leave empty to crawl everything on the host. ## `excludeUrlGlobs` (type: `array`): Never crawl URLs matching any of these patterns, e.g. */tags/*, */login*. ## `minTextLength` (type: `integer`): Pages whose extracted content is shorter than this many characters are skipped (menus, stubs, pagination shells) — you don't pay for them. ## `proxyConfiguration` (type: `object`): Proxy to use. Apify Proxy (automatic) is recommended. ## Actor input object example ```json { "startUrls": [ { "url": "https://books.toscrape.com" } ], "maxPages": 100, "maxDepth": 3, "useSitemaps": true, "includeUrlGlobs": [], "excludeUrlGlobs": [], "minTextLength": 200, "proxyConfiguration": { "useApifyProxy": true } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://books.toscrape.com" } ], "includeUrlGlobs": [], "excludeUrlGlobs": [], "proxyConfiguration": { "useApifyProxy": true } }; // Run the Actor and wait for it to finish const run = await client.actor("sturdydata/website-markdown-exporter").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://books.toscrape.com" }], "includeUrlGlobs": [], "excludeUrlGlobs": [], "proxyConfiguration": { "useApifyProxy": True }, } # Run the Actor and wait for it to finish run = client.actor("sturdydata/website-markdown-exporter").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://books.toscrape.com" } ], "includeUrlGlobs": [], "excludeUrlGlobs": [], "proxyConfiguration": { "useApifyProxy": true } }' | apify call sturdydata/website-markdown-exporter --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=sturdydata/website-markdown-exporter", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Website to Markdown — LLM & RAG Content Exporter", "description": "Crawl any website and get one clean Markdown document per page — ready for RAG pipelines, vector databases, LLM fine-tuning, or docs migration. Boilerplate (nav, footers, cookie banners) stripped, main content auto-detected, sitemap-seeded crawling, robots.txt respected. HARD page caps and flat p...", "version": "0.1", "x-build-id": "NvqxDuSPZdbfZsQuq" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/sturdydata~website-markdown-exporter/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-sturdydata-website-markdown-exporter", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/sturdydata~website-markdown-exporter/runs": { "post": { "operationId": "runs-sync-sturdydata-website-markdown-exporter", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/sturdydata~website-markdown-exporter/run-sync": { "post": { "operationId": "run-sync-sturdydata-website-markdown-exporter", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "Where to start crawling. The crawler stays on the same host.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxPages": { "title": "Max pages (hard cap)", "minimum": 1, "maximum": 50000, "type": "integer", "description": "The run exports at most this many pages and then stops. This is a HARD cap — the run can never cost more than maxPages results.", "default": 100 }, "maxDepth": { "title": "Max link depth", "minimum": 0, "maximum": 10, "type": "integer", "description": "How many clicks away from the start page to follow links.", "default": 3 }, "useSitemaps": { "title": "Seed from sitemap.xml", "type": "boolean", "description": "Also read the site's sitemap.xml to find pages the link crawl might miss.", "default": true }, "includeUrlGlobs": { "title": "Include URL patterns (optional)", "type": "array", "description": "Only crawl URLs matching at least one pattern, e.g. https://docs.example.com/guide/*. Leave empty to crawl everything on the host.", "items": { "type": "string" } }, "excludeUrlGlobs": { "title": "Exclude URL patterns (optional)", "type": "array", "description": "Never crawl URLs matching any of these patterns, e.g. */tags/*, */login*.", "items": { "type": "string" } }, "minTextLength": { "title": "Minimum text length", "minimum": 0, "type": "integer", "description": "Pages whose extracted content is shorter than this many characters are skipped (menus, stubs, pagination shells) — you don't pay for them.", "default": 200 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Proxy to use. Apify Proxy (automatic) is recommended.", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```