# Web to Markdown Converter: AI-Ready Scraper for RAG & LLMs (`raional/web-to-markdown-converter`) Actor Convert any webpage into clean Markdown or JSON for AI, RAG, and LLM pipelines. Strips ads, navigation, and cookie banners. Optionally follows links to convert an entire site. Powered by the open-source Crawl4AI library. - **URL**: https://apify.com/raional/web-to-markdown-converter.md - **Developed by:** [Raion Al](https://apify.com/raional) (community) - **Categories:** AI, Agents, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $5.00 / 1,000 pages This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Web to Markdown Converter — AI-Ready Scraper for RAG & LLMs Convert **any webpage into clean Markdown or JSON**. Powered by [Crawl4AI](https://github.com/unclecode/crawl4ai) (Apache-2.0, 72,000+ GitHub stars), an open-source crawling library built for feeding LLMs and RAG pipelines. Ads, navigation, cookie banners, and other boilerplate are stripped automatically. Optionally follows links to pull an entire site, not just one page. Great for: **web to markdown**, **url to markdown**, AI/RAG knowledge bases, scraping docs sites for LLM context, feeding an AI agent live web content. ### What it does Give it one or more page URLs — it renders each page in a real headless browser (so JS-rendered sites work too), strips out clutter, and returns clean structured output. Turn on "Follow links" to also crawl and convert every page linked from your starting URL, up to a page cap you set. ### Example input ```json { "startUrls": ["https://docs.apify.com/academy"], "crawlLinks": true, "maxPages": 20, "outputFormat": "both" } ```` ### Output One row per page: | Field | Description | |---|---| | `url` | The page converted | | `title` | Page title | | `markdown` | Clean Markdown output (if requested) | | `links` | Internal/external links found on the page (if requested) | | `wordCount` | Word count of the converted content | | `status` | `ok` or `error` | ### Pricing Billed **per page successfully converted**. Failed pages are **not charged**. ### FAQ **Does this handle JavaScript-rendered pages?** Yes — pages are rendered in a real headless Chromium browser, not just fetched as raw HTML. **What does "Follow links" do?** Starting from your URL(s), it crawls linked pages on the same site (or across domains if you allow it) and converts each one too — useful for pulling a whole docs site in one run. **Does this respect robots.txt?** Yes, by default. ### Please note Only scan sites you own or have permission to crawl. Pages are processed transiently and not retained beyond the run. Built with [Apify Python SDK](https://docs.apify.com/sdk/python/) + [Crawl4AI](https://github.com/unclecode/crawl4ai). # Actor input Schema ## `startUrls` (type: `array`): Webpage URLs to convert to Markdown/JSON. ## `crawlLinks` (type: `boolean`): If enabled, also crawls and converts pages linked from the URLs above (e.g. to pull an entire docs site), up to Max pages below. ## `maxPages` (type: `integer`): Safety cap on total pages converted in one run. Only matters when 'Follow links' is enabled. ## `sameDomainOnly` (type: `boolean`): When following links, only crawl pages on the same domain as the starting URL. ## `outputFormat` (type: `string`): markdown = clean Markdown text. json = structured fields (title, links, word count). both = markdown + JSON. ## Actor input object example ```json { "startUrls": [ "https://docs.apify.com/academy" ], "crawlLinks": false, "maxPages": 20, "sameDomainOnly": true, "outputFormat": "both" } ``` # Actor output Schema ## `pages` (type: `string`): One dataset row per page: url, title, markdown, wordCount, status. ## `summary` (type: `string`): Counts of pages processed and errors. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://docs.apify.com/academy" ] }; // Run the Actor and wait for it to finish const run = await client.actor("raional/web-to-markdown-converter").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://docs.apify.com/academy"] } # Run the Actor and wait for it to finish run = client.actor("raional/web-to-markdown-converter").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://docs.apify.com/academy" ] }' | apify call raional/web-to-markdown-converter --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=raional/web-to-markdown-converter", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Web to Markdown Converter: AI-Ready Scraper for RAG & LLMs", "description": "Convert any webpage into clean Markdown or JSON for AI, RAG, and LLM pipelines. Strips ads, navigation, and cookie banners. Optionally follows links to convert an entire site. Powered by the open-source Crawl4AI library.", "version": "0.1", "x-build-id": "D2Uaas6kWrD0n8Unv" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/raional~web-to-markdown-converter/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-raional-web-to-markdown-converter", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/raional~web-to-markdown-converter/runs": { "post": { "operationId": "runs-sync-raional-web-to-markdown-converter", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/raional~web-to-markdown-converter/run-sync": { "post": { "operationId": "run-sync-raional-web-to-markdown-converter", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Page URLs", "type": "array", "description": "Webpage URLs to convert to Markdown/JSON.", "items": { "type": "string" } }, "crawlLinks": { "title": "Follow links on these pages", "type": "boolean", "description": "If enabled, also crawls and converts pages linked from the URLs above (e.g. to pull an entire docs site), up to Max pages below.", "default": false }, "maxPages": { "title": "Max pages per run", "minimum": 1, "maximum": 200, "type": "integer", "description": "Safety cap on total pages converted in one run. Only matters when 'Follow links' is enabled.", "default": 20 }, "sameDomainOnly": { "title": "Stay on the same domain", "type": "boolean", "description": "When following links, only crawl pages on the same domain as the starting URL.", "default": true }, "outputFormat": { "title": "Output format", "enum": [ "markdown", "json", "both" ], "type": "string", "description": "markdown = clean Markdown text. json = structured fields (title, links, word count). both = markdown + JSON.", "default": "both" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```