# Webpage to Markdown Converter (`automation-lab/webpage-to-markdown-converter`) Actor Convert webpages to clean Markdown for LLM/RAG pipelines. Uses @mozilla/readability to strip ads, navigation, and footers. Outputs structured JSON. - **URL**: https://apify.com/automation-lab/webpage-to-markdown-converter.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** AI, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## 📄 Webpage to Markdown Converter Convert any webpage into clean, structured Markdown optimized for LLMs, RAG pipelines, AI knowledge bases, and content research. Uses [@mozilla/readability](https://github.com/mozilla/readability) to extract main content and strips ads, navigation, footers, and other noise — giving you only the substance. ### 🔍 What does it do? Webpage to Markdown Converter takes a list of URLs, fetches each page, extracts the main readable content using Mozilla's battle-tested Readability engine, and converts it to clean Markdown using Turndown. The result is structured JSON with the Markdown content, page title, word count, publication metadata, and error information for failed URLs. **Key capabilities:** - 🧠 Smart content extraction with @mozilla/readability (same engine Firefox uses for Reader Mode) - 📝 Clean Markdown output via Turndown — no JavaScript, no browser needed - 🚀 Fast HTTP-only processing — 256MB memory, no proxy needed - 🗃️ Rich metadata: author, description, published date, site name, language - ⚠️ Graceful error handling — bad URLs never crash the run, errors captured per URL - ⚙️ Configurable: toggle images, links, set content length limits ### 👤 Who is it for? **AI/LLM Developers** building RAG pipelines, vector databases, or knowledge bases who need clean text from URLs without building their own scraper infrastructure. **Content Researchers** collecting and analyzing web content for training data, competitor analysis, or documentation aggregation. **Data Engineers** building automated content processing pipelines that need to ingest web pages as structured data. **No-code users** on Make, Zapier, or n8n who want to convert webpages to text as part of automation workflows. ### 💡 Why use it? | Feature | This Actor | Competitors | |---------|-----------|-------------| | Price per page | **$0.002** | $0.005–$0.05 | | Content extraction | Mozilla Readability (smart) | Basic HTML strip | | Memory needed | 256 MB | 256–2048 MB | | Metadata fields | 5 fields (author, description, siteName, date, lang) | None | | Error details | Per-URL status code + message | Crash or skip silently | | Word count | ✅ Yes | ❌ No | The top competitor charges $0.05/page — **25x more** for less output. ### 📊 Output data For each URL, you receive a structured JSON object: | Field | Type | Description | |-------|------|-------------| | `url` | string | Input URL | | `title` | string | Page title from HTML/readability | | `markdown` | string | Clean Markdown content | | `wordCount` | integer | Word count of the Markdown | | `extractedAt` | string | ISO 8601 timestamp | | `metadata.author` | string\|null | Author from meta tags | | `metadata.description` | string\|null | Meta description | | `metadata.siteName` | string\|null | Site name (og:site_name) | | `metadata.publishedDate` | string\|null | Publication date | | `metadata.language` | string\|null | Content language code | | `statusCode` | integer\|null | HTTP response code | | `success` | boolean | Whether conversion succeeded | | `error` | string\|null | Error message (null on success) | ### 💰 How much does it cost to convert webpages to Markdown? **$0.002 per successfully converted page.** Failed URLs (404s, timeouts) are not charged. **Examples:** - 100 pages → ~$0.20 - 1,000 pages → ~$2.00 - 10,000 pages (monthly RAG pipeline) → ~$20.00 This is **25x cheaper** than the most popular competitor, with richer output and smarter content extraction. ### 🚀 How to use #### Step 1: Provide URLs Add URLs in the **URLs to convert** field. You can add as many as you need — the actor processes them sequentially. #### Step 2: Configure options (optional) - **Include images**: Keep or strip image links in the Markdown output - **Include links**: Keep or strip hyperlinks (useful for plain-text LLM input) - **Max content length**: Limit Markdown chars per page (useful for LLM token budgets) #### Step 3: Run and retrieve results Start the actor. Results appear in the dataset in real-time. Download as JSON, CSV, or JSONL. ### 📥 Input ```json { "urls": [ "https://en.wikipedia.org/wiki/Markdown", "https://docs.python.org/3/tutorial/", "https://news.ycombinator.com" ], "includeImages": true, "includeLinks": true, "maxContentLength": 0, "requestTimeout": 30, "maxRetries": 2 } ```` | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `urls` | string\[] | required | List of URLs to convert | | `includeImages` | boolean | `true` | Include image links in Markdown | | `includeLinks` | boolean | `true` | Include hyperlinks in Markdown | | `maxContentLength` | integer | `0` (unlimited) | Max chars per page (0 = unlimited) | | `requestTimeout` | integer | `30` | HTTP timeout in seconds | | `maxRetries` | integer | `2` | Retry attempts on network errors | ### 📤 Output ```json { "url": "https://en.wikipedia.org/wiki/Markdown", "title": "Markdown", "markdown": "From Wikipedia, the free encyclopedia\n\n### Overview\n\nMarkdown is a lightweight markup language...", "wordCount": 2859, "extractedAt": "2026-04-06T12:00:00.000Z", "metadata": { "author": "Contributors to Wikimedia projects", "description": null, "siteName": "Wikimedia Foundation, Inc.", "publishedDate": "2005-08-09T19:56:00Z", "language": "en" }, "statusCode": 200, "success": true, "error": null } ``` **Failed URL output:** ```json { "url": "https://example.com/page-not-found", "title": null, "markdown": null, "wordCount": 0, "extractedAt": "2026-04-06T12:00:01.000Z", "metadata": { "author": null, "description": null, "siteName": null, "publishedDate": null, "language": null }, "statusCode": 404, "success": false, "error": "HTTP 404: Not Found" } ``` ### 💡 Tips **For LLM/RAG pipelines:** - Set `includeImages: false` and `includeLinks: false` for cleaner text input - Use `maxContentLength` to match your LLM's context window (e.g., `50000` chars ≈ ~12k tokens) - The `wordCount` field helps you estimate token usage before sending to an LLM **For content research:** - Keep both images and links enabled (default) for full-fidelity Markdown - The `metadata.publishedDate` field is useful for freshness filtering - Failed URLs are always included in results (with `success: false`), so you know exactly what didn't work **For Wikipedia / documentation:** - These convert especially well — Readability excels at article-format content - Table content is preserved as Markdown tables **Performance:** - The actor processes URLs sequentially; for large batches (1000+), consider running multiple instances in parallel via the API ### 🔗 Integrations #### Zapier / Make / n8n Use the **Apify** integration to trigger this actor from any workflow: 1. Add an **Apify** step with actor `automation-lab/webpage-to-markdown-converter` 2. Pass your URL list as input 3. Read results from the dataset output #### LangChain ```python from apify_client import ApifyClient client = ApifyClient("YOUR_APIFY_TOKEN") run = client.actor("automation-lab/webpage-to-markdown-converter").call(run_input={ "urls": ["https://example.com"], "includeLinks": False }) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): if item["success"]: ## Feed to LangChain document loader docs.append(Document(page_content=item["markdown"], metadata={"source": item["url"]})) ``` #### LlamaIndex ```python from llama_index.core import Document ## Fetch converted pages and create LlamaIndex documents pages = [item for item in dataset_items if item["success"]] documents = [Document(text=p["markdown"], metadata={"url": p["url"], "title": p["title"]}) for p in pages] index = VectorStoreIndex.from_documents(documents) ``` #### Pinecone / Weaviate The `markdown` field feeds directly into any embedding pipeline. The `wordCount` helps you batch-split documents that exceed embedding model token limits. ### 🤖 API Usage #### Node.js ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' }); const run = await client.actor('automation-lab/webpage-to-markdown-converter').call({ urls: ['https://en.wikipedia.org/wiki/Web_scraping', 'https://docs.apify.com'], includeImages: false, maxContentLength: 50000, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); for (const item of items) { if (item.success) { console.log(`${item.url}: ${item.wordCount} words`); console.log(item.markdown.substring(0, 500)); } } ``` #### Python ```python from apify_client import ApifyClient client = ApifyClient("YOUR_APIFY_TOKEN") run = client.actor("automation-lab/webpage-to-markdown-converter").call(run_input={ "urls": ["https://en.wikipedia.org/wiki/Web_scraping", "https://docs.apify.com"], "includeImages": False, "maxContentLength": 50000 }) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): if item["success"]: print(f"{item['url']}: {item['wordCount']} words") ``` #### cURL ```bash curl -X POST "https://api.apify.com/v2/acts/automation-lab~webpage-to-markdown-converter/runs?token=YOUR_APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "urls": ["https://en.wikipedia.org/wiki/Web_scraping"], "includeImages": false }' ``` ### 🧑‍💻 MCP (Claude Code & Desktop) Use this actor directly from Claude Code or Claude Desktop via the Apify MCP server: **Claude Code** — run in your terminal: ```bash claude mcp add --transport http "https://mcp.apify.com?tools=automation-lab/webpage-to-markdown-converter" ``` **Claude Desktop** — add to your `claude_desktop_config.json`: ```json { "mcpServers": { "apify": { "command": "npx", "args": ["-y", "@apify/actors-mcp-server"], "env": { "APIFY_TOKEN": "YOUR_APIFY_TOKEN", "ACTORS": "automation-lab/webpage-to-markdown-converter" } } } } ``` **Example Claude prompts:** - "Convert https://docs.python.org/3/tutorial/ to Markdown for my knowledge base" - "Fetch these 5 URLs and convert them to clean text for RAG ingestion" - "Extract the article content from this news page without images or links" ### ⚖️ Legality This actor fetches publicly accessible webpages using standard HTTP requests (no browser automation, no captcha bypassing). It is the user's responsibility to comply with the target website's Terms of Service and robots.txt. Content extracted is for the user's own use — ensure compliance with copyright laws when storing or redistributing extracted content. ### ❓ FAQ **Q: Does it work on JavaScript-rendered pages?** A: No — this actor uses HTTP-only requests for speed and cost efficiency. For JavaScript-rendered pages (React/Vue SPAs), you need a browser-based crawler that renders JavaScript before extracting content. **Q: Why does my page return partial content?** A: Some pages serve different content to bots. Try increasing `requestTimeout`. If the page heavily relies on JavaScript for content rendering, it may not work with this actor. **Q: The Markdown has a lot of links/navigation — how do I fix it?** A: Set `includeLinks: false` to strip all hyperlinks, or the actor's Readability engine should remove most navigation. If you're still getting noise, the page may have unusual structure. **Q: Can I convert PDFs or other file types?** A: No — this actor only processes HTML pages. PDF conversion requires a different tool. **Q: How many URLs can I process per run?** A: No hard limit — the actor processes URLs sequentially with a 600-second timeout. For very large batches (1000+ URLs), consider splitting across multiple runs. **Q: Is my data private?** A: Yes — results are stored in your private Apify dataset. No extracted content is shared or retained by the actor developer. ### 🔗 Related actors - [Color Contrast Checker](https://apify.com/automation-lab/color-contrast-checker) — validate WCAG 2.1 AA/AAA color contrast for accessibility - [JSON Schema Generator](https://apify.com/automation-lab/json-schema-generator) — generate JSON schemas from sample data # Actor input Schema ## `urls` (type: `array`): List of webpage URLs to convert to Markdown. Supports any public webpage. ## `includeImages` (type: `boolean`): Include images as Markdown image links (![alt](url)). Disable to get text-only output. ## `includeLinks` (type: `boolean`): Include hyperlinks as Markdown links ([text](url)). Disable for plain text output. ## `maxContentLength` (type: `integer`): Maximum characters in the Markdown output per page. 0 = unlimited. Useful for LLM token limits. ## `requestTimeout` (type: `integer`): HTTP request timeout in seconds. Increase for slow websites. ## `maxRetries` (type: `integer`): Number of retry attempts on network errors or 5xx responses. ## Actor input object example ```json { "urls": [ "https://en.wikipedia.org/wiki/Markdown", "https://en.wikipedia.org/wiki/Artificial_intelligence" ], "includeImages": true, "includeLinks": true, "maxContentLength": 0, "requestTimeout": 30, "maxRetries": 2 } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://en.wikipedia.org/wiki/Markdown", "https://en.wikipedia.org/wiki/Artificial_intelligence" ] }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/webpage-to-markdown-converter").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "https://en.wikipedia.org/wiki/Markdown", "https://en.wikipedia.org/wiki/Artificial_intelligence", ] } # Run the Actor and wait for it to finish run = client.actor("automation-lab/webpage-to-markdown-converter").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://en.wikipedia.org/wiki/Markdown", "https://en.wikipedia.org/wiki/Artificial_intelligence" ] }' | apify call automation-lab/webpage-to-markdown-converter --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/webpage-to-markdown-converter", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Webpage to Markdown Converter", "description": "Convert webpages to clean Markdown for LLM/RAG pipelines. Uses @mozilla/readability to strip ads, navigation, and footers. Outputs structured JSON.", "version": "0.0", "x-build-id": "6LTrvYzU0b0SPchp4" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~webpage-to-markdown-converter/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-webpage-to-markdown-converter", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~webpage-to-markdown-converter/runs": { "post": { "operationId": "runs-sync-automation-lab-webpage-to-markdown-converter", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~webpage-to-markdown-converter/run-sync": { "post": { "operationId": "run-sync-automation-lab-webpage-to-markdown-converter", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "urls" ], "properties": { "urls": { "title": "🌐 URLs to convert", "type": "array", "description": "List of webpage URLs to convert to Markdown. Supports any public webpage.", "items": { "type": "string" } }, "includeImages": { "title": "Include images", "type": "boolean", "description": "Include images as Markdown image links (![alt](url)). Disable to get text-only output.", "default": true }, "includeLinks": { "title": "Include links", "type": "boolean", "description": "Include hyperlinks as Markdown links ([text](url)). Disable for plain text output.", "default": true }, "maxContentLength": { "title": "Max content length (chars)", "minimum": 0, "type": "integer", "description": "Maximum characters in the Markdown output per page. 0 = unlimited. Useful for LLM token limits.", "default": 0 }, "requestTimeout": { "title": "Request timeout (seconds)", "minimum": 5, "maximum": 120, "type": "integer", "description": "HTTP request timeout in seconds. Increase for slow websites.", "default": 30 }, "maxRetries": { "title": "Max retries", "minimum": 0, "maximum": 5, "type": "integer", "description": "Number of retry attempts on network errors or 5xx responses.", "default": 2 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```