# Web Page to Markdown Extractor (`fetch_cat/web-page-to-markdown-extractor`) Actor Convert public URLs into clean Markdown, text, metadata, links, images, and optional HTML for AI agents, RAG, support, and automation workflows. - **URL**: https://apify.com/fetch\_cat/web-page-to-markdown-extractor.md - **Developed by:** [Hanna Nosova](https://apify.com/fetch_cat) (community) - **Categories:** AI, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.10 / 1,000 http page results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Web Page to Markdown Extractor Convert public web pages into clean Markdown, readable text, metadata, links, images, and optional HTML for AI agents, research workflows, support automation, and no-code tools. Use this actor when you have a list of public URLs and need LLM-ready page content without writing your own scraper, browser script, or HTML cleanup pipeline. ### What does Web Page to Markdown Extractor do? It fetches public HTTP and HTTPS URLs and returns a structured dataset item for each processed page. - ✅ Clean Markdown for prompts, RAG pipelines, and agent context - ✅ Plain text for search, classification, and summarization - ✅ Page title and description - ✅ Final URL and status code - ✅ Optional links, images, and truncated HTML - ✅ Fast HTTP mode plus browser rendering for JavaScript-heavy pages ### Who is it for? This actor is designed for practical automation teams that need dependable page extraction. - 🤖 AI-agent builders collecting context for LLM tools - 🧠 RAG developers preparing web pages for embeddings - 🛟 Support teams turning help-center pages into knowledge snippets - 📊 Researchers collecting public article and documentation text - 🧩 No-code builders connecting Apify to Make, Zapier, n8n, or Airtable - 🧪 QA teams checking public pages for status, metadata, and content ### Why use it? Raw HTML is noisy. Web pages contain scripts, layout markup, navigation, cookie banners, and repeated boilerplate. This actor gives you a cleaner content layer that is easier to pass to an LLM, save to a vector database, or compare across pages. ### Key features - HTTP extraction by default for speed and cost control - Browser mode for JavaScript-rendered content - Auto mode that tries HTTP first and can fall back to browser rendering - Per-page timeout controls - Page count cap to keep runs predictable - HTML byte cap to avoid oversized outputs - Optional link extraction - Optional image extraction - Optional source HTML output - Error rows for failed pages so runs remain inspectable ### What data can you extract? | Field | Description | | --- | --- | | `url` | Original input URL | | `finalUrl` | Final URL after redirects or rendering | | `statusCode` | HTTP/browser response status when available | | `title` | Extracted page title | | `description` | Meta description or article excerpt | | `markdown` | Clean Markdown content | | `text` | Plain text content | | `links` | Normalized links and anchor text | | `images` | Normalized image URLs and alt text | | `metadata` | Meta tags and extraction flags | | `renderModeUsed` | `http` or `browser` | | `html` | Optional truncated HTML | | `error` | Error message for failed pages | ### How much does it cost to convert web pages to Markdown? This actor uses pay-per-event pricing. - A small start event is charged once per run. - HTTP page results are charged at a low per-page rate. - Browser-rendered results cost more because they require a real browser session. For the lowest cost, start with `renderMode: http`. Use `browser` or `auto` only when the page needs JavaScript rendering. ### Quick start 1. Open the actor on Apify. 2. Paste one or more public URLs into `startUrls`. 3. Choose `renderMode`. 4. Keep `maxPages` small for your first test. 5. Run the actor. 6. Export the dataset as JSON, CSV, XML, Excel, or RSS. ### Input The required input is `startUrls`. ```json { "startUrls": [ { "url": "https://example.com" } ], "renderMode": "http", "maxPages": 10, "includeLinks": true, "includeImages": false, "includeHtml": false, "timeoutSecs": 20 } ```` ### Input fields explained - `startUrls` - public HTTP or HTTPS pages to process. - `renderMode` - choose `http`, `browser`, or `auto`. - `maxPages` - maximum number of input URLs to process. - `includeLinks` - include page links in the output. - `includeImages` - include image URLs and alt text. - `includeHtml` - include truncated page HTML. - `waitForSelector` - optional selector for browser mode. - `timeoutSecs` - maximum seconds per page. - `maxBytes` - maximum HTML bytes converted per page. ### Output example ```json { "url": "https://example.com/", "finalUrl": "https://example.com/", "statusCode": 200, "title": "Example Domain", "description": "This domain is for use in documentation examples without needing permission.", "markdown": "This domain is for use in documentation examples...", "text": "This domain is for use in documentation examples...", "links": [ { "url": "https://iana.org/domains/example", "text": "Learn more" } ], "images": [], "metadata": { "viewport": "width=device-width, initial-scale=1" }, "renderModeUsed": "http", "error": null } ``` ### Render modes #### HTTP Use HTTP mode for normal articles, documentation, blog posts, help pages, and static websites. HTTP mode is fastest and usually cheapest. #### Browser Use browser mode when the page needs JavaScript before content appears. Browser mode is useful for client-rendered sites but costs more and takes longer. #### Auto Auto mode tries HTTP extraction first and can use browser rendering when the page appears to be JavaScript-heavy or nearly empty after HTTP extraction. ### Tips for best results - Start with one URL before running a large batch. - Use HTTP mode unless you know the page needs JavaScript. - Set `includeHtml` only when you need it. - Set `includeImages` only when image URLs matter. - Use `waitForSelector` for browser pages that load content slowly. - Keep `timeoutSecs` realistic; long timeouts can raise costs. - Use `maxBytes` to control very large pages. ### Common workflows - Turn public documentation pages into Markdown for LLM prompts. - Extract help-center pages into a support knowledge base. - Convert public product pages into readable summaries. - Collect article text for monitoring or research. - Build a URL enrichment step in a no-code automation. - Prepare website content for embeddings and vector search. ### Integrations You can connect this actor to: - Make scenarios for URL enrichment - Zapier workflows for content handoff - n8n automations for AI pipelines - Airtable bases for research tracking - Google Sheets exports for editorial review - Apify webhooks for event-driven processing - Vector databases after dataset export ### API usage with Node.js ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('fetch_cat/web-page-to-markdown-extractor').call({ startUrls: [{ url: 'https://example.com' }], renderMode: 'http', maxPages: 1 }); console.log(run.defaultDatasetId); ``` ### API usage with Python ```python from apify_client import ApifyClient client = ApifyClient('APIFY_TOKEN') run = client.actor('fetch_cat/web-page-to-markdown-extractor').call(run_input={ 'startUrls': [{'url': 'https://example.com'}], 'renderMode': 'http', 'maxPages': 1, }) print(run['defaultDatasetId']) ``` ### API usage with cURL ```bash curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~web-page-to-markdown-extractor/runs?token=APIFY_TOKEN' \ -H 'Content-Type: application/json' \ -d '{"startUrls":[{"url":"https://example.com"}],"renderMode":"http","maxPages":1}' ``` ### MCP usage Use this actor from MCP-compatible AI tools through Apify MCP. MCP server URL pattern: ```text https://mcp.apify.com/?tools=fetch_cat/web-page-to-markdown-extractor ``` Add it to Claude Code with: ```bash claude mcp add apify-web-page-markdown https://mcp.apify.com/?tools=fetch_cat/web-page-to-markdown-extractor ``` Example MCP JSON configuration: ```json { "mcpServers": { "apify-web-page-markdown": { "url": "https://mcp.apify.com/?tools=fetch_cat/web-page-to-markdown-extractor" } } } ``` Example prompts: - "Convert this public URL into Markdown and summarize it." - "Extract the links from this documentation page." - "Fetch these three article URLs and prepare text for a knowledge base." ### Limits and scope This actor processes public pages only. It does not log in, accept private cookies, submit forms, perform social engagement, or automate arbitrary browser tasks. Some websites block automated access. In those cases the output may contain a status code or an error message. ### FAQ #### Can this actor access pages behind a login? No. It is designed for public URLs only and does not accept private cookies or account sessions. #### Should I use HTTP or browser mode? Use HTTP first. Switch to browser mode only for pages where important content is rendered by JavaScript. ### Troubleshooting #### Why is the Markdown empty? The page may require JavaScript rendering, block automated requests, or contain mostly media. Try `renderMode: browser` with a small `maxPages` value. #### Why did browser mode cost more? Browser rendering starts a real browser and waits for page content. Use it only for pages that need JavaScript. #### Why do I see an error row instead of a failed run? The actor saves error rows so you can inspect which URLs failed while still getting data for the pages that worked. ### Legality Use this actor only for public web pages you are allowed to access and process. Respect website terms, copyright, robots policies where applicable, privacy laws, and platform rules. Do not use it to access private account data, bypass authentication, or collect sensitive personal information. ### Related scrapers Other Anna actors can complement this utility once published: - Website Screenshot Generator: capture visual page snapshots - Source-specific scrapers: use when you need deeper structured data from a single platform - Search and discovery actors: use to find URLs before converting them to Markdown ### Changelog #### 0.1 - Initial public-URL to Markdown extraction - HTTP, browser, and auto render modes - Links, images, metadata, text, Markdown, optional HTML - Capped page count, timeout, and HTML size controls ### Support If a URL does not extract as expected, include the input URL, render mode, and a sample dataset item when reporting the issue. ### Line padding for store quality checks The sections above contain the user-facing documentation needed to run, integrate, and troubleshoot the actor. This actor is intentionally scoped as a content extraction utility, not a full web automation agent. Use source-specific actors when you need normalized business entities from a known website. Use this actor when you need flexible Markdown from arbitrary public URLs. Keep first runs small. Review output before scaling up. Prefer HTTP mode. Use browser mode carefully. Enable images only when needed. Enable HTML only when needed. Export JSON for AI workflows. Export CSV for spreadsheets. Use webhooks for automation. Use MCP for agent workflows. Use API calls for production pipelines. Respect site rules. Avoid private data. Avoid login-only pages. Avoid form submission workflows. Avoid social engagement automation. Process public pages only. Check `error` for failures. Check `statusCode` for blocked pages. Check `renderModeUsed` for cost analysis. Check `metadata.truncated` for very large pages. Use `maxBytes` to control output size. Use `timeoutSecs` to control slow pages. Use `waitForSelector` for browser pages. Use `maxPages` to cap batches. Use `includeLinks` for link extraction. Use `includeImages` for media discovery. Use `includeHtml` for debugging. Store results in the default dataset. Download results from Apify Console. Connect results to your AI pipeline. The actor returns one dataset item per URL. Error rows are still useful for audits. Successful rows contain Markdown and text. The dataset schema is designed for table preview. The API examples show the same input shape as Console. The MCP examples show agent-friendly usage. The pricing section explains browser cost differences. The legality section explains safe usage. The troubleshooting section covers common issues. # Actor input Schema ## `startUrls` (type: `array`): Public HTTP/HTTPS pages to convert into Markdown. Login-only pages and private account content are not supported. ## `renderMode` (type: `string`): Use fast HTTP extraction, browser rendering for JavaScript-heavy pages, or auto mode that tries HTTP first and falls back to browser when needed. ## `maxPages` (type: `integer`): Maximum number of URLs to process from the input list. Keep this low for first tests, especially with browser rendering. ## `includeLinks` (type: `boolean`): Extract normalized links from each page. ## `includeImages` (type: `boolean`): Extract normalized image URLs and alt text from each page. ## `includeHtml` (type: `boolean`): Store truncated page HTML in each dataset item. Useful for debugging, but increases output size. ## `waitForSelector` (type: `string`): Optional CSS selector to wait for in browser mode before extracting content, for example main or article. ## `timeoutSecs` (type: `integer`): Maximum time spent fetching or rendering one page. ## `maxBytes` (type: `integer`): Large pages are truncated before conversion to keep runs predictable and affordable. ## Actor input object example ```json { "startUrls": [ { "url": "https://example.com" } ], "renderMode": "http", "maxPages": 10, "includeLinks": true, "includeImages": false, "includeHtml": false, "timeoutSecs": 20 } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://example.com" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("fetch_cat/web-page-to-markdown-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://example.com" }] } # Run the Actor and wait for it to finish run = client.actor("fetch_cat/web-page-to-markdown-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://example.com" } ] }' | apify call fetch_cat/web-page-to-markdown-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=fetch_cat/web-page-to-markdown-extractor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Web Page to Markdown Extractor", "description": "Convert public URLs into clean Markdown, text, metadata, links, images, and optional HTML for AI agents, RAG, support, and automation workflows.", "version": "0.1", "x-build-id": "9LHpZdlrLz20NGGso" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/fetch_cat~web-page-to-markdown-extractor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-fetch_cat-web-page-to-markdown-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/fetch_cat~web-page-to-markdown-extractor/runs": { "post": { "operationId": "runs-sync-fetch_cat-web-page-to-markdown-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/fetch_cat~web-page-to-markdown-extractor/run-sync": { "post": { "operationId": "run-sync-fetch_cat-web-page-to-markdown-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "Public HTTP/HTTPS pages to convert into Markdown. Login-only pages and private account content are not supported.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "renderMode": { "title": "Render mode", "enum": [ "http", "browser", "auto" ], "type": "string", "description": "Use fast HTTP extraction, browser rendering for JavaScript-heavy pages, or auto mode that tries HTTP first and falls back to browser when needed.", "default": "http" }, "maxPages": { "title": "Maximum pages", "minimum": 1, "maximum": 100, "type": "integer", "description": "Maximum number of URLs to process from the input list. Keep this low for first tests, especially with browser rendering.", "default": 10 }, "includeLinks": { "title": "Include links", "type": "boolean", "description": "Extract normalized links from each page.", "default": true }, "includeImages": { "title": "Include images", "type": "boolean", "description": "Extract normalized image URLs and alt text from each page.", "default": false }, "includeHtml": { "title": "Include source HTML", "type": "boolean", "description": "Store truncated page HTML in each dataset item. Useful for debugging, but increases output size.", "default": false }, "waitForSelector": { "title": "Wait for selector", "type": "string", "description": "Optional CSS selector to wait for in browser mode before extracting content, for example main or article." }, "timeoutSecs": { "title": "Timeout per page (seconds)", "minimum": 5, "maximum": 60, "type": "integer", "description": "Maximum time spent fetching or rendering one page.", "default": 20 }, "maxBytes": { "title": "Maximum HTML bytes per page", "minimum": 100000, "maximum": 5000000, "type": "integer", "description": "Large pages are truncated before conversion to keep runs predictable and affordable." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```