# Website to Markdown for AI - Fast, Cheap, RAG-Ready Crawler (`dottti/web-to-markdown`) Actor Turn URLs or whole websites into clean Markdown for LLMs and RAG. Main content only, tables, code, links. Title, language, dates, word and token counts, optional chunks. Plain HTTP first, browser only for JavaScript pages. PDFs included. Failed pages are free. - **URL**: https://apify.com/dottti/web-to-markdown.md - **Developed by:** [Dottti](https://apify.com/dottti) (community) - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 pages This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Website to Markdown for AI Turn web pages, PDFs or whole websites into clean Markdown for LLMs, RAG pipelines, vector databases and AI agents. You give it page URLs, start URLs to crawl, or both. You get one row per page with: - the main content as Markdown: headings, lists, tables, code blocks with their language, links and image alt text - menus, footers, cookie banners, share bars, "related stories" and popups removed - metadata: title, description, canonical URL, language, author, published and modified dates, site name, preview image - word count and token count (o200k tokenizer, used by GPT-4o and newer) - optional chunks for RAG, with chunk ids, token counts and the heading path of each chunk Pages are fetched with plain HTTP, which is fast and cheap. A headless browser is used only for pages that need JavaScript to show their content, and those pages are priced separately. ### Who it is for - Teams that feed websites, documentation or news into a chatbot, search index or RAG system - AI agent builders who need "read this URL" as a clean, short Markdown text - Anyone who wants to keep a knowledge base in sync with a website (monitor mode) Works with any language. Arabic, Chinese and other non-Latin sites are decoded correctly, including old charsets such as windows-1256, and link URLs stay readable in the Markdown. ### Input Single pages (no crawling): ```json { "urls": [ "https://www.bbc.com/news/articles/cwzd6q630vjeo", "https://arxiv.org/pdf/1706.03762" ] } ``` Crawl a documentation section into 512-token chunks: ```json { "startUrls": [{ "url": "https://docs.python.org/3/tutorial/index.html" }], "maxPages": 100, "maxCrawlDepth": 3, "chunkSize": 512, "chunkOverlap": 64 } ``` Main options: | Option | What it does | | --- | --- | | Page URLs | Pages to convert. Links are not followed. | | Start URLs | Sites or sections to crawl. Same site only, inside the start URL's folder by default. | | Max pages per start URL | Stops each crawl after this many saved pages. | | Max link depth | How many clicks from the start URL. | | Use sitemaps | Also finds pages from sitemap.xml. They are used after the linked pages. | | Only crawl / Skip URLs matching | URL patterns such as `https://example.com/blog/**` or `**/tag/**`. | | Render JavaScript | `auto` (browser only when needed), `never`, `always`. | | Chunk size / overlap | Token-based chunks for RAG. 0 = off. | | Main content CSS selector | Force the content area, for example `article`. | | Keep images / Keep link URLs | Turn off to save tokens. | | Respect robots.txt | On by default, including Crawl-delay. | | Only new or changed pages | Monitor mode for scheduled runs. | ### Output A saved page (real run, Markdown shortened): ```json { "url": "https://www.bbc.com/news/articles/cwzd6q630vjeo", "loadedUrl": "https://www.bbc.com/news/articles/cwzd6q630vjeo", "status": "ok", "httpStatus": 200, "title": "Eric Bauza, voice of Bugs Bunny and Daffy Duck, dies aged 46", "description": "Bauza had voiced a number of characters in this summer's animated hit Coyote vs. Acme.", "canonicalUrl": "https://www.bbc.com/news/articles/cwzd6q630vjeo", "language": "en-GB", "author": "Roland Hughes", "publishedAt": "2026-10-26T06:16:15.893Z", "modifiedAt": "2026-10-26T06:16:15.893Z", "siteName": "BBC News", "imageUrl": "https://ichef.bbci.co.uk/news/1024/branded_news/2ba1/live/0257f500-d100-11f1-9d25-c3fafb2ab4de.jpg", "wordCount": 294, "tokenCount": 579, "charCount": 2259, "markdown": "# Eric Bauza, voice of Bugs Bunny and Daffy Duck, dies aged 46\n\n![AFP via Getty Images A man in a dark suit with a close-cropped goatee smiles at a premiere](https://ichef.bbci.co.uk/news/480/cpsprodpb/2ba1/live/0257f500-d100-11f1-9d25-c3fafb2ab4de.jpg.webp)AFP via Getty Images\n\nEric Bauza, the voice artist behind Looney Tunes characters including Bugs Bunny, Daffy Duck and Porky Pig, has died aged 46. ...", "fetchedWith": "http", "extraction": "readability", "depth": 0, "startUrl": null, "referrerUrl": null, "contentHash": "2d6a1e37aeadcad01d2aa4e8311688c9e7da4ec1", "crawledAt": "2026-10-26T12:10:27.654Z", "chunks": [ { "chunkId": "b82a8d99b288-0000", "chunkIndex": 0, "text": "# Eric Bauza, voice of Bugs Bunny and Daffy Duck, dies aged 46\n\n...", "tokenCount": 168, "headingPath": "Eric Bauza, voice of Bugs Bunny and Daffy Duck, dies aged 46" } ] } ``` `fetchedWith` is `http`, `browser`, `pdf` or `text`. `chunks` appears only when Chunk size is above 0. Chunk ids are stable for the same URL, so you can upsert them into a vector database. A page that could not be saved gets a free status row: ```json { "url": "https://docs.python.org/3/nope.html", "status": "not_found", "error": "HTTP 404", "httpStatus": 404, "loadedUrl": "https://docs.python.org/3/nope.html", "depth": 0, "startUrl": null, "referrerUrl": null, "charged": false, "crawledAt": "2026-10-26T12:10:27.657Z" } ``` Status values: `ok` (paid), and free ones: `not_found`, `http_error`, `login_required`, `blocked`, `network_error`, `fetch_failed`, `robots_disallowed`, `needs_javascript`, `empty`, `duplicate_content`, `pdf_no_text`, `pdf_failed`, `skipped_pdf`, `skipped_not_html`, `too_large`, `render_failed`, `invalid_input`, `skipped_cost_limit`. The run always ends as SUCCEEDED. The key-value store record `RUN_SUMMARY` lists pages saved, status counts, each crawl's scope, timings, requests and bytes. ### How JavaScript pages are handled With Render JavaScript = auto, every page is first fetched with plain HTTP. The browser is used only when: - the page is an empty app shell (an empty `#root`, `#app` or `#__next`, a "please enable JavaScript" notice, or almost no text and many scripts), or - the site refuses plain requests but may serve a browser. The browser version is used, and charged as a browser-rendered page, only when it shows clearly more text than the plain version. Images, fonts, video and styles are not downloaded. Captchas are never solved: a page that still shows a bot check gets a free `blocked` row. ### PDF links PDF files (up to 8 MB, first 300 pages) are converted to Markdown text with `` markers. Scanned PDFs have no text layer and get a free `pdf_no_text` row. OCR is not included. ### Monitor mode (only new or changed pages) Turn on "Only new or changed pages" and schedule the actor (Apify Console > Schedules, for example daily). Each run compares the Markdown of every page with the previous run and saves only new or changed pages, with `changeStatus` set to `new` or `changed`. Unchanged pages are not saved and not charged. Use a different Monitor name for each schedule. In monitor mode, unchanged pages do not count toward Max pages, so each run can also discover pages it has not seen before. ### Pricing Pay per saved page. Platform usage is included in the price. | Event | Price | | --- | --- | | Page (plain HTTP, PDF or text file) | $1.00 per 1,000 pages | | Browser-rendered page (only pages that need JavaScript) | $3.00 per 1,000 pages | Status rows (errors, blocked, skipped, duplicates, robots.txt) are free. Unchanged pages in monitor mode are free. Set a maximum cost per run in the run options and the actor stops cleanly when it is reached. ### Limits - Public pages only. No login, no cookies, no captcha solving. - Sites that block automated visitors with captchas return `blocked` rows. - Pages larger than 8 MB are skipped. - Token counts use the o200k tokenizer. Other model families count slightly differently. - Main content detection is automatic. For unusual layouts, set Main content CSS selector. - Default memory is 1024 MB. Plain HTTP pages work at 512 MB; JavaScript pages need 1024 MB or more (one browser tab below 2 GB, two at 2 GB, three at 4 GB). ### Tips - Skip old documentation versions with patterns such as `**/docs/3.*/**`. - For the fewest tokens, turn off Keep images and Keep link URLs. - Put Page URLs and Start URLs in the same run; each crawl has its own page limit. # Actor input Schema ## `urls` (type: `array`): Single pages to convert, one per line. Each URL gives one row. Links on these pages are not followed. PDF links are read too. Example: https://en.wikipedia.org/wiki/Markdown ## `startUrls` (type: `array`): Websites or sections to crawl. The crawler follows links on the same site, stays inside the start URL's folder (see below) and stops at Max pages per start URL. Example: https://crawlee.dev/js/docs/quick-start crawls /js/docs/. ## `maxPages` (type: `integer`): Stop each crawl after this many saved pages. Failed, skipped and duplicate pages do not count. Pages in Page URLs are not limited by this. ## `maxCrawlDepth` (type: `integer`): How many clicks away from the start URL to go. 0 = only the start URL (plus sitemap pages if Use sitemaps is on), 1 = the start URL and the pages it links to. ## `useSitemaps` (type: `boolean`): Also read the site's sitemap.xml (and sitemaps listed in robots.txt) to find pages that are not linked. Sitemap pages are used after the followed links, and only those inside the crawl scope. ## `stayInsideStartPath` (type: `boolean`): On: a start URL of https://example.com/docs/intro only crawls pages under https://example.com/docs/. Off: the whole site. ## `includeSubdomains` (type: `boolean`): Also crawl subdomains of the start URL's site (blog.example.com when starting from example.com). ## `includeUrlGlobs` (type: `array`): Optional URL patterns. When set, the crawler only saves and follows matching URLs (start URLs are always fetched). \* matches inside one path part, \*\* matches anything. Example: https://example.com/blog/\*\*. A pattern without https:// matches anywhere in the URL, for example /guide/. ## `excludeUrlGlobs` (type: `array`): Optional URL patterns to skip, for example **/tag/** or \*\*?page=\*. Same syntax as above. ## `renderJs` (type: `string`): auto (recommended): plain HTTP for every page, and a headless browser only for pages that come back as an empty JavaScript app shell or refuse plain requests. Those pages are charged as browser-rendered pages. never: plain HTTP only (cheapest). always: every page in the browser (slow, charged as browser-rendered). ## `jsWaitSecs` (type: `integer`): Longest wait for a JavaScript page to finish loading. The wait ends earlier once the page text stops changing. ## `chunkSize` (type: `integer`): 0 = no chunks. Above 0, each row also gets a chunks array: pieces of about this many tokens (o200k tokenizer) that keep paragraphs, lists, tables and code blocks whole where possible. Each chunk has an id, its heading path and its token count. Typical values: 256 to 1024. ## `chunkOverlap` (type: `integer`): Tokens from the end of one chunk repeated at the start of the next, so context is not lost at the cut. At most half the chunk size. ## `contentSelector` (type: `string`): Optional. A CSS selector for the main content, for example article or .docs-content. When it matches, only that part of the page is converted. Leave empty for automatic detection. ## `removeSelectors` (type: `string`): Optional. CSS selectors of extra parts to drop before conversion, comma separated. Menus, footers, cookie banners and share bars are already removed. ## `includeImages` (type: `boolean`): Keep images as !\[alt text]\(image URL). Off removes them (fewer tokens). ## `includeLinksInMarkdown` (type: `boolean`): Keep links as [text](URL). Off keeps only the link text (fewer tokens). ## `includeLinkList` (type: `boolean`): Adds a links array (URL and anchor text) with every link found on the page, menus included. Up to 1,000 per page. ## `handlePdfs` (type: `boolean`): Extract the text of PDF files (up to 8 MB, 300 pages) as Markdown. Scanned PDFs without a text layer return a free status row. Off: PDFs are skipped. ## `respectRobotsTxt` (type: `boolean`): Skip pages the site's robots.txt does not allow, and honour its Crawl-delay (up to 10 seconds). Skipped pages return a free status row. ## `skipDuplicateContent` (type: `boolean`): When two URLs give exactly the same Markdown, save the first and return a free status row for the others. ## `onlyChanged` (type: `boolean`): Remembers each page's content between runs and saves only pages that are new or whose text changed. Unchanged pages are not saved and not charged. Schedule the actor to keep a knowledge base fresh. The first run saves everything. ## `stateName` (type: `string`): Separate memory for each monitor, so two schedules do not share history. Only used with monitor mode. ## `maxConcurrency` (type: `integer`): Pages fetched at the same time across all sites. ## `maxConcurrencyPerSite` (type: `integer`): Polite limit for one website. The actor also slows down by itself when a site answers 429 Too Many Requests. ## `requestTimeoutSecs` (type: `integer`): Time limit for one plain HTTP request. ## `maxRetries` (type: `integer`): Retries after network errors, 429 and 5xx answers, with growing pauses. ## `proxyConfiguration` (type: `object`): Optional. By default pages load without a proxy and a site that blocks us moves to Apify datacenter proxy. Set a proxy here to use it for every request. ## `autoProxyFallback` (type: `boolean`): When a site blocks us (403 or 429 with every header set), move that site to Apify datacenter proxy. ## `allowResidentialFallback` (type: `boolean`): Also try residential proxy for sites that block datacenter IPs. Off by default because residential traffic is slow; the price per page does not change. ## Actor input object example ```json { "urls": [ "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" ], "startUrls": [ { "url": "https://crawlee.dev/js/docs/quick-start" } ], "maxPages": 10, "maxCrawlDepth": 2, "useSitemaps": true, "stayInsideStartPath": true, "includeSubdomains": false, "renderJs": "auto", "jsWaitSecs": 8, "chunkSize": 512, "chunkOverlap": 64, "includeImages": true, "includeLinksInMarkdown": true, "includeLinkList": false, "handlePdfs": true, "respectRobotsTxt": true, "skipDuplicateContent": true, "onlyChanged": false, "stateName": "default", "maxConcurrency": 10, "maxConcurrencyPerSite": 4, "requestTimeoutSecs": 30, "maxRetries": 2, "autoProxyFallback": true, "allowResidentialFallback": false } ``` # Actor output Schema ## `results` (type: `string`): All rows. Paid rows have status ok; filter on status to drop the free status rows. ## `runSummary` (type: `string`): Pages saved, status counts, crawl scope per start URL, timings, requests and bytes. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" ], "startUrls": [ { "url": "https://crawlee.dev/js/docs/quick-start" } ], "maxPages": 10, "chunkSize": 512 }; // Run the Actor and wait for it to finish const run = await client.actor("dottti/web-to-markdown").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": ["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"], "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }], "maxPages": 10, "chunkSize": 512, } # Run the Actor and wait for it to finish run = client.actor("dottti/web-to-markdown").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" ], "startUrls": [ { "url": "https://crawlee.dev/js/docs/quick-start" } ], "maxPages": 10, "chunkSize": 512 }' | apify call dottti/web-to-markdown --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,dottti/web-to-markdown" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/pGh4cdKCtZc0DjseX/builds/f7adnuepz7nxfEj0r/openapi.json