`, the most specific content container) and converts it to Markdown with headings, lists, code blocks, quotes and GFM tables. - **RAG chunks:** optional heading-aware chunks of N tokens with overlap, ready for embeddings. - **llms.txt generator:** for each site you list, discovers pages from the sitemap (or the home page links), groups them by section and writes a spec-compliant [llms.txt](https://llmstxt.org) with titles and one-line descriptions. Optionally also `llms-full.txt` with the Markdown of every page. Files are saved in the run's key-value store and linked from the dataset row. - Detects whether the site already publishes `/llms.txt`. - **Respects publishers:** robots.txt, and AI opt-outs (`noai` robots tags, or robots.txt rules that block AI crawlers such as GPTBot, CCBot, Google-Extended, ClaudeBot) are honoured by default. ### Why use it? - **No headless browser:** fast and inexpensive for server-rendered pages (docs, blogs, wikis, government and company sites). - **Clean output:** no navigation noise, no tracking-pixel images, no empty links, stable field names. - **Privacy by design:** e-mail addresses and phone numbers in page text are redacted by default. - **Safe:** honest user agent, retries with backoff, max 2 requests per site, private-network addresses blocked (including redirects and DNS tricks). ### Input | Field | Default | What it does | |---|---|---| | `urls` | – | Public pages to convert (`[{ "url": "https://..." }]`, plain strings also work). | | `llmsTxtSites` | – | Websites to build an llms.txt for. A home page covers the whole site; a sub-path such as `https://docs.python.org/3/` limits the file to pages under that path. | | `maxPagesPerSite` | `25` | Pages per site for llms.txt / llms-full.txt (max 500). | | `generateLlmsFullTxt` | `false` | Also convert every listed page and write `llms-full.txt` (each page is charged as a page). | | `includeUrlPattern` / `excludeUrlPattern` | – | Regular expressions to filter discovered site pages. | | `contentMode` | `main` | `main` = main content only; `full` = whole body without scripts/styles. | | `includeLinks` | `true` | Keep links as `[text](url)`; off = plain text. | | `includeImages` | `false` | Keep images as `![alt](src)`. | | `includeLinkList` | `false` | Also return a deduplicated list of links found in the content. | | `chunkTokens` / `chunkOverlapTokens` | `0` / `0` | Chunk size in estimated tokens (0 = no chunking) and overlap. | | `maxMarkdownChars` | `200000` | Truncate very long pages (flagged with `truncated: true`). | | `redactContactDetails` | `true` | Replace e-mail addresses and phone numbers with `[email removed]` / `[phone removed]`. | | `respectRobotsTxt` | `true` | Skip (free) pages that robots.txt disallows. | | `respectAiOptOuts` | `true` | Skip (free) pages whose site opts out of AI use (`noai`, AI-crawler blocks in robots.txt). | | `maxConcurrency`, `requestTimeoutSecs`, `maxRetries` | `5`, `20`, `1` | Speed and robustness. | | `userAgent` | WebToMarkdownBot/1.0 | Honest bot user agent. | ```json { "urls": [{ "url": "https://www.python.org/about/" }], "llmsTxtSites": [{ "url": "https://docs.python.org/3/" }], "maxPagesPerSite": 20, "chunkTokens": 500 } ``` ### Output One row per page (`type: "page"`) and one per site (`type: "llms-txt"`). Example page (shortened): ```json { "type": "page", "url": "https://www.python.org/about/", "statusCode": 200, "title": "About Python™ | Python.org", "description": "The official home of the Python Programming Language", "lang": "en", "markdown": "# About\n\n## Python is powerful... and fast; plays well with others; runs everywhere; is friendly & easy to learn; is Open.\n\nThese are some of the reasons people who use Python would rather not use anything else...", "wordCount": 430, "tokensEstimate": 640, "headings": ["About", "Python is powerful... and fast; ..."], "contentHash": "5f1c…", "contactDetailsRedacted": 0, "fetchedAt": "2026-10-27T01:00:00.000Z" } ``` Example llms.txt row: `site`, `pageCount`, `sections`, `discovery` (`sitemap`, `home-page-links` or `sitemap+home-page-links`), `existingLlmsTxt`, the `llmsTxt` text itself and `llmsTxtUrl` / `llmsFullTxtUrl` download links. Failed pages are listed with `error` and `errorCode` (`HTTP_404`, `TIMEOUT`, `NETWORK_ERROR`, `NOT_HTML`, `EMPTY_CONTENT`, `ROBOTS_DISALLOWED`, `AI_OPT_OUT`, `BLOCKED_NON_PUBLIC`, `INVALID_URL`, `TOO_MANY_REDIRECTS`, `NO_PAGES`) and are **not charged**. A `SUMMARY` record has the run totals. ### How much does it cost? | Event | Price | |---|---| | Page converted | $0.001 | | llms.txt generated (per site) | $0.02 | - 1,000 pages ≈ **$1**. An llms.txt for one site ≈ **$0.02**; with `llms-full.txt` of 25 pages ≈ $0.045. - Apify's tiny run-start fee applies (the default run uses 512 MB). - Errors, blocked, robots- or AI-opt-out-skipped pages are free. - Set **Maximum cost per run**: the Actor stops cleanly before exceeding it and keeps everything done so far. ### Use it from code or an AI agent Available through the Apify API, the JS/Python clients and **Apify's MCP server** (Claude, ChatGPT, Cursor...). Give an agent a URL, get Markdown back. ```bash curl -X POST "https://api.apify.com/v2/acts/thriving_adder~web-to-markdown-llms-txt/run-sync-get-dataset-items" \ -H "Authorization: Bearer YOUR_APIFY_TOKEN" -H 'Content-Type: application/json' -d '{"urls":[{"url":"https://example.com"}]}' ``` ### Legal and data notes - **Only public pages.** No logins, cookies or paywalls. Localhost and private-network addresses are rejected. - **Copyright stays with the publisher.** Converting a page does not give you rights to republish it. Use the output for search, analysis, summaries, RAG over content you are allowed to use, or for llms.txt files of **your own** sites. Check the site's terms before commercial reuse. - **robots.txt and AI opt-outs are respected by default.** You can switch them off only for sites you own or have permission for. - **Personal data:** e-mail addresses and phone numbers are redacted by default; the Actor does not collect author profiles or user data. If the content you convert contains personal data, you are the controller under GDPR/PDPA. ### Use cases - **RAG and chatbots:** feed clean Markdown chunks of your docs, help centre or blog into a vector database. - **llms.txt for your own site:** generate a first `llms.txt` / `llms-full.txt` so AI assistants can find your key pages, then edit and publish it. - **AI agents:** give an agent a URL and get readable text back through the API or Apify's MCP server. - **Content audits and migrations:** export page text and headings of a site you manage to compare versions or move CMS. - **Research and summaries** of public pages you are allowed to use. ### FAQ **Does it render JavaScript?** No. Pages that build their content in the browser (some SPAs) return `EMPTY_CONTENT` (free). Most docs, blogs and company sites work. **Why is my llms.txt short?** Only pages found in the sitemap or home page that pass robots.txt and your patterns are listed, up to `maxPagesPerSite`. **Problems or ideas?** Open an issue in the Issues tab with the URL and input. We answer within 2 business days. ### Related Actors By the same publisher, built on the same safe, pay-per-result design: - [Page Metadata, OpenGraph & JSON-LD Extractor](https://apify.com/thriving_adder/page-metadata-extractor): titles, OpenGraph and JSON-LD only, cheaper per page. - [Sitemap URL Extractor & SEO Health Checker](https://apify.com/thriving_adder/sitemap-url-health): list and health-check every URL of a site. - [Academic Paper Search (OpenAlex, Crossref, arXiv)](https://apify.com/thriving_adder/academic-research-papers): scholarly papers with abstracts for RAG. # Changelog This Actor's version history is a separate document: https://apify.com/thriving_adder/web-to-markdown-llms-txt/changelog.md # Actor input Schema ## `urls` (type: `array`): Public http(s) pages to convert to Markdown (max 50,000 per run). Each item is {"url": "https://..."}; plain strings and URLs without a scheme are accepted. Charged per converted page (page-converted). Localhost/private-network URLs are rejected and not charged. ## `llmsTxtSites` (type: `array`): Site home pages (e.g. https://docs.example.com) to generate an llms.txt for. Pages are found in the site's sitemap (or home-page links), each page's title and description are read, and the file is grouped into sections by URL path. Charged once per generated file (llms-txt-generated). ## `maxPagesPerSite` (type: `integer`): How many pages to read and list per site (1-500). The most important (shallowest) URLs are picked first. Included in the llms.txt price. ## `generateLlmsFullTxt` (type: `boolean`): Also build llms-full.txt (the full Markdown of every listed page in one file). Each included page is then also returned as a page row and charged as page-converted. ## `includeUrlPattern` (type: `string`): Optional regular expression: only sitemap URLs matching it are considered for llms.txt (e.g. /docs/). ## `excludeUrlPattern` (type: `string`): Optional regular expression: sitemap URLs matching it are ignored (e.g. /(tag|author|page)/). ## `contentMode` (type: `string`): main: find the main content (main/article/role=main) and drop navigation, headers, footers, sidebars and cookie/consent banners. full: convert the whole (scripts and forms still removed). ## `includeLinks` (type: `boolean`): Keep hyperlinks as \[text]\(absolute URL). Off: plain text only (fewer tokens). ## `includeImages` (type: `boolean`): Keep images as !\[alt]\(absolute URL). Off by default to save tokens. ## `includeLinkList` (type: `boolean`): Add a links array (absolute URLs found in the content, max 500) to every page row, e.g. for an agent that wants to follow links. ## `chunkTokens` (type: `integer`): If > 0, also split each page's Markdown into chunks of about this many tokens on heading/paragraph boundaries (chunks array with heading path). 0 = no chunks. ## `chunkOverlapTokens` (type: `integer`): Tokens repeated from the end of one chunk at the start of the next. ## `maxMarkdownChars` (type: `integer`): Markdown longer than this is cut and flagged truncated=true. ## `redactContactDetails` (type: `boolean`): Remove e-mail addresses, international phone numbers and mailto:/tel: links from the Markdown (privacy by default). Each row reports contactDetailsRedacted. ## `respectRobotsTxt` (type: `boolean`): Skip URLs that robots.txt disallows for our user agent (WebToMarkdownBot) and honour Crawl-delay (capped at 10 s). Skipped URLs are free. ## `respectAiOptOuts` (type: `boolean`): Also skip pages whose robots.txt blocks common AI crawlers (GPTBot, CCBot, Google-Extended, ClaudeBot, PerplexityBot...) or that send noai in X-Robots-Tag or meta robots. Recommended: the site owner has asked not to be used for AI. Skipped pages are free. ## `maxConcurrency` (type: `integer`): Parallel page requests (1-20). At most 2 requests run at the same time against any one host. ## `requestTimeoutSecs` (type: `integer`): Timeout per HTTP request, in seconds. ## `maxRetries` (type: `integer`): Retries for timeouts, network errors and HTTP 408/429/5xx (with backoff; Retry-After honoured up to 30 s). Retries are free. ## `userAgent` (type: `string`): User-Agent header sent to sites. The default identifies this Actor honestly so site owners can recognise and control it in robots.txt. ## Actor input object example ```json { "urls": [ { "url": "https://www.python.org/about/" }, { "url": "https://en.wikipedia.org/wiki/Markdown" } ], "llmsTxtSites": [], "maxPagesPerSite": 25, "generateLlmsFullTxt": false, "contentMode": "main", "includeLinks": true, "includeImages": false, "includeLinkList": false, "chunkTokens": 0, "chunkOverlapTokens": 0, "maxMarkdownChars": 200000, "redactContactDetails": true, "respectRobotsTxt": true, "respectAiOptOuts": true, "maxConcurrency": 5, "requestTimeoutSecs": 20, "maxRetries": 1, "userAgent": "Mozilla/5.0 (compatible; WebToMarkdownBot/1.0; +https://apify.com/thriving_adder/web-to-markdown-llms-txt)" } ``` # Actor output Schema ## `results` (type: `string`): One row per page (type=page, with markdown) and one row per site (type=llms-txt, with the llms.txt text and download links). ## `files` (type: `string`): Key-value store with -llms.txt and -llms-full.txt files (text/plain). ## `summary` (type: `string`): Counts of converted, failed and skipped pages and error codes. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ { "url": "https://www.python.org/about/" }, { "url": "https://en.wikipedia.org/wiki/Markdown" } ], "llmsTxtSites": [] }; // Run the Actor and wait for it to finish const run = await client.actor("thriving_adder/web-to-markdown-llms-txt").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ { "url": "https://www.python.org/about/" }, { "url": "https://en.wikipedia.org/wiki/Markdown" }, ], "llmsTxtSites": [], } # Run the Actor and wait for it to finish run = client.actor("thriving_adder/web-to-markdown-llms-txt").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ { "url": "https://www.python.org/about/" }, { "url": "https://en.wikipedia.org/wiki/Markdown" } ], "llmsTxtSites": [] }' | apify call thriving_adder/web-to-markdown-llms-txt --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,thriving_adder/web-to-markdown-llms-txt" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/7eaIx9Moy4zzctnz2/builds/SP3ySuAq0fi9x3rbC/openapi.json