Web Page to Markdown & llms.txt Generator

Turn any public web page into clean, LLM-ready Markdown, and generate llms.txt / llms-full.txt for whole websites. Boilerplate (menus, cookie banners, footers, ads, sidebars) is removed, tables become GitHub-flavoured Markdown tables, links are made absolute, and every page comes with title, description, language, headings, word and token counts and optional RAG chunks. $1 per 1,000 pages, $0.02 per llms.txt. Failed pages are free.

What does it do?

• Page → Markdown: fetches each URL with a fast HTTP client (no browser), finds the main content (<main>, <article>, the most specific content container) and converts it to Markdown with headings, lists, code blocks, quotes and GFM tables.
• RAG chunks: optional heading-aware chunks of N tokens with overlap, ready for embeddings.
• llms.txt generator: for each site you list, discovers pages from the sitemap (or the home page links), groups them by section and writes a spec-compliant llms.txt with titles and one-line descriptions. Optionally also llms-full.txt with the Markdown of every page. Files are saved in the run's key-value store and linked from the dataset row.
• Detects whether the site already publishes /llms.txt.
• Respects publishers: robots.txt, and AI opt-outs (noai robots tags, or robots.txt rules that block AI crawlers such as GPTBot, CCBot, Google-Extended, ClaudeBot) are honoured by default.

Why use it?

• No headless browser: fast and inexpensive for server-rendered pages (docs, blogs, wikis, government and company sites).
• Clean output: no navigation noise, no tracking-pixel images, no empty links, stable field names.
• Privacy by design: e-mail addresses and phone numbers in page text are redacted by default.
• Safe: honest user agent, retries with backoff, max 2 requests per site, private-network addresses blocked (including redirects and DNS tricks).

Input

Field	Default	What it does
urls	–	Public pages to convert ([{ "url": "https://..." }], plain strings also work).
llmsTxtSites	–	Websites to build an llms.txt for. A home page covers the whole site; a sub-path such as https://docs.python.org/3/ limits the file to pages under that path.
maxPagesPerSite	25	Pages per site for llms.txt / llms-full.txt (max 500).
generateLlmsFullTxt	false	Also convert every listed page and write llms-full.txt (each page is charged as a page).
includeUrlPattern / excludeUrlPattern	–	Regular expressions to filter discovered site pages.
contentMode	main	main = main content only; full = whole body without scripts/styles.
includeLinks	true	Keep links as [text](url); off = plain text.
includeImages	false	Keep images as .
includeLinkList	false	Also return a deduplicated list of links found in the content.
chunkTokens / chunkOverlapTokens	0 / 0	Chunk size in estimated tokens (0 = no chunking) and overlap.
maxMarkdownChars	200000	Truncate very long pages (flagged with truncated: true).
redactContactDetails	true	Replace e-mail addresses and phone numbers with [email removed] / [phone removed].
respectRobotsTxt	true	Skip (free) pages that robots.txt disallows.
respectAiOptOuts	true	Skip (free) pages whose site opts out of AI use (noai, AI-crawler blocks in robots.txt).
maxConcurrency, requestTimeoutSecs, maxRetries	5, 20, 1	Speed and robustness.
userAgent	WebToMarkdownBot/1.0	Honest bot user agent.

{
  "urls": [{ "url": "https://www.python.org/about/" }],
  "llmsTxtSites": [{ "url": "https://docs.python.org/3/" }],
  "maxPagesPerSite": 20,
  "chunkTokens": 500
}

Output

One row per page (type: "page") and one per site (type: "llms-txt"). Example page (shortened):

{
  "type": "page",
  "url": "https://www.python.org/about/",
  "statusCode": 200,
  "title": "About Python™ | Python.org",
  "description": "The official home of the Python Programming Language",
  "lang": "en",
  "markdown": "# About\n\n## Python is powerful... and fast; plays well with others; runs everywhere; is friendly & easy to learn; is Open.\n\nThese are some of the reasons people who use Python would rather not use anything else...",
  "wordCount": 430,
  "tokensEstimate": 640,
  "headings": ["About", "Python is powerful... and fast; ..."],
  "contentHash": "5f1c…",
  "contactDetailsRedacted": 0,
  "fetchedAt": "2026-10-27T01:00:00.000Z"
}

Example llms.txt row: site, pageCount, sections, discovery (sitemap, home-page-links or sitemap+home-page-links), existingLlmsTxt, the llmsTxt text itself and llmsTxtUrl / llmsFullTxtUrl download links.

Failed pages are listed with error and errorCode (HTTP_404, TIMEOUT, NETWORK_ERROR, NOT_HTML, EMPTY_CONTENT, ROBOTS_DISALLOWED, AI_OPT_OUT, BLOCKED_NON_PUBLIC, INVALID_URL, TOO_MANY_REDIRECTS, NO_PAGES) and are not charged. A SUMMARY record has the run totals.

How much does it cost?

Event	Price
Page converted	$0.001
llms.txt generated (per site)	$0.02

• 1,000 pages ≈ $1. An llms.txt for one site ≈ $0.02; with llms-full.txt of 25 pages ≈ $0.045.
• Apify's tiny run-start fee applies (the default run uses 512 MB).
• Errors, blocked, robots- or AI-opt-out-skipped pages are free.
• Set Maximum cost per run: the Actor stops cleanly before exceeding it and keeps everything done so far.

Use it from code or an AI agent

Available through the Apify API, the JS/Python clients and Apify's MCP server (Claude, ChatGPT, Cursor...). Give an agent a URL, get Markdown back.

curl -X POST "https://api.apify.com/v2/acts/thriving_adder~web-to-markdown-llms-txt/run-sync-get-dataset-items" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" -H 'Content-Type: application/json' -d '{"urls":[{"url":"https://example.com"}]}'

Legal and data notes

• Only public pages. No logins, cookies or paywalls. Localhost and private-network addresses are rejected.
• Copyright stays with the publisher. Converting a page does not give you rights to republish it. Use the output for search, analysis, summaries, RAG over content you are allowed to use, or for llms.txt files of your own sites. Check the site's terms before commercial reuse.
• robots.txt and AI opt-outs are respected by default. You can switch them off only for sites you own or have permission for.
• Personal data: e-mail addresses and phone numbers are redacted by default; the Actor does not collect author profiles or user data. If the content you convert contains personal data, you are the controller under GDPR/PDPA.

Use cases

• RAG and chatbots: feed clean Markdown chunks of your docs, help centre or blog into a vector database.
• llms.txt for your own site: generate a first llms.txt / llms-full.txt so AI assistants can find your key pages, then edit and publish it.
• AI agents: give an agent a URL and get readable text back through the API or Apify's MCP server.
• Content audits and migrations: export page text and headings of a site you manage to compare versions or move CMS.
• Research and summaries of public pages you are allowed to use.

FAQ

Does it render JavaScript? No. Pages that build their content in the browser (some SPAs) return EMPTY_CONTENT (free). Most docs, blogs and company sites work.

Why is my llms.txt short? Only pages found in the sitemap or home page that pass robots.txt and your patterns are listed, up to maxPagesPerSite.

Problems or ideas? Open an issue in the Issues tab with the URL and input. We answer within 2 business days.

Related Actors

By the same publisher, built on the same safe, pay-per-result design:

• Page Metadata, OpenGraph & JSON-LD Extractor: titles, OpenGraph and JSON-LD only, cheaper per page.
• Sitemap URL Extractor & SEO Health Checker: list and health-check every URL of a site.
• Academic Paper Search (OpenAlex, Crossref, arXiv): scholarly papers with abstracts for RAG.