Website to Markdown - Content Extractor for LLM & RAG

What does Website to Markdown do?

Website to Markdown is a fast web page to Markdown converter and website content extractor for LLMs, RAG pipelines and AI agents. Give it a list of URLs, or one start URL to crawl, and it returns the main content of every page as clean Markdown (and optionally plain text): no menus, cookie banners, footers or ads. Each page also comes with its title, meta description, site name, language, published date, H1–H3 headings, word count and, if you want, ready-made RAG chunks.

It uses plain HTTP requests and the open-source trafilatura extraction engine, with no headless browser. That makes it cheap and quick: a 10-page docs crawl finishes in about 20 seconds. robots.txt is always respected.

Why use Website to Markdown?

• LLM-ready output. Markdown keeps headings, lists, tables and code blocks, which is the format language models and embedding pipelines handle best. Boilerplate is removed so you don't pay tokens for navigation menus.
• Bulk URL lists or crawling. Paste 5,000 URLs with crawl depth 0, or give one docs or blog URL and let it follow links within the same folder.
• RAG chunks built in. Set chunkSize (e.g. 1,500 characters) and chunkOverlap and every page gets a chunks array split on paragraph boundaries, ready for your vector database.
• Cheap and fast. No browser means low compute cost and a low price per page. You pay only for pages that returned content. Errors, 404s, robots-blocked and empty pages are free.
• Polite by default. robots.txt rules and crawl-delay are honoured, concurrency is capped, and the user agent identifies the actor.
• Stable schema. The same fields for every site, ready for JSON, CSV, Excel, a webhook or the Apify API.

What data can you extract?

Field	Example	Description
url	https://docs.python.org/3/tutorial/classes.html	The requested URL
loadedUrl	https://docs.python.org/3/tutorial/classes.html	Final URL after redirects
statusCode	200	HTTP status
crawlDepth	1	0 = start URL, 1 = linked from a start URL, …
title	9. Classes	Page title
description	Classes provide a means of bundling data…	Meta / Open Graph description
siteName	Python documentation	Site name from metadata
language	en	<html lang> value
publishedDate	2024-10-15	Publication or last-modified date when the page exposes one
headings	[{"level": 1, "text": "9. Classes"}, …]	H1–H3 outline (navigation and footer headings removed)
markdown	# 9. Classes\n\nClasses provide…	Main content as Markdown
text	9. Classes Classes provide…	Plain text (optional, includePlainText)
wordCount	5456	Words in the extracted Markdown
chunks	[{"index": 0, "text": "…", "chars": 1480}, …]	RAG chunks (optional, chunkSize)
links	["https://docs.python.org/3/library/…", …]	All absolute links on the page (optional, includeLinks)
warning	Very little text: the page may be rendered by JavaScript	Hints about thin pages
error	http_404, blocked_by_robots_txt, timeout	Why a page returned no content (free row)
scrapedAt	2026-10-21T19:30:00Z	Extraction time (UTC)

How to use Website to Markdown (step by step)

1. Open the actor and click Try for free.
2. In Start URLs, paste the pages you want, one per line. You can also upload a text file or a Google Sheet of URLs.
3. Choose Crawl depth: 0 converts only the URLs you gave (bulk mode); 1 or more also follows links found on them, staying within the same site and folder.
4. Set Max pages to cap the run (and the cost).
5. Optional: set RAG chunk size, switch on plain text, links or images, or add include/exclude URL patterns.
6. Click Start. When the run finishes, open the Pages view, or download the dataset as JSON, CSV, Excel or HTML.

Input example

{
  "startUrls": [
    { "url": "https://docs.python.org/3/tutorial/index.html" }
  ],
  "crawlDepth": 1,
  "maxPages": 50,
  "contentMode": "main",
  "chunkSize": 1500,
  "chunkOverlap": 150,
  "excludeUrlPatterns": ["/genindex", "?highlight="]
}

Bulk mode for a URL list (no crawling):

{
  "startUrls": [
    { "url": "https://blog.apify.com/what-is-web-scraping/" },
    { "url": "https://www.gov.uk/browse/tax" }
  ],
  "crawlDepth": 0,
  "maxPages": 1000,
  "includePlainText": true
}

Output example

[
  {
    "url": "https://blog.apify.com/what-is-web-scraping/",
    "loadedUrl": "https://blog.apify.com/what-is-web-scraping/",
    "statusCode": 200,
    "crawlDepth": 0,
    "title": "What is web scraping? Complete guide to web data extraction",
    "description": "Learn what web scraping is, how it works and what it is used for.",
    "siteName": "Apify Blog",
    "language": "en",
    "publishedDate": "2024-10-15",
    "headings": [
      { "level": 1, "text": "What is web scraping?" },
      { "level": 2, "text": "How does web scraping work?" }
    ],
    "markdown": "Web scraping is the process of automatically extracting data from a website. You use a program called a web scraper…",
    "wordCount": 1455,
    "scrapedAt": "2026-10-21T19:31:02Z"
  },
  {
    "url": "https://www.python.org/nonexistent-page",
    "loadedUrl": null,
    "statusCode": 404,
    "crawlDepth": 0,
    "error": "http_404",
    "scrapedAt": "2026-10-21T19:31:01Z"
  }
]

The second row is an error row. It is not charged. Every dataset can be exported to CSV, Excel, JSON, XML or HTML from the run page or through the API.

How much does it cost?

Website to Markdown uses pay-per-event pricing: you pay only for pages that returned content.

Event	Price (Free plan)	When it is charged
Actor start	$0.0005	Once per run
Page extracted	$0.0015 ($1.50 per 1,000)	One page converted to Markdown

Failed pages (404, 5xx, timeouts, DNS errors), pages blocked by robots.txt, non-HTML files and pages without any text are free. Paid Apify plans get Store discounts on the per-page price (Bronze −15%, Silver −30%, Gold and up −45%).

Worked examples (Free plan prices):

• A 200-page documentation site for a RAG chatbot: 200 × $0.0015 + $0.0005 ≈ $0.30.
• 10,000 URLs from a spreadsheet (bulk mode, depth 0): ≈ $15.
• A weekly refresh of a 500-page knowledge base: ≈ $0.75 per run, about $3 per month.
• An AI agent fetching 3 pages per question, 100 questions a day: ≈ $0.45 per day.

The Apify Free plan includes monthly platform credit, which covers many runs like these.

Use cases

• RAG and chatbots. Turn product docs, help centres or policy pages into chunks for a vector database (Pinecone, Qdrant, Weaviate, pgvector) and keep them fresh with a scheduled run.
• LLM context for AI agents. Give an agent a cheap tool that returns the readable content of any URL instead of raw HTML full of scripts.
• Content analysis and SEO. Pull the text, headings and word counts of competitor pages or your own blog to compare topics, length and structure.
• Dataset building. Collect clean article text for classification, summarisation or fine-tuning datasets (check each site's terms for reuse).
• Monitoring. Schedule a daily run on a set of pages and compare the Markdown between runs to spot changes in prices, terms or documentation.
• Migration. Export the content of an old website as Markdown before moving it to a new CMS or static site generator.

Use with AI agents, MCP and the API

Run it from any language with one HTTP call (replace <TOKEN> with your Apify API token):

curl -X POST "https://api.apify.com/v2/acts/teddythebuilder~website-to-markdown/run-sync-get-dataset-items?token=<TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docs.python.org/3/tutorial/classes.html"}],"crawlDepth":0}'

Python with the apify-client package:

from apify_client import ApifyClient

client = ApifyClient("<TOKEN>")
run = client.actor("teddythebuilder/website-to-markdown").call(run_input={
    "startUrls": [{"url": "https://docs.python.org/3/tutorial/"}],
    "crawlDepth": 1,
    "maxPages": 30,
    "chunkSize": 1500,
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    for chunk in page.get("chunks", []):
        print(page["url"], chunk["index"], chunk["chars"])

MCP: the actor works as a tool in the Apify MCP server (https://mcp.apify.com), so Claude, ChatGPT, Cursor and other MCP clients can call it to read web pages as Markdown. Agents get short, clean content, which saves tokens compared with raw HTML.

Integrations

Connect results to Make, Zapier, n8n, LangChain, LlamaIndex, Google Sheets, Airtable, Slack or your own webhook. Apify's integrations run when a run finishes, so a scheduled crawl can refresh a vector database or a spreadsheet automatically.

Data source and legality

The actor downloads public web pages over HTTP, like a browser without JavaScript, and extracts the readable content with trafilatura (Apache-2.0 licence). It reads robots.txt before fetching any page and skips URLs that the site disallows; crawl-delay rules are followed. It does not log in, does not bypass paywalls, CAPTCHAs or bot protection, and does not use proxies to hide.

You are responsible for how you use the content: copyright and site terms still apply to the text you extract, and pages can contain personal data. Use it for analysis, search and AI context on content you are allowed to process.

FAQ

Does it render JavaScript? No. It is a fast HTTP extractor. Server-rendered sites, blogs, news, documentation, e-commerce product pages and most CMS sites work well. Pure single-page apps that show "enable JavaScript" return little text; those rows carry a warning, and pages with no text at all are free.

How is "main content" chosen? trafilatura scores the HTML blocks of the page and keeps the article body, dropping navigation, sidebars, footers, comments and cookie notices. Use contentMode: "fullPage" to keep more (lists, sidebars) or "precise" for the least boilerplate.

Which pages does the crawler follow? Only links on the same site (www and non-www count as the same), and by default only under the start URL's folder. Add includeUrlPatterns / excludeUrlPatterns to fine-tune, or useSitemap to queue URLs from sitemap.xml.

How big can a run be? Set maxPages up to 100,000. The crawler stops before the run timeout and keeps everything it extracted. Raise the timeout and memory for very large sites.

Do I pay for errors? No. 404s, timeouts, robots-blocked URLs, PDFs and other non-HTML files, and pages without text are returned as free rows with an error field.

Can it read PDFs? Not yet; PDF links are skipped. A dedicated PDF text extractor is planned.

Can I get the raw HTML? No. The actor is built for clean text. If you need HTML, use a general-purpose crawler.

I found a page that extracts badly. Open an issue with the URL. We reply within 24 hours. If the actor works well for you, a rating on the Store page helps other users find it.

Related actors

• Website SEO Audit Crawler & Broken Link Checker: titles, meta, canonical, noindex, broken links and sitemap coverage for a whole site.
• Tech Stack Detector: which CMS, analytics and e-commerce tools a website uses.
• Domain WHOIS & DNS Lookup API: registration, DNS, email security and SSL data for domains.
• Hacker News Scraper: HN stories by keyword and "Who is hiring?" jobs.

Changelog and support

• 1.0 (October 2026): first release. Bulk URL and crawl modes, main-content Markdown and text, metadata and headings, RAG chunks, sitemap seeding, robots.txt and crawl-delay support, free error rows.

Questions or feature requests? Open an issue on the actor page. We reply within 24 hours.