Website & URL to Markdown Crawler + llms.txt Generator

Crawl a website, or convert a list of URLs, into clean Markdown for LLMs, RAG, AI agents and fine-tuning datasets. It keeps the main content (navigation, ads, cookie banners and footers are removed), plus headings, tables, code blocks and links. You also get metadata, token counts, optional RAG chunks and a ready-made llms.txt / llms-full.txt.

It's an HTTP crawler with no headless browser, so it's fast and cheap: about 10 pages/second on documentation sites.

How to use

1. Enter one or more start URLs (a docs site, blog or any website).
2. Choose crawl (follow links) or list (only the given URLs), and set Max pages.
3. Click Start and download clean Markdown for every page, plus an llms.txt file.

Features

• Main-content extraction with Mozilla Readability and a boilerplate-removal fallback, or full-page mode.
• GitHub-flavored Markdown: tables, fenced code blocks with language, lists, absolute links, and optional images.
• Crawl control: depth, max pages, include/exclude globs or regex, subdomains, sitemap.xml seeding and robots.txt compliance.
• De-duplication: redirects, canonical URLs and identical content are skipped and not charged.
• Metadata: title, description, canonical, language, author, published/modified dates, og:image, site name, headings outline, word count and token estimate.
• RAG chunks split on headings and paragraphs, with heading path, token counts and overlap.
• llms.txt + llms-full.txt saved to the key-value store for free, ready to drop on your site or feed to an agent.
• possiblyJsRendered flag for pages that need a browser.

Why this Actor

	This Actor	Typical website-to-Markdown Actors
Price	$1 / 1,000 pages, no start fee	$2.30 to $3 / 1,000 + start fees
llms.txt generation	Yes (free)	No
RAG chunks with heading path	Yes	Rarely
Sitemap seeding + robots.txt	Yes	Varies
Duplicates/redirects charged	No	Often yes

Input example

{ "startUrls": ["https://docs.example.com/"], "maxPages": 500, "includeUrlPatterns": ["https://docs.example.com/**"], "useSitemap": true, "chunking": true }

Output

{ "url": "https://docs.example.com/getting-started", "title": "Getting started", "description": "...", "canonical": "...", "language": "en",
  "markdown": "# Getting started\n\nInstall the SDK...\n\n```bash\nnpm i example\n```", "headings": [{ "level": 2, "text": "Install" }],
  "wordCount": 812, "tokensEstimate": 1350, "depth": 1, "statusCode": 200, "extraction": "readability",
  "chunks": [{ "index": 0, "text": "...", "headingPath": "Getting started > Install", "tokens": 640, "url": "..." }] }

Key-value store: llms.txt (index of pages with titles and descriptions) and llms-full.txt (all page Markdown).

Pricing (pay per event)

$0.001 per page ($1 per 1,000 pages). Failed fetches, non-HTML files, duplicates and pages below Min words are free.

How much does it cost?

You only pay for results, never for compute time:

You get	You pay (Starter/Bronze plan price)
100 pages	$0.10
1,000 pages	$1.00
10,000 pages	$10.00

Apify's free plan includes $5 of usage every month, and free-plan users pay half price on this Actor, so that covers about 10,000 pages at no cost. Paid plans get volume discounts (Silver -10%, Gold -20%). Set a maximum cost per run and the Actor stops cleanly at your limit.

Notes

• Respect each site's terms. robots.txt is honored by default, and each host is limited to 5 requests/second.
• Sites that render content only with JavaScript (empty HTML) are flagged possiblyJsRendered. Use a browser-based crawler for those.

Run it from code (API, Python, JavaScript)

curl -X POST "https://api.apify.com/v2/acts/api-co~website-to-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" -d '{"startUrls": ["https://docs.apify.com/academy/web-scraping-for-beginners"], "maxItems": 50}'

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("api-co/website-to-markdown").call(run_input={"startUrls": ["https://docs.apify.com/academy/web-scraping-for-beginners"], "maxItems": 50})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('api-co/website-to-markdown').call({"startUrls": ["https://docs.apify.com/academy/web-scraping-for-beginners"], "maxItems": 50});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

It also works from AI agents through the Apify MCP server, and with Zapier, Make, n8n, Google Sheets and webhooks. You can schedule it in the Apify Console.

FAQ

How is this different from Website Content Crawler? It's a lightweight HTTP crawler focused on docs and content sites. It's fast and cheap ($1 per 1,000 pages), and it generates llms.txt / llms-full.txt for free. For JavaScript-heavy apps that need a real browser, use a browser-based crawler.

Does it respect robots.txt? Yes. It's sitemap-aware and robots.txt-aware.

What do I get per page? Clean main-content Markdown (no nav or footer), title, description, headings, tables and code blocks, token count and RAG chunks.

Can I feed it straight into a vector database? Yes. Use the chunks field, or connect the dataset to LangChain, LlamaIndex, Pinecone or Qdrant through Apify integrations.

Use cases

• RAG chatbots: turn a documentation site into Markdown chunks for your vector database.
• llms.txt generation: create an llms.txt for your own site so AI assistants can read it.
• AI agents: give an agent clean page content instead of raw HTML.
• Content audits and migrations: export a whole site's text with titles, headings and metadata.