Website Content to Markdown & llms.txt Crawler - AI & RAG Ready

A lightweight, high-velocity web scraper built on Crawlee for Python that transforms websites into clean, token-efficient Markdown and automatically generates standards-compliant /llms.txt and /llms-full.txt manifests directly in your Apify Key-Value Store.

---

Why Choose This Actor?

Traditional website crawlers rely on heavy headless browsers (Playwright/Puppeteer with Chromium) that allocate 4,096 MB to 8,192 MB of RAM, costing $3.20+ per compute hour and suffering frequent memory leaks and navigation timeouts during deep crawls.

Website Content to Markdown & llms.txt Crawler solves this with an asynchronous HTTP and BeautifulSoup architecture:

• 90% Compute Cost Reduction: Runs efficiently within a 512 MB - 1,024 MB RAM footprint.
• 10x - 20x Throughput Advantage: Crawls 15 to 25 pages per second on standard broadband connections.
• Automated /llms.txt & /llms-full.txt Generation: Built according to the llmstxt.org standard for zero-friction AI ingestion.
• Intelligent Boilerplate Pruning: Strips navigation menus, cookie banners, tracking scripts, and footers, keeping vector databases clean.
• Token Accounting: Calculates accurate character, word, and estimated LLM token counts per page for RAG context budgeting.
• 100% Windows & Linux Compatible: Defensive UTF-8 handling and pure ASCII logging ensure zero encoding crashes.

---

Performance & Capability Comparison

Metric / Capability	Legacy Headless Browser Crawlers	Basic Web Scrapers	Website Content to llms.txt Crawler (This Actor)
Default RAM Allocation	8,192 MB (Heavy Browser)	1,024 MB	512 MB - 1,024 MB
Throughput (pages/sec)	1 - 2 pages/sec	5 - 10 pages/sec	15 - 25 pages/sec
Pricing Model	Platform CUs (High)	$3.00 / 1,000 pages PPE	Standard Platform CUs (Ultra-Low)
/llms.txt Generation	No (Read-only seed)	No	Yes (Automatic in Key-Value Store)
/llms-full.txt Single File	No	No	Yes (Concatenated RAG context)
Boilerplate Stripping	Manual regex/selectors	Minimal	Automated + Configurable Selectors
LLM Token Estimation	No	No	Yes (~4 chars/token heuristic)
Headings Hierarchy Tree	Flat list	None	Structured AST (Levels 1 - 6)

---

Input Parameters

Parameter	Type	Default	Description
startUrls	Array	[{"url": "https://crawlee.dev"}]	Seed URLs from which crawling commences.
maxPages	Integer	20	Maximum number of pages to crawl (1 - 1,000).
maxDepth	Integer	3	Maximum link depth from start URLs (0 - 10).
includeLlmsTxtGeneration	Boolean	true	Synthesize /llms.txt and /llms-full.txt in Key-Value Store.
removeElementsCssSelector	String	Standard boilerplate selector	Elements stripped from DOM before Markdown conversion.
proxyConfiguration	Object	{ "useApifyProxy": true }	Proxy settings to prevent IP bans and rate limiting.

---

Outputs

1. Default Dataset

Each crawled page emits a record with:

{
  "url": "https://crawlee.dev/python/docs/quickstart",
  "title": "Quick Start | Crawlee for Python",
  "description": "Learn how to build your first scraper using Crawlee in Python.",
  "markdown": "# Quick Start\n\nInstall Crawlee using pip:\n\n```bash\npip install crawlee[beautifulsoup]\n```\n\n## First Steps\n\n...",
  "tokenCountEstimate": 420,
  "wordCount": 315,
  "characterCount": 1680,
  "headings": [
    { "level": 1, "text": "Quick Start" },
    { "level": 2, "text": "First Steps" }
  ],
  "metadata": {
    "canonical": "https://crawlee.dev/python/docs/quickstart",
    "author": "Apify",
    "language": "en",
    "ogTitle": "Quick Start | Crawlee for Python",
    "ogDescription": "Learn how to build your first scraper using Crawlee in Python.",
    "ogImage": "https://crawlee.dev/img/og-image.png",
    "ogType": "article"
  },
  "crawledAt": "2026-09-07T11:00:00.000Z"
}

2. Key-Value Store Artifacts

• LLMS_TXT: A standard /llms.txt file linking to the key documentation and article pages with concise summaries.
• LLMS_FULL_TXT: A consolidated single-file Markdown document containing all crawled pages clearly delimited, ready for direct upload into:
  • Cursor AI (.cursorrules or context)
  • Claude Projects
  • OpenAI Custom GPTs
  • Google NotebookLM
  • LangChain / LlamaIndex Vector Stores

---

⚡ How to Use in 3 Simple Steps

1. Set Seed URLs: Enter your target website or documentation URL (e.g., https://crawlee.dev).
2. Configure Scope: Choose maxPages (e.g., 20 for a quick scan, or up to 1000 for an entire documentation site) and keep includeLlmsTxtGeneration checked.
3. Run & Ingest: Click Start. When complete, download your clean Markdown dataset or copy LLMS_TXT and LLMS_FULL_TXT directly from the Key-Value Store tab for your AI workflows.

---

💡 Use Cases & AI Integrations

• Cursor & AI Code Editors: Drop LLMS_FULL_TXT directly into .cursorrules or editor context for instant documentation retrieval.
• Claude Projects & OpenAI Custom GPTs: Upload the clean, consolidated Markdown artifact directly as a persistent knowledge base file.
• RAG & Vector Pipelines: Ingest boilerplate-free Markdown into LangChain, LlamaIndex, Pinecone, or Weaviate with pre-calculated token counts.
• Google NotebookLM: Feed clean, noise-free website content into NotebookLM for instant Q&A and podcast generation.

---

❓ FAQ & Support

Does this crawler require a heavy headless browser?

No. By using an asynchronous HTTP and BeautifulSoup architecture, this Actor achieves 15–25 pages per second with only 512 MB–1,024 MB RAM, cutting compute costs by up to 90% compared to heavy browser crawlers.

Where can I find the generated /llms.txt manifests?

Open the Storage > Key-Value Store tab of your run:

• LLMS_TXT: Standard summary manifest conforming to the llmstxt.org standard.
• LLMS_FULL_TXT: Complete, concatenated full-text Markdown document ready for direct LLM context upload.

Need custom scrapers or feature enhancements?

Submit an inquiry or bug report on the Issues tab in the Apify Console. We respond promptly and actively maintain this Actor.