Website to Markdown & Text Crawler — AI, RAG & LLM Data 📄

Turn any website into clean Markdown and plain text for AI. This website content crawler crawls an entire site, strips away navigation, headers, footers, ads and scripts, and exports the boilerplate-free main content of every page as Markdown and plain text — ready to feed straight into LLM training sets, RAG pipelines, embeddings, vector databases and AI agents.

Give it one URL — it discovers and extracts every page automatically. No login, no headless browser, one clean row per page.

Looking to scrape a website for an LLM, convert HTML to Markdown, build RAG data, or extract text from a website at scale? That's exactly what this actor does.

---

✨ Key features

• 🕷️ Full-site crawl — start from one URL and follow internal links across the whole domain.
• 📝 Clean Markdown + plain text — main content only, with nav/header/footer/sidebar/scripts removed.
• 🔗 Absolute links & images — relative URLs are rewritten to absolute, so the Markdown is portable.
• 🧠 Built for AI / RAG / LLM — chunk-ready output for embeddings, fine-tuning and retrieval.
• 🏷️ Rich page metadata — title, meta description, H1, language, canonical and word count.
• ⚡ Fast & cheap — pure HTTP, no browser, high concurrency.

💡 Use cases

• RAG & knowledge bases — turn docs, blogs and help centers into clean Markdown chunks for retrieval-augmented generation.
• LLM fine-tuning datasets — collect high-quality text at scale from any set of websites.
• AI agents & chatbots — feed your agent fresh, structured website content.
• Content migration & archiving — export an entire website to Markdown.
• Semantic search & embeddings — generate clean text to embed into a vector database (Pinecone, Weaviate, pgvector, …).

📦 What you get

One row per crawled page:

Field	Description
url	Page URL
title	Page title
metaDescription	Meta description
h1	First H1 heading
lang	Page language
canonical	Canonical URL
wordCount	Word count of the main content
text	Clean main-content text (boilerplate removed)
markdown	The same content converted to Markdown
html	Cleaned main-content HTML (optional)
crawledAt	ISO 8601 timestamp

Example output

{
  "url": "https://docs.example.com/getting-started",
  "title": "Getting Started",
  "metaDescription": "Set up the SDK in 5 minutes.",
  "h1": "Getting Started",
  "wordCount": 812,
  "text": "Getting Started Install the package...",
  "markdown": "# Getting Started\n\nInstall the package...",
  "crawledAt": "2026-05-25T14:13:00.000Z"
}

🚀 How to use it

1. Click Try for free / Start.
2. Paste one or more website URLs into Start URLs.
3. (Optional) Set Max pages to crawl — use 0 to crawl the whole site.
4. (Optional) Toggle Save Markdown, Save plain text, Save HTML.
5. Click Save & Start.
6. Export your dataset as JSON, CSV, Excel or via API, or pull it straight into your AI pipeline.

⚙️ Input

Option	Description	Default
startUrls	Websites to crawl	– (required)
maxPagesToCrawl	Max pages per run (0 = whole site)	1000
saveMarkdown	Include Markdown output	true
saveText	Include plain-text output	true
saveHtml	Include cleaned main-content HTML	false
maxConcurrency	Parallel requests	10

Example input

{
  "startUrls": [{ "url": "https://docs.apify.com" }],
  "maxPagesToCrawl": 2000,
  "saveMarkdown": true,
  "saveText": true
}

🔍 How it works

The crawler follows internal links within the same domain as your Start URLs. For each page it removes scripts, styles, navigation, headers, footers and sidebars, isolates the main content (<main> / <article> / body), rewrites relative links and images to absolute URLs, and exports the result as clean text and Markdown. It's pure HTTP — fast and cheap, with no headless browser.

🧰 Tips & best practices

• Set maxPagesToCrawl to 0 to capture an entire site for a knowledge base.
• Keep saveText and saveMarkdown on for maximum flexibility downstream; turn on saveHtml if you need raw HTML.
• Use the wordCount field to filter out thin pages before embedding.
• Lower maxConcurrency if a site rate-limits you.

❓ FAQ

Does it render JavaScript? No — it parses server-rendered HTML, which keeps runs fast and cheap and works for the large majority of websites and documentation sites.

Is the Markdown clean enough for RAG? Yes — navigation, headers, footers, ads and scripts are stripped, and links/images are absolute, so the output is ready to chunk and embed.

How do I crawl the whole site? Set maxPagesToCrawl to 0.

Can I crawl multiple sites at once? Yes — add several Start URLs.

What formats can I export? JSON, CSV, Excel, HTML and a full REST API.

🔗 Related actors by the same author

• Sitemap to URL Crawler — extract every URL from a sitemap.xml to feed this crawler.
• Website SEO Audit Crawler — on-page SEO audit for every page.
• Website Image & Media Crawler — extract all images and media for multimodal datasets.
• JSON-LD Schema & Meta Tag Extractor — structured data and meta tags from any page.

---

Changelog

• 2026-05-25 — Maintenance & reliability pass: pulled the latest source and rebuilt the Actor on the current base image; build verified.

Last reviewed: 2026-05-25.