Pricing

from $0.70 / 1,000 page crawleds

Try for free

Go to Apify Store

Website Content Crawler → Clean Text & Markdown for AI

Try for free

🔥 Up to 30% OFF! Crawl any website → clean text + Markdown for AI/RAG, plus 40+ metadata fields: SEO tags, Open Graph, headings outline, reading time, links, images and JSON-LD structured data. No API key, no login. Feed LLMs, RAG & agents. ⚡

Pricing

from $0.70 / 1,000 page crawleds

Rating

0.0

(0)

Developer

Bruno

Actor stats

Bookmarked

Total users

Monthly active users

10 days ago

Last modified

What it actually does (nothing invented)

Crawls a website from your Start URLs, following links to the depth you choose (same-domain by default).
Extracts the main readable content of each page using Mozilla Readability (the same engine behind Firefox Reader View).
Returns both clean plain text and clean Markdown (headings, lists, links and code blocks preserved).
Gives you page metadata: title, description, author (byline), site name, published time, word count.
No API key, no login. You only pay per page crawled.

Output (per page)

{
  "url": "https://example.com/blog/post",
  "title": "How it works",
  "description": "A short summary of the page",
  "byline": "Jane Doe",
  "siteName": "Example",
  "publishedTime": "2026-02-01T00:00:00.000Z",
  "wordCount": 812,
  "charCount": 5120,
  "text": "Clean main text of the page...",
  "markdown": "# How it works\n\nClean **Markdown** of the page...",
  "crawlDepth": 1
}

Great for

📚 RAG / knowledge bases — ingest docs, help centers and blogs as clean Markdown.
🤖 LLM context — feed pages to ChatGPT/Claude without the HTML noise.
🧠 AI agents — let an agent read websites as structured text.
📝 Content & research — archive and analyze site content at scale.

Why choose this Actor

✅ Clean Markdown out of the box — not raw HTML; ready to paste into an LLM.
✅ Pay only per page — no subscription, no minimum, no API key.
✅ Simple — paste URLs, set depth, run.
✅ Tunable — depth, max pages, same-domain, min word count, optional raw HTML, optional residential proxy for tougher sites.

Limits (honest)

It fetches the server-rendered HTML (fast & cheap). Sites that render their content only via JavaScript may return little — enable the residential-proxy option for blocking sites, and for heavy JS SPAs expect reduced content.
Content extraction is automatic; unusual page layouts may extract more or less than a human would.

⭐ Enjoying this Actor?

If it saved you time, please leave an honest review — it genuinely helps. Found a bug or want a feature? Open an Issue and we'll jump on it fast.

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

4.7K

4.9

(7)

Website Content Crawler - Clean Text, Markdown & RAG Chunks

topapi/website-content-crawler

Crawl any website or URL list for $1.20 per 1,000 pages: one row per page with clean text, Markdown, HTML, metadata, links, headings, tables and RAG chunks. Plain HTTP first, headless Chrome only for JavaScript pages. Sitemaps, llms.txt, PDF/DOCX text, Bing/DuckDuckGo search, change monitor.

Top Api

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

338

5.0

(2)

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

Website Content Crawler – Markdown for AI, Emails & Contacts

locomint/website-crawler-content-contacts

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.

Locomint

Website Content Crawler - Markdown, 45 Fields

parseforge/website-content-crawler

Crawls websites from start URLs with depth, domain and URL-pattern limits. 45 fields per page: Markdown and plain text, word count, title and meta tags, headings, Open Graph and JSON-LD, author and dates, emails, phones and social links, detected technologies, links and HTTP status. Ready for LLMs.

ParseForge

176

Website Content Crawler — Text, Titles & Metadata

hipersoft/website-content-crawler

Extract clean, readable content from any list of websites: page title, meta description, headings, main body text, word count and link/image counts. Optional same-domain crawl. Bulk-ready. Great for LLM/RAG ingestion, content audits and research.

hiper soft

Website Content Extractor for RAG: Markdown, HTML, Text

nezha/website-content-crawler

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, AI knowledge bases, and internal search. Crawl from start URLs or sitemaps and keep the crawl in scope.

nezha

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

575

3.7

(3)

Website Content Crawler — Markdown for LLM & RAG

brilliant_gum/web-content-crawler

Deep-crawl websites and extract LLM-ready Markdown with OG tags, JSON-LD, author, dates, token estimates, native RAG chunking, language filtering, content-hash dedup, and per-page error reporting. Enforced timeouts. Zero silent failures.