Pricing

from $1.00 / 1,000 pages

Try for free

Go to Apify Store

Website to Markdown Crawler for LLM & RAG

Try for free

Crawl any website or docs and get clean Markdown for LLMs, RAG and AI agents. Strips menus, footers and cookie banners, keeps headings, lists, tables, code and links. Follows links or the sitemap. Pay per page.

Pricing

from $1.00 / 1,000 pages

Rating

0.0

(0)

Developer

Oğulcan Katmer

Actor stats

Bookmarked

Total users

Monthly active users

12 days ago

Last modified

Use cases

RAG / knowledge bases: load docs or help centers into a vector database
Custom GPTs and AI agents: give an assistant a company's website as context
LLM fine-tuning datasets: collect clean text from many pages
Content audits and migration: export a whole site as Markdown

Input

Field	What it does
Start URLs	Sites, docs sections or single pages, e.g. `https://docs.example.com/guide/`
Max pages	Total pages to save (default 20)
Max link depth	How many clicks from the start URL to follow (0 = only the start pages)
Also use sitemap.xml	Discover pages from the sitemap too
Stay under the start path	Crawl only `/guide/...` when you start at `/guide/`
Only / Skip URLs matching	Glob filters like `/blog/`
Minimum words per page	Thin pages are skipped and not charged

Output

Field	Description
`url`, `canonicalUrl`	Page address
`title`, `description`, `language`	From the page head
`markdown`	Clean Markdown of the main content
`wordCount`, `depth`	Size and distance from the start URL
`text`, `links`	Optional plain text and link list

Example

{
  "url": "https://docs.example.com/guide/install",
  "title": "Install – Example Docs",
  "wordCount": 412,
  "markdown": "# Install\n\nRun the installer...\n\n## Requirements\n\n- Node.js 20+\n- 2 GB RAM"
}

Pricing

$1.00 per 1,000 pages saved. Pages under the minimum word count, failed pages and sitemap files are free.

Limits

It reads the HTML the server sends. Pages that only render text with JavaScript in the browser may come back empty.
Sites that block bots or need a login are not supported.

Website Content Crawler | Markdown for RAG

xtech/news-source-crawler

Turn public news, blog, and documentation pages into clean Markdown or plain text. Discover pages through sitemaps and feeds, or extract known URLs directly. Export content, metadata, and change-tracking fields for RAG and monitoring, while respecting robots.txt and website boundaries.

Xtech

445

4.8

(3)

Website & Page to Markdown for RAG and LLMs

humble-echidna/page-to-markdown

Turn web pages or whole sites into clean Markdown for RAG and LLMs: main content only, with title, links and word count. Whole-site mode reads the sitemap first, can return only changed pages, and splits pages into heading-based chunks. Respects robots.txt and AI opt-outs. No browser.

Michael Costa

Website Content Crawler - Clean Text, Markdown & RAG Chunks

topapi/website-content-crawler

Crawl any website or URL list for $1.20 per 1,000 pages: one row per page with clean text, Markdown, HTML, metadata, links, headings, tables and RAG chunks. Plain HTTP first, headless Chrome only for JavaScript pages. Sitemaps, llms.txt, PDF/DOCX text, Bing/DuckDuckGo search, change monitor.

Top Api

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

341

5.0

(2)

Website Content Extractor for RAG: Markdown, HTML, Text

nezha/website-content-crawler

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, AI knowledge bases, and internal search. Crawl from start URLs or sitemaps and keep the crawl in scope.

nezha

Website Content Crawler

worshipful_knife/website-content-crawler

Deep crawl websites and extract clean text, Markdown, or HTML for LLMs, RAG, and AI apps. Removes navigation, ads, cookie banners. Supports headless browser & HTTP. Sitemap discovery, URL scoping, file downloads. Feed ChatGPT, LangChain, LlamaIndex, Pinecone. The cheapest content crawler on Apify.

kata Kuri

Website Content Crawler - Markdown, 45 Fields

parseforge/website-content-crawler

Crawls websites from start URLs with depth, domain and URL-pattern limits. 45 fields per page: Markdown and plain text, word count, title and meta tags, headings, Open Graph and JSON-LD, author and dates, emails, phones and social links, detected technologies, links and HTTP status. Ready for LLMs.

ParseForge

189

Website To Markdown

smart_api/website-to-markdown

Convert any webpage into clean, LLM-ready Markdown in seconds — perfect for AI training data, RAG pipelines, and content archiving.

SmartApi

5.0

(1)

AI-Ready Website Crawler

optimus-fulcria/ai-ready-website-crawler

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

Fulcria Labs

Site Crawler: Website → Markdown Corpus for LLM/RAG

boxbox10/site-crawler

Crawl a whole website or docs site and get one clean, LLM-ready Markdown + JSON record per page (title, headings, content, links, token count). Built for RAG ingestion and AI knowledge bases.