Pricing

from $4.00 / 1,000 results

Try for free

Go to Apify Store

Website Content Crawler & Markdown Extractor

Try for free

Crawls a website and extracts clean, boilerplate-free text/Markdown from every page -- ready for RAG pipelines, LLM context, or content research. Respects robots.txt.

Pricing

from $4.00 / 1,000 results

Rating

0.0

(0)

Developer

victor

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

Features

Website to Markdown converter — strips nav, footers, ads and cookie banners, keeps the actual content
RAG data extraction — the exact clean-text format retrieval pipelines and LLM ingestion need, no HTML noise
Respects robots.txt — checked per URL before fetching, not an afterthought
Same-domain crawling with configurable depth and page limits, so you can't accidentally crawl the entire internet from one start URL
Works on any public website — no login, no API key, no per-platform setup

Input

{
  "startUrls": ["https://example.com/"],
  "maxPagesPerDomain": 20,
  "maxDepth": 2,
  "outputFormat": "markdown"
}

Output

{
  "url": "https://example.com/blog/post-1",
  "title": "Post title",
  "content": "# Post title\n\nClean article text in Markdown...",
  "wordCount": 842,
  "depth": 1
}

Who this is for

Teams building a RAG knowledge base from their own docs or a client's website
Anyone feeding a website into an LLM's context and tired of copy-pasting HTML cruft
Content researchers and competitive analysts who need many pages of a site as clean text, fast

Pricing

Pay per use — no subscription:

$0.01 per run
$0.004 per page crawled

More tools from this developer

Website to Markdown – Clean LLM & RAG Content Extractor

dataquarry/website-to-markdown

Convert any public web page to clean, LLM-ready Markdown with metadata — by URL, a list of URLs, or a whole-site crawl. Strips nav/ads/boilerplate, keeps headings/lists/tables/code. Respects robots.txt. No API key.

Daniel Brenner

Website Content Crawler

crawlerbros/website-content-crawler

Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.

Crawler Bros

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

Website to Markdown API & Crawler – RAG/LLM

signal_lab/website-to-markdown-crawler

Website-to-Markdown API and crawler for RAG and LLM datasets. Extract clean Markdown, SEO metadata, links, JSON-LD, sitemaps, and robots.txt-compliant pages.

Signal Lab

Website to Markdown - Clean LLM-Ready Content

ambitious_door/web-to-markdown

Convert any webpage into clean markdown stripped of navigation, ads, and boilerplate. Perfect for RAG pipelines, LLM context, and content extraction. Token counts included.

C. K.

Website Content Crawler - Markdown, 45 Fields

excitable_pickle/web-markdown-rag-crawler

Crawls websites from starting URLs and returns each page's full text content as Markdown and cleaned HTML, ready for AI and LLM pipelines.

Pavan

5.0

Website to Markdown Crawler for LLM & RAG

logiover/website-text-markdown-crawler

Crawl any website to clean Markdown and plain text for LLM training and RAG. HTML to Markdown, no API or login. Export website text to CSV or JSON.

Logiover

Website Content Crawler - Markdown & Text for LLM / RAG

pear_fight/website-content-crawler-markdown-text-for-llm-rag

Crawl any website and extract clean article text and Markdown, ready to feed into LLMs, ChatGPT, vector databases and RAG pipelines. Removes navigation, ads and boilerplate. Configurable crawl depth and page limits. Export to JSON, CSV, Excel.

Harald

Website to Markdown for RAG & LLMs

hereditary_model/website-to-markdown

Crawls a website and converts every page into clean, LLM-ready Markdown for RAG pipelines, vector databases, and AI agents. Removes nav, ads, and boilerplate. Predictable pricing: $0.004 per page converted.

Aaron Marxsen

Website Content Crawler – Markdown for AI, Emails & Contacts

locomint/website-crawler-content-contacts

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.