Pricing

Pay per event

Website Content Extractor for LLM and RAG

Crawl any website and extract the main readable content of each page as clean markdown - nav, footers, scripts and ads stripped. Built for RAG pipelines and LLM ingestion. One record per page {url, title, markdown, word_count} plus a final site inventory.

Pricing

Pay per event

Rating

0.0

(0)

Developer

Martello Systems

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Website Content Extractor for LLM / RAG

Turn any website into clean markdown for RAG / LLM ingestion. Point it at a domain, and it discovers the site's pages, fetches each one, and extracts only the main readable content - navigation, headers, footers, scripts, cookie banners and ads are stripped out. You get tidy markdown ready to embed, chunk, or feed straight into a vector store.

What it does

Discovers pages - parses sitemap.xml (and any sitemap index it points to, plus sitemaps named in robots.txt). If there's no sitemap, it falls back to a breadth-first crawl of same-domain links from the start URL.
Extracts main content - uses trafilatura to pull the article body as markdown, with a readability-style heuristic fallback. No boilerplate, no markup noise.
Outputs one record per page plus a final site inventory record.

Egress is routed through Apify residential proxies when available, so it keeps working on sites that block datacenter IPs.

Input

Field	Type	Default	Description
`start_url`	string (required)	-	Domain or page to start from, e.g. `https://example.com`
`max_pages`	integer	`25`	Cap on pages fetched and extracted (1-500)
`same_domain_only`	boolean	`true`	Stay on the start URL's registered domain when following links

{
  "start_url": "https://example.com",
  "max_pages": 25,
  "same_domain_only": true
}

Output

One dataset record per page:

{
  "url": "https://example.com/pricing",
  "title": "Pricing - Example",
  "markdown": "# Pricing\n\nSimple, transparent plans...\n\n## Starter\n...",
  "word_count": 412
}

Plus a final inventory record (also stored in the key-value store under SITE_INVENTORY):

{
  "record_type": "inventory",
  "start_url": "https://example.com",
  "domain": "example.com",
  "discovery_method": "sitemap",
  "pages_discovered": 25,
  "pages_extracted": 24,
  "pages_failed": 1,
  "total_words": 9123,
  "extractor": "trafilatura",
  "pages": [{ "url": "...", "title": "...", "word_count": 412 }]
}

Typical uses

Build a knowledge base from a company or docs site for a RAG chatbot.
Snapshot a competitor's site as markdown for LLM analysis.
Bulk-ingest content for embeddings / semantic search.

Site Crawler: Website → Markdown Corpus for LLM/RAG

boxbox10/site-crawler

Crawl a whole website or docs site and get one clean, LLM-ready Markdown + JSON record per page (title, headings, content, links, token count). Built for RAG ingestion and AI knowledge bases.

Marvin Eguilos

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

Web to Markdown — AI-Ready Text from Any URL

wsgcjj/web-to-markdown

Convert any web page URL to clean Markdown format. Perfect for LLM training data, RAG pipelines, and AI content processing. Extracts main content, strips ads/nav/footers.

陈俊杰

Website Content Crawler — AI & RAG Ready

santamaria-automations/website-content-crawler

Crawl any website and extract clean Markdown and plain text optimized for AI ingestion, RAG pipelines, and LLM context. Readability-style main content extraction removes ads, navs, and footers. Configurable depth, concurrency, and page limits. Pay-per-page.

NanoScrape

Website to Markdown — LLM & RAG Content Exporter

sturdydata/website-markdown-exporter

Crawl any website and get one clean Markdown document per page — ready for RAG pipelines, vector databases, LLM fine-tuning, or docs migration. Boilerplate (nav, footers, cookie banners) stripped, main content auto-detected, sitemap-seeded crawling, robots.txt respected. HARD page caps and flat p...

Sturdy Data

LLM-Ready Web Extractor

phantom_horse/my-actor-1

Turn any web page into clean, LLM-ready Markdown. Strips scripts, nav, and page chrome, then converts the main content to tidy Markdown with title, meta description, and token counts. Perfect for AI prompts and RAG ingestion pipelines.

NATNAEL FIKRE

Web Content Extractor - Clean Markdown for AI

geekguymj/web-content-extractor

Extract clean, readable markdown content from any web page. Removes navigation, ads, footers, and boilerplate — outputs structured markdown optimized for LLM training, RAG pipelines, and AI agents. Pay-per-event pricing. $0.002/page.

Matthew Jenkins

Website to Markdown

cool_ya/website-to-markdown

Convert any web page into clean, LLM-ready Markdown. Strips nav, ads and boilerplate and returns the main article text plus title, description and word count. Perfect for RAG and AI pipelines.

Y A

AI Web Extractor: URL → Clean Markdown + JSON for LLM/RAG

boxbox10/ai-web-extractor

Turn any URL into clean, LLM-ready Markdown + structured JSON (title, headings, main content, links, metadata, token count). Perfect for RAG pipelines, AI agents, and LLM context.

Marvin Eguilos

Website to Markdown - Clean LLM-Ready Content

ambitious_door/web-to-markdown

Convert any webpage into clean markdown stripped of navigation, ads, and boilerplate. Perfect for RAG pipelines, LLM context, and content extraction. Token counts included.