Pricing

from $5.00 / 1,000 page extracteds

Try for free

Go to Apify Store

Website Content Extractor for LLM and RAG

Try for free

Crawl any website and extract the main readable content of each page as clean markdown - nav, footers, scripts and ads stripped. Built for RAG pipelines and LLM ingestion. One record per page {url, title, markdown, word_count} plus a final site inventory.

Pricing

from $5.00 / 1,000 page extracteds

Rating

0.0

(0)

Developer

Martello Systems

Actor stats

Bookmarked

Total users

Monthly active users

a month ago

Last modified

Website Content Extractor for LLM / RAG

Turn any website into clean markdown for RAG / LLM ingestion. Point it at a domain, and it discovers the site's pages, fetches each one, and extracts only the main readable content - navigation, headers, footers, scripts, cookie banners and ads are stripped out. You get tidy markdown ready to embed, chunk, or feed straight into a vector store.

What it does

Discovers pages - parses sitemap.xml (and any sitemap index it points to, plus sitemaps named in robots.txt). If there's no sitemap, it falls back to a breadth-first crawl of same-domain links from the start URL.
Extracts main content - uses trafilatura to pull the article body as markdown, with a readability-style heuristic fallback. No boilerplate, no markup noise.
Outputs one record per page plus a final site inventory record.

Egress is routed through Apify residential proxies when available, so it keeps working on sites that block datacenter IPs.

Input

Field	Type	Default	Description
`start_url`	string (required)	-	Domain or page to start from, e.g. `https://example.com`
`max_pages`	integer	`25`	Cap on pages fetched and extracted (1-500)
`same_domain_only`	boolean	`true`	Stay on the start URL's registered domain when following links

{
  "start_url": "https://example.com",
  "max_pages": 25,
  "same_domain_only": true
}

Output

One dataset record per page:

{
  "url": "https://example.com/pricing",
  "title": "Pricing - Example",
  "markdown": "# Pricing\n\nSimple, transparent plans...\n\n## Starter\n...",
  "word_count": 412
}

Plus a final inventory record (also stored in the key-value store under SITE_INVENTORY):

{
  "record_type": "inventory",
  "start_url": "https://example.com",
  "domain": "example.com",
  "discovery_method": "sitemap",
  "pages_discovered": 25,
  "pages_extracted": 24,
  "pages_failed": 1,
  "total_words": 9123,
  "extractor": "trafilatura",
  "pages": [{ "url": "...", "title": "...", "word_count": 412 }]
}

Typical uses

Build a knowledge base from a company or docs site for a RAG chatbot.
Snapshot a competitor's site as markdown for LLM analysis.
Bulk-ingest content for embeddings / semantic search.

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.

Ryan Clinton

AI-Ready Website Crawler

optimus-fulcria/ai-ready-website-crawler

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

Fulcria Labs

Website Content Crawler — AI & RAG Ready

santamaria-automations/website-content-crawler

Crawl any website and extract clean Markdown and plain text optimized for AI ingestion, RAG pipelines, and LLM context. Readability-style main content extraction removes ads, navs, and footers. Configurable depth, concurrency, and page limits. Pay-per-page.

NanoScrape

Company & Website Lead Enricher

martellosystems/company-lead-enricher

Input a list of domains. For each, crawl the homepage plus contact/about pages and extract emails, phone numbers, social profile links, physical address, and mailto/tel hrefs. Deterministic (regex + HTML parse), no LLM. One enriched record per domain.

Martello Systems

Website Tech Stack Detector

martellosystems/website-tech-stack-detector

Give it a list of domains. For each, it fetches the homepage and deterministically detects the technology stack from response headers, HTML, scripts, and JSON-LD. No LLM. One record per domain, technologies grouped by category.

Martello Systems

AI Website Content Extractor

scrapeai/ai-website-content-extractor

Crawl website pages, strip noise, and convert the main content to clean Markdown for RAG/LLM training.

ScrapeAI

5.0

(2)

Website to AI / RAG Data Extractor

scrapyx/website-rag-extractor

Turns any website into a clean, embedding-ready corpus. Strips navigation, footers and cookie banners, converts the real content to markdown, and splits it into overlapping chunks that each carry their own URL, title and metadata. Crawl by URL list, sitemap or link graph.

Ibnu Adzim

Clean Content Extractor for RAG & AI Agents

hadster/clean-content-extractor-for-rag-ai-agents

Turn any article into clean markdown for LLMs. Renders JS-heavy pages, strips ads and navigation, pay only for successful extractions. Best for blog posts, news articles, and documentation pages — not homepage or listing pages.

Hadi Fadlallah

WordPress Content Scraper

bindler/wordpress-content-scraper

Scrape posts, pages and content from any WordPress site via the REST API. Clean plain text with authors, categories and tags, ready for LLM training and RAG.

Neil Sangwaiya

Website Content to Markdown & llms.txt Crawler - AI & RAG Ready

abi4/website-content-llmstxt-crawler

High-performance, lightweight asynchronous crawler converting web pages into clean Markdown, computing token estimates, and generating standard /llms.txt manifests for AI and RAG ingestion.