Website Content Extractor for RAG: Markdown, HTML, Text

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, internal search, and AI knowledge bases.

What this Actor does

Most teams do not need "a crawler." They need a faster way to turn a website into usable content for:

• embeddings and chunking pipelines
• internal search and AI assistants
• help center or docs ingestion
• markdown, text, or HTML exports that do not require manual copy-paste

This Actor helps you go from a website URL or sitemap to a structured content dataset with cleaned page text, markdown, HTML, headings, crawl metadata, and optional clean HTML records in key-value store.

Quick start

1. Paste a docs site, help center, blog, or website section into Website, Docs, or Help Center URLs.
2. Keep maxPages: 3, crawlMode: auto, and outputFormat: markdown for the first run.
3. Click Run.
4. Check the dataset and OUTPUT_SUMMARY, then raise maxPages for a larger crawl.

auto mode tries sitemap discovery first because it is usually faster for docs sites and help centers. If no crawlable sitemap pages are found, it falls back to following links from the pasted start URLs.

Use cases

Docs site to RAG
Crawl developer docs, product docs, or API docs, then export markdown or clean HTML ready for chunking, embeddings, and retrieval.

Help center to AI support
Extract support articles as clean text or markdown for internal search, support copilots, and FAQ assistants.

Website to knowledge base
Capture blog posts, product pages, and guide content as structured text with titles, headings, canonical URLs, and crawl metadata.

Output preview

Here is a simplified preview of the extracted dataset:

URL	Title	Format	Words	Language	Depth
/academy/web-scraping-for-beginners	Web scraping for beginners	markdown	1842	en	1
/academy/api-integration-guide	API integration guide	markdown	1267	en	1
/academy/rag-pipeline-basics	RAG pipeline basics	markdown	2135	en	1

The same record can also include:

Extra field group	Example value
Content outputs	content, markdown, text, html
Structure signals	title, description, headings, canonicalUrl
Crawl metadata	depth, httpStatusCode, language, wordCount, crawledAt
Clean HTML storage	CLEAN_HTML_INDEX plus separate clean HTML records
Run diagnostics	OUTPUT_SUMMARY, FAILED_PAGES, SKIPPED_PAGES

Typical fields include:

• page identity: url, title, description, canonicalUrl
• main content outputs: content, markdown, text, html, cleanHtml
• page structure: headings
• crawl metadata: contentFormat, wordCount, language, depth, httpStatusCode, crawledAt
• run-level outputs: OUTPUT_SUMMARY, FAILED_PAGES, SKIPPED_PAGES, CLEAN_HTML_INDEX

Full JSON preview

If you want to inspect a more complete example record, open the preview below.

Examples

Option 1: Fast preview for RAG content

Best for a first run. It keeps cost low and returns enough pages to validate selectors, scope, and output quality.

{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "maxPages": 3,
  "crawlMode": "auto",
  "outputFormat": "markdown",
  "maxDepth": 1,
  "sameDomainOnly": true,
  "saveCleanHtml": false
}

Option 2: Crawl directly from website pages

Best when you want to start from one section and follow links recursively.

{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "maxPages": 20,
  "crawlMode": "website",
  "outputFormat": "markdown",
  "maxDepth": 2,
  "sameDomainOnly": true,
  "saveCleanHtml": true
}

Option 3: Crawl from sitemap URLs

Best when the target site already has a sitemap and you want broader coverage with cleaner URL discovery.

{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "maxPages": 50,
  "crawlMode": "sitemap",
  "sitemapUrls": [
    "https://docs.apify.com/sitemap.xml"
  ],
  "maxDepth": 0,
  "outputFormat": "markdown",
  "sameDomainOnly": true,
  "saveCleanHtml": true
}

Best practices

1. Documentation into a vector database

Use the Actor to crawl product docs or API docs, then send markdown or clean HTML into your chunking and embedding pipeline.

This is useful for:

• RAG systems
• developer assistants
• internal technical search

2. Help center into an AI support knowledge base

Use the Actor to crawl support articles and export them as markdown or text.

This is useful for:

• support copilots
• FAQ assistants
• internal support search

3. Website content into an internal knowledge base

Use the Actor to capture blog posts, guides, and product pages in a consistent format.

This is useful for:

• AI knowledge bases
• content migration
• website analysis and archiving

Why the dataset feels complete

This Actor does more than return a list of URLs.

• You get the main content in markdown, text, and HTML.
• You get structure signals such as titles, headings, descriptions, and canonical URLs.
• You get crawl metadata such as word count, depth, language, status code, and crawl time.
• You can store clean HTML separately for downstream parsing or chunking.
• You also get run diagnostics for failed pages, skipped pages, and summary totals.

That combination makes the output useful not just for scraping, but for ingestion, QA, chunking, embeddings, search, and AI application pipelines.

API access

Developers can run this Actor programmatically through the Apify API or the Apify Python and JavaScript clients.

• API reference: Apify API
• Client docs: Apify clients