🧠 Website to Clean Markdown (AI & RAG Ready)

Turn any web page — or an entire website — into clean, structured Markdown with optional RAG chunks. Built for AI developers, RAG pipelines, and LLM engineers working with ChatGPT, Claude, LangChain, LlamaIndex, and vector databases.

Why this Actor?

Raw HTML wastes thousands of tokens on navigation, ads, scripts, and markup. This Actor strips the noise and returns readable Markdown (or ready-to-embed chunks), so you spend less and get better answers from your models.

• Single page or whole site: convert a batch of URLs, crawl a website, or discover pages via sitemap.xml.
• Automatic rendering: pages are fetched with fast plain HTTP, and JavaScript single-page apps are detected and re-fetched with a headless browser automatically — no configuration needed.
• AI-native output: clean Markdown plus token estimate, content hash, and optional RAG chunks.
• Readability extraction: uses Mozilla Readability to keep the real article content and drop the page chrome.
• RAG chunking: split pages into overlapping, line-aware chunks for Pinecone, Weaviate, Qdrant, or pgvector.
• Change detection: every item carries a hash so you can skip unchanged pages across runs.
• Noise removal: drops scripts, styles, headers, footers, navs, and forms.
• Optional proxy: enable Apify Proxy for protected or rate-limited sites.
• Reliable: automatic retries and per-URL error reporting — one bad URL never fails the run.

Use cases

• RAG pipelines: build a chunked Markdown corpus for Pinecone, Weaviate, Qdrant, or pgvector.
• Custom GPTs & assistants: ingest up-to-date docs, blogs, and knowledge bases.
• Fine-tuning datasets: build instruction/context corpora from web content.
• Content repurposing: turn blog posts into newsletters, threads, or summaries.
• Monitoring: snapshot pages as Markdown and compare hashes over time to detect changes.

How to use

1. Paste one or more URLs into URLs to convert.
2. For a whole site, enable Crawl website (and set Max pages / Max crawl depth) or Read from sitemap.
3. (Optional) enable Return RAG chunks and tune Extract only main content / Remove links.
4. Click Start and download results as JSON, CSV, Excel, or XML.

Input

Field	Type	Default	Description
urls	array	–	Required. Starting web page URLs.
crawl	boolean	false	Follow links to convert a whole website.
maxPages	integer	50	Page limit when crawling or reading a sitemap.
maxDepth	integer	2	How many link levels deep to follow when crawling.
sameDomainOnly	boolean	true	When crawling, stay on the same hostname.
useSitemap	boolean	false	Discover pages from /sitemap.xml.
onlyMainContent	boolean	true	Extract only main content (Mozilla Readability).
removeLinks	boolean	false	Keep anchor text but remove URLs to save tokens.
chunk	boolean	false	Also return overlapping RAG chunks per page.
chunkSize	integer	1200	Max chunk length in characters (when chunking).
chunkOverlap	integer	200	Characters shared between chunks (when chunking).
proxyConfiguration	object	{ "useApifyProxy": false }	Enable Apify Proxy for protected sites.
maxConcurrency	integer	10	Number of pages processed in parallel.
requestTimeoutSecs	integer	30	Per-request timeout in seconds.

Example input — crawl a docs site into chunks

{
  "urls": ["https://crawlee.dev"],
  "useSitemap": true,
  "maxPages": 100,
  "chunk": true,
  "chunkSize": 1200,
  "chunkOverlap": 200
}

Example input — a single page

{
  "urls": ["https://apify.com/about"],
  "onlyMainContent": true,
  "removeLinks": false
}

Example input — a JavaScript single-page app

{
  "urls": ["https://crawlee.dev"],
  "onlyMainContent": true
}

JS-rendered pages are detected and converted with a headless browser automatically — no extra settings required.

Output

The Actor writes one item per URL to the default dataset:

Field	Type	Description
url	string	Final URL of the page (after redirects).
title	string | null	Page title (Readability or <title>).
description	string | null	Meta description or og:description.
statusCode	integer | null	HTTP response status code.
contentType	string | null	Response Content-Type.
markdown	string	The page as clean Markdown.
charCount	integer	Number of characters in the Markdown.
wordCount	integer	Approximate word count.
tokenEstimate	integer	Estimated LLM tokens (~characters / 4).
chunkCount	integer | null	Number of chunks (null if chunking off).
chunks	array | null	Overlapping Markdown chunks (null if chunking off).
hash	string | null	Short SHA-256 of the Markdown, for change detection.
crawledAt	string	ISO 8601 timestamp of when the page was processed.
error	string | null	Error message if the page failed, otherwise null.

Example output

{
  "url": "https://apify.com/about",
  "title": "About Apify",
  "description": "We're building the largest marketplace of tools for AI...",
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "markdown": "# About Apify\n\nWe're building the world's largest platform...",
  "charCount": 8756,
  "wordCount": 566,
  "tokenEstimate": 2189,
  "chunkCount": 8,
  "chunks": ["# About Apify\n\nWe're building the world's largest platform..."],
  "hash": "9f2c4b1e7a3d5f60",
  "crawledAt": "2026-01-31T12:00:00.000Z",
  "error": null
}

Pricing

Fast by design. Most pages are fetched with plain HTTP requests instead of a headless browser, so they consume minimal Compute Units — ideal for articles, blogs, docs, and marketing pages. Only JavaScript-rendered single-page apps trigger the browser automatically, and images are never downloaded, keeping even those runs lean. Keep the proxy off and maxPages modest for the cheapest runs.

Integrations

• Apify API: trigger a run and read the dataset programmatically:

  curl -X POST "https://api.apify.com/v2/acts/<your-username>~website-to-clean-markdown-ai-rag-ready/runs?token=<YOUR_TOKEN>" \
    -H "Content-Type: application/json" \
    -d '{ "urls": ["https://apify.com/about"], "chunk": true }'
  

• MCP server / AI agents: call this Actor directly from Claude, Cursor, and other MCP-compatible clients through the Apify MCP server.

SEO keywords

website to markdown, website crawler to markdown, html to markdown, url to markdown, web scraper for LLM, LLM data extraction, RAG data ingestion, RAG chunking, markdown chunks for vector database, ChatGPT data, Claude data, LangChain loader, LlamaIndex, vector database content, AI training data, content extraction, sitemap to markdown, markdown converter.

FAQ

How is this different from a normal web scraper?

A normal scraper returns raw HTML, which is noisy, bloated, and expensive to feed to an LLM. This Actor returns clean Markdown with the navigation, scripts, and styling already removed, plus a token estimate so you know the cost before calling your model.

Can it convert a whole website, not just one page?

Yes. Enable Crawl website to follow links (with a page limit and depth limit), or Read from sitemap to load every page listed in the site's sitemap.xml.

What are RAG chunks?

With Return RAG chunks enabled, each page's Markdown is split into overlapping pieces sized for embedding. You get a chunks array plus chunkCount, ready to push straight into a vector database.

Which websites are supported?

Any publicly accessible page that returns HTML over HTTP(S). Server-rendered sites are fetched with fast plain HTTP. When a page looks empty or blocked, the Actor automatically re-fetches it with a headless browser, so JavaScript single-page apps (React, Vue, Angular) are handled too — no configuration required.

Can I use it with scheduling?

Yes. Create a task from this Actor and schedule it to monitor pages or refresh your knowledge base — use the hash field to detect what changed.

What happens if a page is blocked or fails?

The Actor retries failed requests a few times and supports optional Apify Proxy for protected sites. If a page still fails, it is written to the dataset with an error message and the run continues with the remaining URLs.

For AI agents & LLM apps

This Actor is callable via the Apify MCP server and the Apify API.

Purpose: convert web pages (or whole websites) into clean Markdown and RAG chunks for LLMs.

Minimal working input:

{
  "urls": ["https://apify.com/about"]
}

Output fields: url, title, description, statusCode, contentType, markdown, charCount, wordCount, tokenEstimate, chunkCount, chunks, hash, crawledAt, error.

Behavior: results are written to the default dataset and can be exported as JSON, CSV, XML, or Excel.

Legal & compliance disclaimer

This Actor is an independent tool. It only accesses publicly available pages and does not bypass authentication, login requirements, or CAPTCHA challenges.

Users are solely responsible for ensuring their use of this Actor complies with the terms of service of the websites they access and with all applicable data-protection laws and regulations. The developer assumes no liability for misuse of this tool or for any consequences resulting from data extraction activities.

Related Actors

Verified public Actors on Apify that pair well with this one:

• Apify Website Content Crawler
• Apify RAG Web Browser
• Apify Google Search Scraper