LLM Web Scraper — Any URL/Website to Clean Markdown for RAG & ChatGPT

Turn any URL or website into clean, LLM-ready Markdown (or plain text) — nav bars, ads, cookie banners and footers stripped out — with metadata (title, author, date, description) and an approximate token count. Built to feed ChatGPT / Claude context windows and RAG ingestion pipelines.

Why this one

The most common thing people do with a scraper today is hand its output to an LLM. This Actor is built for exactly that: give it URLs and get back the clean article body as Markdown — no HTML soup, no boilerplate — ready to paste into a prompt or chunk into a vector database.

• Fast by default — an HTTP fetch with Chrome impersonation handles most pages cheaply.
• Browser fallback — JS-heavy or bot-protected pages are rendered with a hardened browser (Camoufox) automatically.
• Crawl mode — optionally follow same-domain links to ingest a whole site or docs section.

What it does

• Convert one or many URLs to Markdown, plain text, or both.
• Keep or drop links and images; truncate to a character budget to fit token limits.
• Returns metadata + wordCount, charCount, and tokensApprox per page.
• Optional same-domain crawl with page and depth caps to protect your budget.
• Export to JSON, CSV, or Markdown, or pull via the Apify API.

Input

Field	Description
startUrls	Pages to convert. One or more URLs.
crawl	Also follow same-domain links from each URL.
maxPages	Total cap on pages scraped.
maxDepth	Link-hops from a start URL when crawling.
outputFormat	markdown, text, or both.
includeLinks / includeImages	Keep links / images in the Markdown.
maxCharsPerPage	Truncate each page (0 = no limit).
useBrowser	auto (fallback), always, or never.
proxyConfiguration	Proxy for fetching.

Example

{
  "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }],
  "crawl": false,
  "outputFormat": "markdown",
  "includeLinks": true,
  "maxCharsPerPage": 0,
  "useBrowser": "auto"
}

Output

One dataset item per page:

{
  "success": true,
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation - Wikipedia",
  "sitename": "Wikimedia Foundation, Inc.",
  "date": "2023-11-05",
  "wordCount": 3200,
  "tokensApprox": 6300,
  "method": "http",
  "markdown": "# Retrieval-augmented generation\n\n**RAG** is a technique that..."
}

Use cases

• RAG ingestion — chunk markdown into a vector DB (Pinecone, pgvector, Weaviate…).
• LLM context — paste a page's clean text straight into ChatGPT/Claude.
• Research & summarisation — pull a set of articles and summarise in bulk.
• Docs / knowledge base — crawl a documentation site into one Markdown corpus.

FAQ

Does it handle JavaScript sites? Yes — set useBrowser to auto (default) or always.

How is the token count calculated? An approximation (~4 characters per token); use it as a budgeting guide.

Can it crawl a whole site? Yes — enable crawl and raise maxPages / maxDepth.

Respect each site's terms of use and robots directives.

🤖 Use with Claude or ChatGPT (MCP)

Run this actor from Claude, ChatGPT, Cursor or any MCP client via the Apify MCP server. In Claude Desktop: Settings → Connectors → Add custom connector → https://mcp.apify.com. Or expose just this tool:

{ "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=get_anything/llm-web-scraper" } } }

Full guide: Connect Apify actors to Claude & ChatGPT.

⭐ Found this useful?

If this Actor saved you time, please leave a rating on its Apify page — reviews genuinely help others discover it and help me keep improving it.