# LLM Web Scraper - Website to Markdown for RAG & ChatGPT (`get_anything/llm-web-scraper`) Actor

Turn any URL or website into clean, LLM-ready Markdown or text - boilerplate, ads and nav stripped - with metadata and token count. Feed ChatGPT/Claude context or build RAG pipelines. HTTP-fast with a hardened-browser fallback for JS-heavy pages. Export JSON, CSV or Markdown.

- **URL**: https://apify.com/get\_anything/llm-web-scraper.md
- **Developed by:** [Get Anything](https://apify.com/get_anything) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## LLM Web Scraper — Any URL/Website to Clean Markdown for RAG & ChatGPT

Turn **any URL or website** into clean, **LLM-ready Markdown** (or plain text) — nav bars, ads, cookie banners and footers stripped out — with metadata (title, author, date, description) and an **approximate token count**. Built to feed **ChatGPT / Claude** context windows and **RAG** ingestion pipelines.

### Why this one

The most common thing people do with a scraper today is **hand its output to an LLM**. This Actor is built for exactly that: give it URLs and get back the clean article body as Markdown — no HTML soup, no boilerplate — ready to paste into a prompt or chunk into a vector database.

- **Fast by default** — an HTTP fetch with Chrome impersonation handles most pages cheaply.
- **Browser fallback** — JS-heavy or bot-protected pages are rendered with a hardened browser (Camoufox) automatically.
- **Crawl mode** — optionally follow same-domain links to ingest a whole site or docs section.

### What it does

- Convert one or many URLs to **Markdown, plain text, or both**.
- **Keep or drop links and images**; **truncate** to a character budget to fit token limits.
- Returns **metadata** + `wordCount`, `charCount`, and `tokensApprox` per page.
- Optional **same-domain crawl** with page and depth caps to protect your budget.
- Export to **JSON, CSV, or Markdown**, or pull via the Apify API.

### Input

| Field | Description |
|-------|-------------|
| `startUrls` | Pages to convert. One or more URLs. |
| `crawl` | Also follow same-domain links from each URL. |
| `maxPages` | Total cap on pages scraped. |
| `maxDepth` | Link-hops from a start URL when crawling. |
| `outputFormat` | `markdown`, `text`, or `both`. |
| `includeLinks` / `includeImages` | Keep links / images in the Markdown. |
| `maxCharsPerPage` | Truncate each page (0 = no limit). |
| `useBrowser` | `auto` (fallback), `always`, or `never`. |
| `proxyConfiguration` | Proxy for fetching. |

#### Example

```json
{
  "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }],
  "crawl": false,
  "outputFormat": "markdown",
  "includeLinks": true,
  "maxCharsPerPage": 0,
  "useBrowser": "auto"
}
```

### Output

One dataset item per page:

```json
{
  "success": true,
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation - Wikipedia",
  "sitename": "Wikimedia Foundation, Inc.",
  "date": "2023-11-05",
  "wordCount": 3200,
  "tokensApprox": 6300,
  "method": "http",
  "markdown": "# Retrieval-augmented generation\n\n**RAG** is a technique that..."
}
```

### Use cases

- **RAG ingestion** — chunk `markdown` into a vector DB (Pinecone, pgvector, Weaviate…).
- **LLM context** — paste a page's clean text straight into ChatGPT/Claude.
- **Research & summarisation** — pull a set of articles and summarise in bulk.
- **Docs / knowledge base** — crawl a documentation site into one Markdown corpus.

### FAQ

**Does it handle JavaScript sites?** Yes — set `useBrowser` to `auto` (default) or `always`.

**How is the token count calculated?** An approximation (~4 characters per token); use it as a budgeting guide.

**Can it crawl a whole site?** Yes — enable `crawl` and raise `maxPages` / `maxDepth`.

*Respect each site's terms of use and robots directives.*

### 🤖 Use with Claude or ChatGPT (MCP)

Run this actor from Claude, ChatGPT, Cursor or any MCP client via the [Apify MCP server](https://mcp.apify.com). In **Claude Desktop**: Settings → Connectors → Add custom connector → `https://mcp.apify.com`. Or expose just this tool:

```json
{ "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=get_anything/llm-web-scraper" } } }
```

Full guide: [Connect Apify actors to Claude & ChatGPT](https://dev.to/get_anything/connect-any-apify-scraper-to-claude-or-chatgpt-in-2-minutes-mcp-37he).

### ⭐ Found this useful?

If this Actor saved you time, please leave a rating on its [Apify page](https://apify.com/get_anything/llm-web-scraper) — reviews genuinely help others discover it and help me keep improving it.

# Actor input Schema

## `startUrls` (type: `array`):

Web pages to convert to clean Markdown/text. Add one or more URLs.

## `crawl` (type: `boolean`):

Also follow same-domain links from each URL, up to the page and depth caps below.

## `maxPages` (type: `integer`):

Total cap on pages to scrape across all URLs (protects your budget).

## `maxDepth` (type: `integer`):

How many link-hops away from a start URL to follow when crawling (0 = only the given URLs).

## `outputFormat` (type: `string`):

What content to return per page.

## `includeLinks` (type: `boolean`):

Preserve inline hyperlinks in the Markdown output.

## `includeImages` (type: `boolean`):

Preserve image references in the Markdown output.

## `maxCharsPerPage` (type: `integer`):

Truncate each page's content to this many characters (0 = no limit). Useful to fit token budgets.

## `useBrowser` (type: `string`):

JS-heavy or bot-protected pages need a real browser. 'Auto' tries a fast HTTP fetch first and falls back to a hardened browser.

## `proxyConfiguration` (type: `object`):

Proxy for fetching. Residential proxies help with geo-blocked or protected sites.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ],
  "crawl": false,
  "maxPages": 20,
  "maxDepth": 1,
  "outputFormat": "markdown",
  "includeLinks": true,
  "includeImages": false,
  "maxCharsPerPage": 0,
  "useBrowser": "auto",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Structured results in the default dataset - one item per page with clean Markdown/text.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("get_anything/llm-web-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }] }

# Run the Actor and wait for it to finish
run = client.actor("get_anything/llm-web-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ]
}' |
apify call get_anything/llm-web-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,get_anything/llm-web-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/nVlYSc9DdXDRJXoOr/builds/hfh9yglOTBDgO4CE8/openapi.json