# LLM-Ready Web Content Extractor (`beyondxl/llm-ready-content-extractor`) Actor

Turn any public web page or shallow site crawl into clean, LLM-ready Markdown + metadata + optional RAG chunks. Boilerplate/nav/ads removed. Honors robots.txt.

- **URL**: https://apify.com/beyondxl/llm-ready-content-extractor.md
- **Developed by:** [Leopold Ai](https://apify.com/beyondxl) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 page extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## LLM-Ready Web Content Extractor

**Give it a URL. Get back clean, LLM-ready Markdown** — main content only,
navigation/ads/boilerplate stripped, with page metadata and optional RAG-ready
chunks. Built for the one thing agents and RAG pipelines actually need: clean web
text, not raw HTML.

### Why this Actor

Raw HTML is noisy — nav bars, cookie banners, ads, footers. This Actor pulls out
just the **main content**, converts it to clean Markdown, and hands you structured,
ready-to-use output. No parsing, no cleanup, no LLM tokens wasted on junk.

### What you get (per page)

| Field | Description |
|---|---|
| `markdown` | Clean main content as Markdown (links & tables preserved) |
| `text` | Plain-text version |
| `title`, `description`, `author`, `date`, `siteName` | Page metadata |
| `wordCount`, `tokenEstimate` | Size at a glance |
| `chunks` | Optional overlapping chunks, ready for a vector DB |

### Example

**Input**

```json
{
  "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Vector_database" }],
  "maxPages": 1,
  "ragChunking": true,
  "chunkWords": 200
}
```

**Output (excerpt)**

```json
{
  "url": "https://en.wikipedia.org/wiki/Vector_database",
  "title": "Vector database - Wikipedia",
  "wordCount": 1528,
  "tokenEstimate": 3121,
  "markdown": "| Part of a series on | Machine learning and data mining | ...",
  "chunks": ["Vector database — a database that stores ...", "..."]
}
```

### Use cases

- **RAG ingestion** — clean text + chunks straight into your vector DB
- **Agent tools** — give an agent a URL, get usable content back
- **Content pipelines** — docs, blogs, news, wikis → structured Markdown
- **Research & analysis** — summarize or analyze clean page content

### Inputs

- `startUrls` — pages to extract (seeds when crawling)
- `maxPages`, `maxCrawlDepth`, `sameDomainOnly` — control scope & cost
- **`renderJs`** — render JavaScript with a headless browser (for client-rendered SPAs)
- `ragChunking`, `chunkWords`, `chunkOverlapWords` — RAG output
- `respectRobots` (default **on**), `includeHtml`, `contactEmail`

### Two modes

- **HTTP (default)** — fast & cheap; great for server-rendered pages (blogs, news,
  docs, wikis — most content sites).
- **`renderJs`** — headless Chromium renders the page first, so **JavaScript-heavy
  SPAs** (client-rendered apps, some docs) work too. Slower; turn on only when needed.

### Responsible by design

Fetches only **public** pages, **honors `robots.txt`**, identifies itself in the
User-Agent, and never touches login/paywalled content or personal data. Use it on
content you're permitted to process.

### Pricing

**Pay per page extracted** — you're charged only for pages that succeed. Skipped,
blocked, or failed pages are free.

# Actor input Schema

## `startUrls` (type: `array`):

Public web pages to extract. With crawl depth > 0, these are the seeds.

## `maxPages` (type: `integer`):

Hard cap on pages extracted (controls cost).

## `maxCrawlDepth` (type: `integer`):

0 = only the start URLs. 1+ = follow links this many hops.

## `sameDomainOnly` (type: `boolean`):

When crawling, only follow links on the first URL's domain.

## `respectRobots` (type: `boolean`):

Honor each site's robots.txt (recommended; keep on).

## `renderJs` (type: `boolean`):

Use a headless Chromium browser to render JavaScript before extracting. Turn on for client-rendered SPAs whose content doesn't appear in the raw HTML. Slower than the default HTTP mode.

## `ragChunking` (type: `boolean`):

Also output the content split into overlapping chunks for vector DBs.

## `chunkWords` (type: `integer`):

Target words per RAG chunk (used when RAG chunking is on).

## `chunkOverlapWords` (type: `integer`):

Words of overlap between consecutive RAG chunks.

## `includeHtml` (type: `boolean`):

Also include the raw page HTML in each record.

## `contactEmail` (type: `string`):

Added to the crawler User-Agent as a courtesy contact. Leave blank to stay anonymous.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform"
    }
  ],
  "maxPages": 10,
  "maxCrawlDepth": 0,
  "sameDomainOnly": true,
  "respectRobots": true,
  "renderJs": false,
  "ragChunking": false,
  "chunkWords": 400,
  "chunkOverlapWords": 40,
  "includeHtml": false,
  "contactEmail": ""
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/platform"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("beyondxl/llm-ready-content-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/platform" }] }

# Run the Actor and wait for it to finish
run = client.actor("beyondxl/llm-ready-content-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform"
    }
  ]
}' |
apify call beyondxl/llm-ready-content-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=beyondxl/llm-ready-content-extractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/Id7TZFMKMMFK1y2UN/builds/xS5eAi0hZADTa1rnp/openapi.json