# RAG Content Crawler — Any Site to Clean Markdown (`eszetael_lab/rag-content-crawler`) Actor

Turn any website into clean Markdown for your LLM or RAG pipeline. Navigation, cookie banners and boilerplate removed; every page says whether the text was truncated. Pay per delivered page — pages that returned nothing are free.

- **URL**: https://apify.com/eszetael\_lab/rag-content-crawler.md
- **Developed by:** [Radosław Szal](https://apify.com/eszetael_lab) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 delivered page as markdowns

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Content Crawler for LLM — Markdown, browser if needed

**What you get back**, one record per page:

```json
{
  "url": "https://docs.example.com/guide/install",
  "title": "Installation",
  "markdown": "# Installation\n\nRun `pip install example`...\n\n| Option | Default |\n|---|---|\n...",
  "charCount": 4812,
  "fetchedWith": "http",
  "tiersTried": ["http"],
  "httpStatus": 200
}
```

`markdown` is the page body with links and tables kept and navigation, cookie banners and footers
removed — ready to embed in a vector database without a second cleaning step.

### Why this one is cheaper to run

Most crawlers in this category start a headless browser for every page. This one tries the cheap
way first and only escalates when a page actually forces it:

| | what it does | when it is used |
|---|---|---|
| `http` | plain request | most pages |
| `tls-fingerprint` | same request with a real browser's TLS handshake | pages behind Cloudflare-style checks |
| `browser` | Chromium renders the page | pages that build their content in JavaScript |

Every record tells you which one paid for it, in `fetchedWith`. You are not asked to take the
saving on trust.

### What we measured, and how

100 pages, two cohorts chosen by **external rankings** rather than by us — the front page of Hacker
News, and the documentation sites of the most-downloaded PyPI packages. Both paths ran against the
same URLs with the **same extractor**, so the difference comes from how the HTML was obtained and
from nothing else.

| | documentation (40) | general web (60) |
|---|---|---|
| plain HTTP is enough | **100%** | 75% |
| adding the TLS fingerprint | — | 83% |
| adding a real browser | — | 93% |
| no method reaches it (paywall, hard block) | 0% | 7% |
| time versus browser-only | 5.3× faster | 10.6× faster |

Three findings worth stating plainly, because they shaped the code:

- Four pages that looked like they needed JavaScript were in fact returning
  `403 Enable JavaScript and cookies to continue`. They gave up their content after the TLS
  handshake changed — no rendering. On `w3.org` that was 14 505 characters where Chromium
  extracted 140. A browser is not a cure for a block; it is an expensive thing that sometimes works.
- On one page a browser was **much worse**: 13 792 characters over HTTP versus 58 rendered. This
  Actor therefore returns the *best* result it obtained, not the last one it tried.
- 7% of the general web returns nothing to any method. Those pages are reported as skipped, with
  the reason, and are not charged.

If you only crawl documentation, set **How hard to try** to `http` and the run will never touch a
browser — on the pages we measured it never needed to.

### Publishers who opted out of AI use are skipped, and you are told why

`robots.txt` says whether a page may be **fetched**. It says nothing about whether its text may be
**used** for text and data mining — and that is exactly what a Markdown corpus is for. Under the EU
copyright directive a publisher may reserve that right in a machine-readable form, and a growing
number do.

This Actor reads those signals from the response it is already fetching — the `TDM-Reservation`
header, `X-Robots-Tag: noai`, and the matching `<meta>` tags in the document head — and when it
finds one, the page is **not converted, not delivered and not charged**. The run status names the
page and the exact signal, so a reserved page never looks like a failure or like a silent gap in
your corpus.

Nothing is guessed from the wording of a page. Only a reservation expressed for machines counts,
because only that one is unambiguous.

### Pricing

You are billed **per delivered page**. A page that returns no extractable content is not delivered
and not charged, and a run that delivers nothing fails with the reason rather than finishing green
and empty.

### Input

| field | meaning |
|---|---|
| `startUrls` | pages to convert |
| `maxPages` | hard cap for the **whole run** — also your cost ceiling |
| `crawlDepth` | 0 = only your URLs, 1–2 = follow links (right for a docs site) |
| `sameDomainOnly` | keep a crawl from wandering off the site |
| `maxTier` | how far it may escalate: `http`, `tls-fingerprint`, `browser` |

### Limits, stated up front

- Pages behind a login or a hard paywall are not reachable and are reported as such.
- `crawlDepth` above 2 mostly discovers navigation pages; it costs more and adds little.
- The run stops at `maxPages` and says so in the run status when more pages were queued.
- A page body is read up to **8 MB** and no further. The largest page in our 100-page sample was
  2.1 MB, so this cuts nothing real — it exists so that a server answering with an endless stream
  cannot take the run down after you have already paid for earlier pages. When the ceiling is hit,
  the page is reported with that reason instead of being escalated to a more expensive method.

### If it worked for you

Ratings are the one thing we cannot build ourselves, and on Apify they feed the quality score that
decides whether anyone finds this Actor at all. If it did the job, a rating on the Store page takes
a few seconds and genuinely changes what happens to it.

If it did **not** do the job, the issue tab is more useful to us than a low rating with no detail —
tell us the URL that failed and what you expected, and it becomes a test case in the next release.
Every defect listed in the changelog got there that way, or from our own gate.

# Actor input Schema

## `startUrls` (type: `array`):

URLs to turn into Markdown. A documentation index, a blog, a single article — anything with readable text. Set Crawl depth above 0 to follow links from these pages.

## `maxPages` (type: `integer`):

Hard cap on delivered pages. You are billed per delivered page, so this is also your cost ceiling: the default of 50 caps a run at $0.05. It counts the WHOLE run, not each start URL.

## `crawlDepth` (type: `integer`):

0 converts only the pages you listed. 1 also follows the links found on them, 2 goes one level further, and so on. Use 1–2 for a documentation site; deeper mostly finds navigation.

## `sameDomainOnly` (type: `boolean`):

Follow links only within the domain they were found on. Turning this off lets a crawl wander into other sites, which is rarely what you want and always costs more.

## `maxTier` (type: `string`):

How far this Actor may escalate to reach a page. `http` is the cheapest and enough for documentation. `tls-fingerprint` also passes Cloudflare-style checks without rendering. `browser` additionally starts Chromium for the few pages that need JavaScript — measured at about one page in ten of the general web, and none of the documentation pages tested. The browser only starts if a page actually forces it.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "maxPages": 50,
  "crawlDepth": 0,
  "sameDomainOnly": true,
  "maxTier": "browser"
}
```

# Actor output Schema

## `pages` (type: `string`):

Every converted page: url, finalUrl, title, markdown and which method fetched it.

## `markdownOnly` (type: `string`):

Just the Markdown bodies — the shape most embedding pipelines want, without the metadata columns.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("eszetael_lab/rag-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }] }

# Run the Actor and wait for it to finish
run = client.actor("eszetael_lab/rag-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ]
}' |
apify call eszetael_lab/rag-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,eszetael_lab/rag-content-crawler"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/2McvXB5bxKH1IL5N2/builds/FN8zAo8NH8rxyejKd/openapi.json