# Website to AI / RAG Data Extractor (`scrapyx/website-rag-extractor`) Actor

Turns any website into a clean, embedding-ready corpus. Strips navigation, footers and cookie banners, converts the real content to markdown, and splits it into overlapping chunks that each carry their own URL, title and metadata. Crawl by URL list, sitemap or link graph.

- **URL**: https://apify.com/scrapyx/website-rag-extractor.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** Developer tools, AI, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website to AI / RAG Data Extractor

Turns any website into a clean, embedding-ready corpus. Strips navigation,
footers and cookie banners, converts the real content to markdown, and splits
it into overlapping chunks that each carry their own URL, title and metadata —
so a chunk can go straight into a vector store without being joined back to
anything.

### Three ways to decide what to fetch

| `discoveryMode` | What it does |
|---|---|
| `urls` | Fetch exactly the URLs you list. Follows nothing. |
| `sitemap` | Read `Sitemap:` from `robots.txt` (falling back to `/sitemap.xml`), follow nested sitemap indexes, then filter with your URL patterns. |
| `crawl` | Walk internal links breadth-first up to `maxDepth`. |

Verified on `docs.apify.com`: `sitemap` mode reads the `robots.txt` declaration
→ a `<sitemapindex>` → 6 nested sitemaps → 3,962 URLs, then narrows to the
section you asked for.

### Three row shapes

- **`CHUNK`** (default) — RAG-ready. Carries `content`, `chunkIndex`,
  `chunkCount`, a stable `chunkId`, `estimatedTokens`, plus the page's `url`,
  `title`, `description`, `language`, `author` and `publishedAt`.
- **`PAGE`** — one row per page: full cleaned markdown, all metadata,
  JSON-LD, OpenGraph, headings outline, internal/external link lists,
  `contentHash` for change detection.
- **`CRAWL_SUMMARY`** — one per run: what was asked, what came back, and a
  `skippedByReason` breakdown.

Plus an `ERROR` row for any URL that genuinely failed, so **every input maps
to at least one row**. A `robots.txt` refusal is *not* an error — it is
reported under `skippedByReason`, because that is the system working.

### Example input

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/" }],
  "discoveryMode": "sitemap",
  "includeUrlPatterns": ["/academy/"],
  "outputGranularity": "chunk",
  "chunkSize": 1500,
  "chunkOverlap": 150,
  "maxPages": 200
}
```

### Chunking

Sizes are in **characters, not tokens** — and that is deliberate. A token
count depends on your tokenizer, and a hardcoded ~4-chars-per-token divisor
produces chunks that are wrong for every non-English corpus. Each chunk also
reports `estimatedTokens`, clearly named as an estimate.

Chunks break at a paragraph, then line, then sentence boundary near the
target, and only hard-cut when none of those appears within 15% of the target.
Breaking mid-sentence hurts retrieval far more than a slightly long chunk
does. `chunkOverlap` repeats the tail of the previous chunk so a fact spanning
a boundary stays retrievable from either side.

### Extraction quality is tagged, not assumed

Every row carries `_source`:

- `S1-trafilatura` — main-content extraction with boilerplate stripped,
  emitted as markdown so heading structure survives into the chunks.
- `S2-selectolax` — fallback for pages trafilatura declines. It returns
  nothing rather than guessing on very short or unusual documents, and "no
  content" is the wrong answer for a page that plainly has some.
- `*-short` — the page really is that short (a stub, a redirect notice).

S2 output is measurably noisier. A pipeline that wants to weight or filter by
extraction quality needs to know which one produced a chunk, so the tag is on
every row rather than inferred.

### Politeness and policy

`respectRobotsTxt` defaults to **on**, and the summary row records which way
the run went. A general-purpose crawler pointed at hosts nobody profiled is
exactly what `robots.txt` exists to govern.

Per RFC 9309, an **unreachable** `robots.txt` (timeout, 5xx, connection error)
means "no policy published" and allows the fetch — a 5xx is not consent
withdrawn. A `robots.txt` that answers with an HTML app shell is likewise
treated as no policy; `shop.tiktok.com` serves a 5.5 KB captcha page there.

Wildcards are matched properly (`*`, `$`, longest-match-wins, `Allow` beats
`Disallow` at equal length). Python's stdlib `robotparser` matches by prefix
only and silently ignores `*`, which produces **false ALLOWs** — the dangerous
direction for a gate to be wrong in.

Turn `respectRobotsTxt` off only for a site you own or have written
authorisation to crawl. The actor logs a warning when you do.

### Known limits

- **PDFs are not parsed.** They are skipped with an explicit `skipped_pdf`
  reason rather than being silently absent, so a documentation site that is
  mostly PDF reports that honestly instead of looking like an empty crawl.
- **No JavaScript.** HTTP-only by design. A site that renders its content
  client-side yields a thin or empty page, reported as a `no_content` error
  row rather than a blank success.
- **Page bodies are capped at 4 MB** before parsing, with `bodyTruncated: true`
  on the row when that bites.
- **URL patterns filter discovered links, never the URLs you typed.** A user
  who names a URL has already decided they want it. (Sitemap mode is the
  exception: there the "seeds" *are* discovered, and narrowing a 40,000-URL
  sitemap to one section is the whole point of the patterns.)

### Transport

`chrome124` by default, rotating through `chrome136`, `firefox133`,
`safari18_0` and `chrome99_android` on a block. Rotating the TLS profile is
the highest-value retry on an unprofiled host: most 403s there are JA3 gates,
not IP ones.

Datacenter Apify Proxy is the default rather than Residential — most
documentation sites, blogs and company websites have no bot mitigation, so
residential rates would be wasted. Switch to Residential for hosts that need it.

# Actor input Schema

## `startUrls` (type: `array`):

Where to start. Must be absolute http(s) URLs — bare domains like 'example.com' are dropped during normalisation. In 'urls' mode these are the only pages fetched; in 'sitemap' mode their sitemaps are read; in 'crawl' mode they seed the link graph.

## `discoveryMode` (type: `string`):

How to decide which pages to fetch. 'urls' fetches exactly the start URLs and follows nothing. 'sitemap' reads the Sitemap: lines from robots.txt (falling back to /sitemap.xml) and follows nested sitemap indexes. 'crawl' walks internal links breadth-first up to Max crawl depth.

## `outputGranularity` (type: `string`):

What a dataset row is. 'chunk' emits RAG-ready chunks, each carrying its own URL, title and metadata so it can go straight into a vector store. 'page' emits one row per page with the full cleaned markdown. 'both' emits pages and chunks, which roughly doubles dataset size.

## `maxPages` (type: `integer`):

Hard cap on how many pages are fetched across the whole run. This is the main cost control — a sitemap or crawl can otherwise expand to tens of thousands of URLs.

## `maxDepth` (type: `integer`):

How many link hops from a start URL to follow. 0 means the start URLs only, 1 adds the pages they link to, and so on. Only used in 'crawl' mode.

## `sameDomainOnly` (type: `boolean`):

In 'crawl' mode, only follow links whose host matches a start URL's host (ignoring a leading www.). Turning this off lets the crawl wander onto any site the pages link to, which is almost never what a RAG corpus wants.

## `includeUrlPatterns` (type: `array`):

If set, a URL is only fetched when at least one of these regular expressions matches it. Useful for pulling just the docs section out of a sitemap, e.g. '/docs/' or '^https://example.com/blog/'.

## `excludeUrlPatterns` (type: `array`):

URLs matching any of these regular expressions are skipped. Applied before the include patterns. Good for dropping tag pages, paginated archives or login flows, e.g. '/tag/' or '?page='.

## `chunkSize` (type: `integer`):

Target chunk length in CHARACTERS, not tokens — a token count would depend on your tokenizer and be wrong for non-English text. Each chunk also reports an estimatedTokens figure. Chunks break at a paragraph, line or sentence boundary near the target rather than cutting mid-sentence.

## `chunkOverlap` (type: `integer`):

How many characters each chunk repeats from the end of the previous one, so a fact spanning a boundary is retrievable from either side. Must be smaller than the chunk size. Set to 0 to disable.

## `includeRawHtml` (type: `boolean`):

Attach each page's raw HTML to its PAGE row. Off by default because it multiplies dataset size several times over and a RAG pipeline consumes the cleaned text, not the markup.

## `respectRobotsTxt` (type: `boolean`):

Check each host's robots.txt before fetching and skip anything it disallows, reporting the skip in the summary row. Leave this on unless you own the site or have written authorisation to crawl it. An unreachable robots.txt is treated as 'no policy published', which allows the fetch.

## `maxConcurrency` (type: `integer`):

Upper bound on pages fetched at the same time. Lower it for small or slow sites — this actor points at hosts nobody profiled, and hammering one is both rude and a fast route to a block.

## `minRequestInterval` (type: `string`):

Minimum delay between the START of consecutive requests. This, not concurrency, is the honest speed control: once the rate cap binds, extra concurrency buys nothing. Raise it if a site starts returning 429s.

## `proxyConfiguration` (type: `object`):

Proxy used for every request. Datacenter Apify Proxy is the default here rather than Residential: most documentation sites, blogs and company websites have no bot mitigation, so residential rates would be wasted. Switch to Residential for hosts that block datacenter traffic.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "discoveryMode": "urls",
  "outputGranularity": "chunk",
  "maxPages": 50,
  "maxDepth": 2,
  "sameDomainOnly": true,
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "includeRawHtml": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "minRequestInterval": "0.2",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/website-rag-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }] }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/website-rag-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ]
}' |
apify call scrapyx/website-rag-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/website-rag-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/nYsWkmajZB99pxlNO/builds/lBkS3IKx0cu5Hahqf/openapi.json