# Website to RAG Dataset (`sebastian-actors/website-to-rag-dataset`) Actor

Convert public websites, docs, blogs, and XML sitemaps into clean Markdown, structured metadata, and stable chunks for RAG pipelines and vector databases.

- **URL**: https://apify.com/sebastian-actors/website-to-rag-dataset.md
- **Developed by:** [Sebastián S](https://apify.com/sebastian-actors) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 processed pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What is Website to RAG Dataset?

Website to RAG Dataset fetches public, server-rendered HTML from HTTP(S) page URLs, an XML sitemap, or a sitemap index. It produces main-content Markdown, metadata, freshness signals, and stable chunks in separate page and chunk datasets. Use them to build a controlled corpus for retrieval, search, or later embedding.

The Actor uses bounded static HTTP requests. It does not render JavaScript, follow links recursively, authenticate to websites, use proxies, create embeddings, or write to a vector database.

### What data can it extract or produce?

- Main-content Markdown, with CSS selectors for inclusion and removal.
- Canonical URL, title, description, language, robots meta, H1, headings, Open Graph, and Twitter metadata.
- Optional JSON-LD types, breadcrumbs, dates, block counts, and parse errors.
- Optional freshness signals from HTTP `Last-Modified`, sitemap `lastmod`, and JSON-LD dates.
- Optional internal and external links, capped at 250 per group.
- Overlapping chunks with stable IDs, heading paths, lengths, token estimates, and SHA-256 hashes.
- Page `ok`, `skipped`, and `error` records plus an `OUTPUT` summary.

Results depend on the source. Missing metadata is `null` or empty; the Actor does not infer absent values.

### What can you use the data for?

- Build documentation corpora for RAG assistants or internal search.
- Convert help-center articles into support-agent knowledge chunks.
- Prepare blogs or knowledge bases for separate embedding and indexing.
- Preserve source URLs, headings, hashes, and freshness for traceable retrieval.
- Create sitemap snapshots for quality or regression checks.

### How much does it cost?

The Actor uses pay-per-event pricing. A `page-processed` event is charged for each page that completes extraction and is accepted for output. Filters, errors, robots skips, duplicate canonicals, spending-limit skips, and empty pages skipped in `chunks` mode are not charged; chunks are not charged individually.

| Apify plan | Per processed page | Per 1,000 processed pages |
| --- | ---: | ---: |
| Free | $0.004 | $4.00 |
| Bronze | $0.003 | $3.00 |
| Silver | $0.002 | $2.00 |
| Gold, Platinum, or Diamond | $0.0015 | $1.50 |

A $0.00005 Actor-start event also applies per run. Formula: `Actor start + (processed pages x plan price)`. For example, 100 processed pages on Gold cost `$0.00005 + (100 x $0.0015) = $0.15005`.

Check the [Pricing tab](https://apify.com/sebastian-actors/website-to-rag-dataset/pricing) before large runs. Use `maxPages` and Apify's maximum total charge limit as guardrails.

### How to use Website to RAG Dataset

1. Add page URLs, a sitemap URL, or both in Apify Console.
2. Choose `pages`, `chunks`, or `pages_and_chunks`, then set the page and sitemap limits.
3. Run the Actor and inspect **Pages and errors**, **RAG chunks**, and **Run summary**.
4. Export the appropriate dataset or pass its ID downstream.

### What input does the Actor accept?

Provide `urls`, `sitemapUrl`, or both. This minimal input uses schema defaults:

```json
{
  "urls": ["https://example.com/"]
}
```

See the Actor's [Input tab](https://apify.com/sebastian-actors/website-to-rag-dataset/input-schema) for the full schema.

| Field | Behavior and interaction |
| --- | --- |
| `urls`, `sitemapUrl` | Direct pages and/or XML sitemap. Fragments and duplicates are removed. Requires public HTTP(S) URLs without credentials. |
| `maxPages`, `maxSitemaps` | Select 1-10,000 eligible pages; fetch 1-1,000 sitemap files. Filtered diagnostic rows have a separate `maxPages`-sized cap; chunks are not page-capped. |
| `sameDomainOnly` | Require the sitemap hostname, or first direct hostname. Subdomains differ. |
| `urlAllowlistPatterns`, `urlBlocklistPatterns` | Regex filters: URLs must pass the allowlist and not match the blocklist. |
| `respectRobotsTxt`, `userAgent` | Check robots rules with this agent. Retrieval failure warns and allows the page request. |
| `outputMode` | Write pages, chunks, or both; diagnostics always use the default dataset. |
| `includeMarkdown`, `includeHtml` | Control page Markdown and `content.cleanedHtml`; chunks still use Markdown. HTML enlarges output. |
| `includeSchema`, `includeFreshnessSignals`, `includeLinks` | Enable JSON-LD, freshness, and links. Freshness can parse dates while schema output is off. |
| `chunkSize`, `chunkOverlap` | Characters/4 estimates. Size: 200-4,000; overlap: 0-1,000 and smaller than size. |
| `minContentChars` | Marks `content.lowContent`, without rejecting the page. Range: 0-10,000. |
| `maxConcurrency`, `requestTimeoutSecs` | 1-25 parallel pages; 5-120 seconds per request. High concurrency risks rate limits. |
| `onlySelectors`, `removeSelectors` | First inclusion match defines content, then removals apply. No match can yield empty content. |
| `dedupeByCanonical` | Keep one result per canonical URL, or final URL as fallback. |
| `saveRunSummaryToKeyValueStore` | Save `OUTPUT` in the default key-value store. |

HTML, links, JSON-LD, high overlap, and `pages_and_chunks` increase processing or output, but not the per-page event rate.

### What does the output look like?

| Location | Contents |
| --- | --- |
| Default dataset (`pages`) | Successful page rows in `pages` or `pages_and_chunks` mode, plus all skipped and error rows in every mode. |
| Named `chunks` dataset | Successful chunk rows in `chunks` or `pages_and_chunks` mode. It never contains page errors. |
| Default key-value store, `OUTPUT` | Run-level totals and warnings when `saveRunSummaryToKeyValueStore` is enabled. |

For the minimal input above, a compact chunk row can look like this:

```json
{
  "itemType": "chunk",
  "url": "https://example.com/",
  "normalizedUrl": "https://example.com/",
  "status": "ok",
  "processedAt": "2026-08-26T15:42:16.972Z",
  "chunkId": "example-com-0f115db062b7-000",
  "chunkIndex": 0,
  "chunkText": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
  "headingPath": ["Example Domain"],
  "estimatedTokens": 42,
  "title": "Example Domain",
  "description": null,
  "schemaTypes": []
}
```

See the run's **Output** tab for full schemas. Page `metadata`, `freshness`, `schema`, `content`, `links`, and `error` are objects or `null` by status and flags. `finalUrl`, `canonicalUrl`, `statusCode`, `contentType`, and descriptive fields can be `null`. Excluded `markdown` and `content.cleanedHtml` are `null`. Chunk `canonicalUrl`, `freshness`, `title`, `description`, and `language` can be `null`; `schemaTypes` may be empty.

Export either dataset through Console or the Dataset API as JSON, JSONL, CSV, HTML, XLSX, XML, or RSS. JSON or JSONL best preserves nested fields.

### How are errors and partial results handled?

Runs may mix outcomes. Compare default-dataset `status` and `error.code` with the counts and `warnings` in `OUTPUT`.

| Symptom | What to check |
| --- | --- |
| Empty default dataset | In `chunks` mode, successes are in `chunks`. Otherwise inspect sitemap warnings and filters. |
| No chunks | Confirm the mode requests chunks; check JavaScript-only content, `onlySelectors`, or `NO_CHUNKABLE_CONTENT`. |
| Partial sitemap | Inspect `maxPagesApplied`, `maxSitemapsApplied`, filters, timeouts, HTTP errors, and `FILTERED_ROWS_TRUNCATED`. |
| Duplicate skipped | `DUPLICATE_CANONICAL` means another target resolved to the same canonical or final URL. Disable deduplication only intentionally. |
| Failed target | Use `error.retryable` and codes such as `HTTP_NOT_OK`, `FETCH_TIMEOUT`, `RESPONSE_TOO_LARGE`, `NON_HTML_CONTENT`, or URL-safety errors. |
| Run stopped for budget | `CHARGE_LIMIT_REACHED` rows are uncharged. Raise the run's total charge limit or reduce `maxPages`. |

### What are the limits, performance characteristics, and best practices?

- Responses are capped at 5 MiB and pages follow at most five redirects. Redirects and DNS results are safety-checked.
- Sitemap indexes traverse to depth two, bounded by `maxSitemaps` and the response cap.
- The 512 MB Actor supports static HTML, not PDFs, authenticated or private-network pages, or client-rendered content.
- Retryable network failures, HTTP 429, and 5xx responses retry up to twice. Time depends on source latency, sitemap size, and throttling.
- Start with low concurrency, a narrow allowlist, and a small `maxPages`; inspect quality before scaling.
- Use `onlySelectors` for a known article container and `removeSelectors` for repeated menus, consent UI, or related-content blocks.
- Token counts are estimates, not tokenizer guarantees. Validate chunk size with the tokenizer used by your embedding model.

### How can you use the API, schedules, webhooks, and integrations?

The Actor's **API** tab provides REST, Node.js, and Python examples for `sebastian-actors/website-to-rag-dataset`. The official [Node.js and Python clients](https://docs.apify.com/actors/running) can start runs and retrieve storage records; keep tokens server-side.

Use [schedules](https://docs.apify.com/actors/running/schedules) for periodic refreshes and [webhooks](https://docs.apify.com/integrations/webhooks) for run events. Export JSONL to your ingestion job or pass the named `chunks` dataset ID to a separate integration such as [Pinecone](https://docs.apify.com/integrations/pinecone). The Actor itself does not embed or index data, and the integration must read `chunks`, not the default page/error dataset.

### Frequently asked questions

#### Does it crawl every link on a website?

No. Only supplied URLs and sitemap discoveries are processed.

#### Can it render JavaScript-heavy pages?

No. Content must be in server-returned HTML.

#### Does it create embeddings?

No. Embedding and indexing are separate steps.

#### Does `minContentChars` remove short pages?

No. It sets `content.lowContent`. Only `chunks` mode skips pages with no chunkable Markdown.

#### Are chunk IDs stable?

Yes, for the same effective URL and chunk order. Content changes can move boundaries; compare hashes too.

### Is it legal and responsible to use this Actor?

Use the Actor only for public pages you are authorized to process. Review site terms, copyright and database rights, privacy obligations, and applicable laws. `respectRobotsTxt` is an operational safeguard, not permission or a substitute for legal review. This is not legal advice.

Avoid personal or sensitive data without a lawful purpose and suitable controls. Do not bypass access controls, authentication, paywalls, or network restrictions.

### Which related Actors should you consider?

| Actor | Use it when |
| --- | --- |
| [Website Crawl Preflight Auditor](https://apify.com/sebastian-actors/website-crawl-preflight-auditor) | Check robots, sitemap scope, static quality, JavaScript risk, and crawl settings first. |
| [RAG Dataset Linter](https://apify.com/sebastian-actors/rag-dataset-linter) | Check existing chunks for duplicates, overlap, ordering, provenance, and malformed content. |

### Where can you get support?

Open the [Issues tab](https://apify.com/sebastian-actors/website-to-rag-dataset/issues) with the run ID, redacted input, relevant error or warning, and expected result. Never post tokens, cookies, private URLs, or sensitive output; include a reproducible public URL when possible.

# Actor input Schema

## `urls` (type: `array`):

List of public web pages to process. You can provide individual URLs, docs pages, blog posts, or landing pages.

## `sitemapUrl` (type: `string`):

Optional XML sitemap or sitemap index URL. If provided, URLs from the sitemap are processed up to Max pages.

## `maxPages` (type: `integer`):

Maximum number of pages to process.

## `sameDomainOnly` (type: `boolean`):

Only process URLs on the same hostname as the first URL or sitemap URL.

## `respectRobotsTxt` (type: `boolean`):

Check robots.txt and skip URLs disallowed for the configured user agent.

## `outputMode` (type: `string`):

Choose whether to output page rows, chunk rows, or both.

## `includeMarkdown` (type: `boolean`):

Include cleaned Markdown in page output rows.

## `includeSchema` (type: `boolean`):

Extract JSON-LD schema types and key schema fields where available.

## `includeFreshnessSignals` (type: `boolean`):

Extract Last-Modified, sitemap lastmod, datePublished, and dateModified signals when available.

## `includeLinks` (type: `boolean`):

Include internal and external links found in the main content.

## `includeHtml` (type: `boolean`):

Include cleaned HTML in page rows. Disabled by default to keep output smaller.

## `chunkSize` (type: `integer`):

Approximate chunk size in estimated tokens. Token estimate uses a simple characters/4 heuristic.

## `chunkOverlap` (type: `integer`):

Approximate overlap between chunks in estimated tokens.

## `minContentChars` (type: `integer`):

Pages with less cleaned content than this value are marked as low-content.

## `maxConcurrency` (type: `integer`):

Number of pages fetched in parallel.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each page request.

## `maxSitemaps` (type: `integer`):

Maximum number of sitemap and sitemap-index files to fetch.

## `userAgent` (type: `string`):

User agent used for requests and robots.txt checks.

## `urlAllowlistPatterns` (type: `array`):

Optional regex patterns. If provided, only matching URLs are processed.

## `urlBlocklistPatterns` (type: `array`):

Optional regex patterns. Matching URLs are skipped.

## `removeSelectors` (type: `array`):

CSS selectors to remove before Markdown conversion.

## `onlySelectors` (type: `array`):

Optional CSS selectors for the content area. If provided, only matched elements are used.

## `dedupeByCanonical` (type: `boolean`):

Avoid processing duplicate pages with the same canonical URL.

## `saveRunSummaryToKeyValueStore` (type: `boolean`):

Save run-level summary statistics to the default key-value store under OUTPUT.

## Actor input object example

```json
{
  "urls": [
    "https://docs.apify.com/actors"
  ],
  "maxPages": 100,
  "sameDomainOnly": true,
  "respectRobotsTxt": true,
  "outputMode": "pages_and_chunks",
  "includeMarkdown": true,
  "includeSchema": true,
  "includeFreshnessSignals": true,
  "includeLinks": false,
  "includeHtml": false,
  "chunkSize": 1200,
  "chunkOverlap": 150,
  "minContentChars": 250,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 20,
  "maxSitemaps": 100,
  "userAgent": "WebsiteToRagDatasetBot/1.0 (+https://apify.com)",
  "urlAllowlistPatterns": [],
  "urlBlocklistPatterns": [],
  "removeSelectors": [
    "nav",
    "footer",
    "header",
    "aside",
    "script",
    "style",
    "noscript",
    "iframe",
    "form"
  ],
  "onlySelectors": [],
  "dedupeByCanonical": true,
  "saveRunSummaryToKeyValueStore": true
}
```

# Actor output Schema

## `pages` (type: `string`):

Successful page rows plus skipped and failed URL rows.

## `chunks` (type: `string`):

Chunk-only dataset for RAG and vector database ingestion.

## `runSummary` (type: `string`):

Aggregate run summary stored in the default key-value store under OUTPUT.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.apify.com/actors"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("sebastian-actors/website-to-rag-dataset").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://docs.apify.com/actors"] }

# Run the Actor and wait for it to finish
run = client.actor("sebastian-actors/website-to-rag-dataset").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.apify.com/actors"
  ]
}' |
apify call sebastian-actors/website-to-rag-dataset --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sebastian-actors/website-to-rag-dataset"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/sZXcKfY3hwIuZb2IY/builds/QHd36lUV5g4DsZLAN/openapi.json