# Web2RAG – Website to RAG Dataset (`mdkhadikul/web2rag`) Actor

Turn permitted public websites into clean, sentence-aware text chunks with source metadata, ready for embeddings, vector databases and AI assistants. Exports JSON, JSONL or Markdown.

- **URL**: https://apify.com/mdkhadikul/web2rag.md
- **Developed by:** [Khadikul](https://apify.com/mdkhadikul) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.02 / 1,000 rag chunks

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web2RAG – Website to RAG Dataset

**Turn a public website into clean, chunked text that is ready for embeddings, vector databases and AI assistants. You don't need to write any code.**

Give Web2RAG a starting URL. It crawls the site politely, extracts the readable content of each page, and removes navigation, ads, cookie banners and duplicate pages. It then splits the text into sentence-aware chunks. Every chunk becomes one record in the run's **Dataset**, with its source URL and page metadata. Download it as JSON, JSONL, CSV or Excel, or load it straight into Pinecone, Weaviate, Qdrant, pgvector, LangChain or LlamaIndex.

### Why Web2RAG?

- **RAG-ready out of the box.** Chunks are sized for embedding models and never cut a sentence in half. Context overlaps between chunks, and headings stay with their section.
- **Clean text.** Navigation, footers, sidebars, scripts, ads, cookie banners and hidden elements are removed. Headings and paragraphs are kept.
- **No duplicates.** Identical and near-identical pages, and boilerplate repeated across pages, are dropped.
- **Honest metadata.** Title, description, language, published date and canonical URL are taken from the page. A missing value stays empty. Nothing is invented.
- **Stable IDs.** Each chunk ID is derived from its source and content. Re-running the crawl and upserting into your vector store doesn't create duplicates.
- **Respectful by default.** Web2RAG obeys robots.txt (including `Crawl-delay`), honors `noindex`, throttles requests, stays on the starting website and identifies itself.
- **Transparent.** Every run produces a crawl report listing which pages were processed, skipped or failed, and why.

**Typical uses:** a support chatbot built on your help center, internal search over documentation, a knowledge base for an AI agent, training or evaluation corpora from your own content, and content audits.

### How to use

1. **Start URL.** Enter the page to start from, such as `https://docs.yourproduct.com/`. Starting from a section (for example `/docs/`) keeps the crawl focused.
2. **Size the crawl.** Choose **Maximum pages** and **Maximum crawl depth**. For a new site, try 10–20 pages first and check the results.
3. **Choose an output format** for the extra download file: JSONL (recommended), JSON or Markdown. The Dataset always contains every chunk, whichever format you pick.
4. **Adjust chunking if needed.** The defaults (1000 characters with 150 overlap) work well for most embedding models.
5. **Leave the crawl rules at their defaults** unless you own the site or have permission to change them.
6. **Click Start.** The status bar shows live progress, and a summary appears when the run finishes.
7. **Get your data.**
   - **Output** tab or **Storage → Dataset:** preview the chunks and export them as JSON, JSONL, CSV, Excel, XML or HTML.
   - **Storage → Key-value store:** `web2rag-output.jsonl` (or `.json`/`.md`) and the crawl report `MANIFEST.json`.
   - **API:** `GET https://api.apify.com/v2/datasets/<datasetId>/items?format=jsonl`

### Input

| Field | Default | What it does |
|---|---|---|
| **Start URL** | (required) | Public `http(s)://` page to start from. Private and local network addresses are rejected. |
| **Maximum pages** | 20 | Upper limit on pages requested (1–5000). Pages blocked by robots.txt don't count. |
| **Maximum crawl depth** | 2 | Clicks away from the start URL (0–20). "Next page" pagination is followed regardless of depth, up to 50 pages per listing. |
| **Output format** | JSONL | Format of the extra download file: JSONL, JSON or Markdown. |
| **Chunk size** | 1000 | Maximum characters per chunk (100–10 000). Chunks end on sentence boundaries. |
| **Chunk overlap** | 150 | Characters repeated between neighbouring chunks (0 to half the chunk size). |
| **Stay on the starting website** | on | Follow only links on the same site. `www.` and the bare domain count as the same site; other subdomains don't. |
| **Respect robots.txt** | on | Skip disallowed pages and honor `Crawl-delay`. |
| **Delay between requests** | 0.5 s | Minimum wait between requests to the site (0–60 s). A longer robots.txt `Crawl-delay` wins, up to 30 s. |
| **Request timeout** *(Advanced)* | 15 s | How long to wait for a page. Timeouts and temporary errors are retried twice. |

Invalid settings stop the run immediately with a plain-language message. For example: *"Chunk overlap (300) must be at most half of the chunk size (400)."*

#### Input examples

**Documentation site (recommended starting point)**

```json
{
  "startUrl": "https://docs.example.com/",
  "maxPages": 200,
  "maxDepth": 3,
  "outputFormat": "jsonl",
  "chunkSize": 1000,
  "chunkOverlap": 150
}
```

**Quick test of a new site**

```json
{
  "startUrl": "https://www.example.com/help/",
  "maxPages": 10,
  "maxDepth": 1
}
```

**Small chunks for precise Q\&A retrieval, gentler crawling**

```json
{
  "startUrl": "https://example.com/faq",
  "maxPages": 50,
  "chunkSize": 500,
  "chunkOverlap": 80,
  "requestDelay": 2
}
```

**Human-readable Markdown export of a blog (pagination is followed automatically)**

```json
{
  "startUrl": "https://example.com/blog/",
  "maxPages": 100,
  "maxDepth": 1,
  "outputFormat": "markdown"
}
```

**Start a run from code** (Python, [`apify-client`](https://docs.apify.com/api/client/python)). Replace `<username>/web2rag` with this Actor's ID:

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("<username>/web2rag").call(run_input={"startUrl": "https://docs.example.com/", "maxPages": 100})
for chunk in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(chunk["source_url"], chunk["content"][:80])
```

### Output

#### Dataset: one item per chunk

```json
{
  "id": "b07cf53371f34151cc02916a3977fabe",
  "source_url": "https://docs.example.com/getting-started",
  "title": "Getting started",
  "content": "Installation\nDownload the installer from the customer portal and run it with administrator rights. The installer checks your system requirements before copying any files.",
  "chunk_index": 0,
  "language": "en",
  "metadata": {
    "description": "Install and run the product.",
    "published_date": "2025-01-15",
    "canonical_url": "https://docs.example.com/getting-started"
  },
  "collected_at": "2026-11-04T17:16:30+00:00"
}
```

| Field | Meaning |
|---|---|
| `id` | Stable chunk ID. Use it as the vector-store primary key. |
| `source_url` | The page the chunk came from, after redirects. Use it for citations. |
| `title` | The page's title, or `""` if it has none. |
| `content` | Clean chunk text. Headings sit on their own lines. |
| `chunk_index` | Position of the chunk within its page (0, 1, 2, …). |
| `language` | The language the page declares (`en`, `es-MX`, …), or `null`. |
| `metadata.description` | The page's meta description, or `""`. |
| `metadata.published_date` | Publication date as stated on the page, or `null`. A date-only value stays a date. |
| `metadata.canonical_url` | The page's declared canonical URL, or `null`. |
| `collected_at` | When the page was fetched (UTC, ISO 8601). |

The Dataset tab has three views: **Chunks** (readable overview), **Page metadata**, and **Full records**. CSV and Excel exports flatten `metadata` into `metadata/description` and similar columns.

#### Download file (key-value store)

`web2rag-output.jsonl`, `web2rag-output.json` or `web2rag-output.md` contains the same records in the format you chose. The Markdown file groups chunks under each page's title and source URL, which makes it easy to review.

#### Crawl report: `MANIFEST.json`

```json
{
  "start_url": "https://docs.example.com/",
  "total_pages_discovered": 18,
  "pages_requested": 16,
  "successfully_processed_pages": 11,
  "pages_with_content": 9,
  "failed_pages": 2,
  "skipped_pages": { "robots_txt": 2, "not_html": 1, "off_site_redirect": 0, "noindex": 1,
                     "unsafe_url": 0, "redirect_to_crawled_page": 1 },
  "empty_pages": 1,
  "duplicate_count": 1,
  "duplicate_chunks": 0,
  "total_chunks": 16,
  "export_format": "jsonl",
  "processing_duration_seconds": 9.4,
  "failures": [ { "url": "https://docs.example.com/old", "outcome": "failed", "reason": "HTTP 404" } ]
}
```

The report also records the settings used and the Web2RAG version, so you can reproduce a run.

#### Using the data

```python
import json
records = [json.loads(line) for line in open("web2rag-output.jsonl", encoding="utf-8")]
texts = [r["content"] for r in records]                       # embed these
metadatas = [{"source": r["source_url"], "title": r["title"]} for r in records]
ids = [r["id"] for r in records]                              # stable upsert keys
```

### Pricing

Web2RAG uses **pay-per-event** pricing: you pay only for the RAG chunks you receive.

| What you pay for | Price |
|---|---|
| Each chunk saved to the dataset | **$0.0002**, which is **$0.20 per 1,000 chunks** |
| Run start (Apify's standard fee) | about $0.00005 per run |

- **Platform usage is included.** You aren't billed separately for compute, storage or data transfer.
- **You only pay for results.** Pages that fail, are skipped (robots.txt, `noindex`, not HTML), are duplicates or have no text don't create chunks, so they cost nothing.
- **Typical cost:** with the default chunk size (1000 characters), a page usually yields 4–7 chunks, so **1,000 pages cost about $1**. For example, a real 20-page crawl of an e-commerce site produced 133 chunks, which cost **$0.027**.
- **Larger chunks are cheaper.** A bigger **Chunk size** produces fewer chunks per page.
- **Stay in control:** set **Maximum cost per run** in the run options. Web2RAG produces only as many chunks as that budget covers, then stops and keeps everything collected so far.
- **Try it free:** the Apify free plan includes **$5 of monthly credit**, enough for about 25,000 chunks (roughly 4,000–6,000 pages).

Start with 10–20 pages to see how many chunks a site produces. Any future price change will be announced on this page in advance.

### Limitations

- **No JavaScript rendering.** Pages that build their content in the browser (some single-page apps) come out empty and are counted as `empty_pages` in the report. Try the site's server-rendered docs, or another URL.
- **HTML only.** PDFs, Office documents, images and other files are skipped.
- **No access bypass.** Logins, paywalls, CAPTCHAs and anti-bot protections are never circumvented. Pages answering 401 or 403 are reported as failed.
- **One site at a time, politely.** Requests are sequential with a delay, so very large sites take longer. The maximum is 5000 pages per run.
- **Scope rules.** Other subdomains (for example `blog.example.com`) aren't crawled from `example.com`. Start a separate run for them. If the start URL redirects (for example to `www.`), the crawl follows the final address.
- **Robots.txt failures are strict.** If a site's robots.txt can't be read (server error, 403 or timeout), Web2RAG crawls nothing on that site rather than guess. A missing robots.txt (404) means no restrictions.
- **Language detection.** The language comes from the page's own markup. Pages that don't declare one get `null`.
- **Query-heavy sites.** To avoid crawler traps, at most 50 query-string variants per path and 50 pages per pagination series are visited.

### Responsible usage policy

By running Web2RAG you confirm that you are allowed to collect the content you target.

- **Only crawl content you're authorized to collect.** This means your own websites, sites whose terms permit it, or sites where you have the operator's permission. Follow the site's terms of service and applicable law, including copyright, database rights and data-protection rules such as GDPR and CCPA.
- **Respect the site's wishes.** Keep **Respect robots.txt** turned on. Turn it off only for sites you own or have explicit permission to crawl. `noindex` pages are always excluded.
- **Don't overload websites.** Keep a reasonable delay. Start with small page limits, and stop if you receive access-denied, rate-limit (429) or anti-bot responses.
- **Personal data.** Don't use Web2RAG to collect personal or sensitive information without a valid legal basis. You're responsible for how you store and use the collected data.
- **Prohibited uses:** evading access controls, collecting private or login-protected content, harassment, and any use that violates law or a site's terms.

Built-in safeguards:

- A descriptive User-Agent (`Web2RAG/<version>`).
- Request throttling and robots.txt compliance on by default.
- Same-site crawling by default.
- Strict page, depth and response-size limits.
- Network protections that block requests to private, local and cloud-metadata addresses, including through redirects.

### FAQ and troubleshooting

**The run finished with 0 chunks.** Open `MANIFEST.json`.

- `failures`: the pages may have returned errors.
- `skipped_pages.robots_txt`: the site's robots.txt disallows crawling.
- `empty_pages`: the site probably renders its content with JavaScript.

**Some pages are missing.** They may sit deeper than **Maximum crawl depth**, beyond **Maximum pages**, on another subdomain, or be disallowed by robots.txt. Every skip reason is counted in the report.

**The run stopped early with "Stopped at your maximum cost per run".** Your **Maximum cost per run** covered only part of the crawl, so Web2RAG stopped instead of doing work you couldn't receive. Everything collected up to that point is saved. Raise the limit in the run options to collect more.

**Can I crawl several sites?** Start one run per site, or call the Actor from code in a loop.

**Can I re-crawl and update my vector store?** Yes. IDs are stable for unchanged content, so upserting by `id` updates changed chunks and leaves unchanged ones alone.

### For developers

Web2RAG's crawler and processing pipeline are plain Python and independent of Apify. The Actor (`src/web2rag/actor.py`) is a thin adapter around `src/web2rag/job.py`. That module defines the public input contract (the same JSON fields as the form above), validation and error messages, and a single `run_job()` entry point. A future SaaS dashboard or HTTP API can reuse it unchanged. Run locally with the [Apify CLI](https://docs.apify.com/cli) using `apify run`.

# Actor input Schema

## `startUrl` (type: `string`):

The public page where crawling starts, for example a documentation home page or a help center. Web2RAG follows links from this page. The address must start with <code>http://</code> or <code>https://</code>. Local and private network addresses are blocked.

## `maxPages` (type: `integer`):

Upper limit on the number of pages requested in this run. Start small (10–20) to check the results for a new site, then increase. Pages blocked by robots.txt do not count toward this limit.

## `maxDepth` (type: `integer`):

How many clicks away from the start URL to go. <code>0</code> = only the start page, <code>1</code> = the start page and the pages it links to, and so on. Pagination ("next page" links) is followed regardless of depth, up to 50 pages per listing.

## `outputFormat` (type: `string`):

Format of the extra download file saved in the run's key-value store. Every chunk is <strong>always</strong> saved in the run's Dataset too, which can be exported as JSON, JSONL, CSV, Excel, XML or HTML.

## `chunkSize` (type: `integer`):

Maximum length of each text chunk (one dataset item). Chunks end at sentence boundaries whenever possible. About 1000 characters (≈200–250 English words) suits most embedding models. Use smaller chunks for precise retrieval and larger ones for more context. You are billed per chunk, so larger chunks also cost less per page.

## `chunkOverlap` (type: `integer`):

How much text from the end of one chunk is repeated at the start of the next, so ideas that cross a boundary are not lost. Whole sentences are repeated when they fit. Must be at most half of the chunk size. Set to 0 to disable.

## `sameDomainOnly` (type: `boolean`):

Only follow links on the start URL's website. <code>www.example.com</code> and <code>example.com</code> count as the same site; other subdomains such as <code>blog.example.com</code> do not. Turn this off only if you are allowed to collect the external websites it links to.

## `respectRobotsTxt` (type: `boolean`):

Skip pages that the website's <code>robots.txt</code> file disallows, and honor its <code>Crawl-delay</code>. Strongly recommended. Turn this off only for websites you own or have explicit permission to crawl. Pages marked <code>noindex</code> are always excluded.

## `requestDelay` (type: `number`):

Minimum wait between two requests to the same website. Higher values are gentler on the site but make the run take longer. If the site's robots.txt asks for a longer delay, that delay is used instead (up to 30 seconds).

## `requestTimeout` (type: `integer`):

How long to wait for a page to respond before giving up. Timeouts and temporary server errors are retried twice before the page is recorded as failed.

## Actor input object example

```json
{
  "startUrl": "https://example.com",
  "maxPages": 20,
  "maxDepth": 2,
  "outputFormat": "jsonl",
  "chunkSize": 1000,
  "chunkOverlap": 150,
  "sameDomainOnly": true,
  "respectRobotsTxt": true,
  "requestDelay": 0.5,
  "requestTimeout": 15
}
```

# Actor output Schema

## `chunks` (type: `string`):

Every text chunk as a structured record. Export it as JSON, JSONL, CSV or Excel from the Dataset tab.

## `manifest` (type: `string`):

Pages discovered, processed, failed and skipped (with reasons), duplicate and empty pages, chunk count and duration.

## `files` (type: `string`):

The export in your chosen format (web2rag-output.jsonl, .json or .md) and the crawl report.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://example.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("mdkhadikul/web2rag").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrl": "https://example.com" }

# Run the Actor and wait for it to finish
run = client.actor("mdkhadikul/web2rag").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://example.com"
}' |
apify call mdkhadikul/web2rag --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,mdkhadikul/web2rag"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/eivka9qeMQW1t0pwF/builds/awHNEgb44vqhZYZmY/openapi.json