Web2RAG – Website to RAG Dataset

Turn a public website into clean, chunked text that is ready for embeddings, vector databases and AI assistants. You don't need to write any code.

Give Web2RAG a starting URL. It crawls the site politely, extracts the readable content of each page, and removes navigation, ads, cookie banners and duplicate pages. It then splits the text into sentence-aware chunks. Every chunk becomes one record in the run's Dataset, with its source URL and page metadata. Download it as JSON, JSONL, CSV or Excel, or load it straight into Pinecone, Weaviate, Qdrant, pgvector, LangChain or LlamaIndex.

Why Web2RAG?

• RAG-ready out of the box. Chunks are sized for embedding models and never cut a sentence in half. Context overlaps between chunks, and headings stay with their section.
• Clean text. Navigation, footers, sidebars, scripts, ads, cookie banners and hidden elements are removed. Headings and paragraphs are kept.
• No duplicates. Identical and near-identical pages, and boilerplate repeated across pages, are dropped.
• Honest metadata. Title, description, language, published date and canonical URL are taken from the page. A missing value stays empty. Nothing is invented.
• Stable IDs. Each chunk ID is derived from its source and content. Re-running the crawl and upserting into your vector store doesn't create duplicates.
• Respectful by default. Web2RAG obeys robots.txt (including Crawl-delay), honors noindex, throttles requests, stays on the starting website and identifies itself.
• Transparent. Every run produces a crawl report listing which pages were processed, skipped or failed, and why.

Typical uses: a support chatbot built on your help center, internal search over documentation, a knowledge base for an AI agent, training or evaluation corpora from your own content, and content audits.

How to use

1. Start URL. Enter the page to start from, such as https://docs.yourproduct.com/. Starting from a section (for example /docs/) keeps the crawl focused.
2. Size the crawl. Choose Maximum pages and Maximum crawl depth. For a new site, try 10–20 pages first and check the results.
3. Choose an output format for the extra download file: JSONL (recommended), JSON or Markdown. The Dataset always contains every chunk, whichever format you pick.
4. Adjust chunking if needed. The defaults (1000 characters with 150 overlap) work well for most embedding models.
5. Leave the crawl rules at their defaults unless you own the site or have permission to change them.
6. Click Start. The status bar shows live progress, and a summary appears when the run finishes.
7. Get your data.
   • Output tab or Storage → Dataset: preview the chunks and export them as JSON, JSONL, CSV, Excel, XML or HTML.
   • Storage → Key-value store: web2rag-output.jsonl (or .json/.md) and the crawl report MANIFEST.json.
   • API: GET https://api.apify.com/v2/datasets/<datasetId>/items?format=jsonl

Input

Field	Default	What it does
Start URL	(required)	Public http(s):// page to start from. Private and local network addresses are rejected.
Maximum pages	20	Upper limit on pages requested (1–5000). Pages blocked by robots.txt don't count.
Maximum crawl depth	2	Clicks away from the start URL (0–20). "Next page" pagination is followed regardless of depth, up to 50 pages per listing.
Output format	JSONL	Format of the extra download file: JSONL, JSON or Markdown.
Chunk size	1000	Maximum characters per chunk (100–10 000). Chunks end on sentence boundaries.
Chunk overlap	150	Characters repeated between neighbouring chunks (0 to half the chunk size).
Stay on the starting website	on	Follow only links on the same site. www. and the bare domain count as the same site; other subdomains don't.
Respect robots.txt	on	Skip disallowed pages and honor Crawl-delay.
Delay between requests	0.5 s	Minimum wait between requests to the site (0–60 s). A longer robots.txt Crawl-delay wins, up to 30 s.
Request timeout (Advanced)	15 s	How long to wait for a page. Timeouts and temporary errors are retried twice.

Invalid settings stop the run immediately with a plain-language message. For example: "Chunk overlap (300) must be at most half of the chunk size (400)."

Input examples

Documentation site (recommended starting point)

{
  "startUrl": "https://docs.example.com/",
  "maxPages": 200,
  "maxDepth": 3,
  "outputFormat": "jsonl",
  "chunkSize": 1000,
  "chunkOverlap": 150
}

Quick test of a new site

{
  "startUrl": "https://www.example.com/help/",
  "maxPages": 10,
  "maxDepth": 1
}

Small chunks for precise Q&A retrieval, gentler crawling

{
  "startUrl": "https://example.com/faq",
  "maxPages": 50,
  "chunkSize": 500,
  "chunkOverlap": 80,
  "requestDelay": 2
}

Human-readable Markdown export of a blog (pagination is followed automatically)

{
  "startUrl": "https://example.com/blog/",
  "maxPages": 100,
  "maxDepth": 1,
  "outputFormat": "markdown"
}

Start a run from code (Python, apify-client). Replace <username>/web2rag with this Actor's ID:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("<username>/web2rag").call(run_input={"startUrl": "https://docs.example.com/", "maxPages": 100})
for chunk in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(chunk["source_url"], chunk["content"][:80])

Output

Dataset: one item per chunk

{
  "id": "b07cf53371f34151cc02916a3977fabe",
  "source_url": "https://docs.example.com/getting-started",
  "title": "Getting started",
  "content": "Installation\nDownload the installer from the customer portal and run it with administrator rights. The installer checks your system requirements before copying any files.",
  "chunk_index": 0,
  "language": "en",
  "metadata": {
    "description": "Install and run the product.",
    "published_date": "2025-01-15",
    "canonical_url": "https://docs.example.com/getting-started"
  },
  "collected_at": "2026-11-04T17:16:30+00:00"
}

Field	Meaning
id	Stable chunk ID. Use it as the vector-store primary key.
source_url	The page the chunk came from, after redirects. Use it for citations.
title	The page's title, or "" if it has none.
content	Clean chunk text. Headings sit on their own lines.
chunk_index	Position of the chunk within its page (0, 1, 2, …).
language	The language the page declares (en, es-MX, …), or null.
metadata.description	The page's meta description, or "".
metadata.published_date	Publication date as stated on the page, or null. A date-only value stays a date.
metadata.canonical_url	The page's declared canonical URL, or null.
collected_at	When the page was fetched (UTC, ISO 8601).

The Dataset tab has three views: Chunks (readable overview), Page metadata, and Full records. CSV and Excel exports flatten metadata into metadata/description and similar columns.

Download file (key-value store)

web2rag-output.jsonl, web2rag-output.json or web2rag-output.md contains the same records in the format you chose. The Markdown file groups chunks under each page's title and source URL, which makes it easy to review.

Crawl report: MANIFEST.json

{
  "start_url": "https://docs.example.com/",
  "total_pages_discovered": 18,
  "pages_requested": 16,
  "successfully_processed_pages": 11,
  "pages_with_content": 9,
  "failed_pages": 2,
  "skipped_pages": { "robots_txt": 2, "not_html": 1, "off_site_redirect": 0, "noindex": 1,
                     "unsafe_url": 0, "redirect_to_crawled_page": 1 },
  "empty_pages": 1,
  "duplicate_count": 1,
  "duplicate_chunks": 0,
  "total_chunks": 16,
  "export_format": "jsonl",
  "processing_duration_seconds": 9.4,
  "failures": [ { "url": "https://docs.example.com/old", "outcome": "failed", "reason": "HTTP 404" } ]
}

The report also records the settings used and the Web2RAG version, so you can reproduce a run.

Using the data

import json
records = [json.loads(line) for line in open("web2rag-output.jsonl", encoding="utf-8")]
texts = [r["content"] for r in records]                       # embed these
metadatas = [{"source": r["source_url"], "title": r["title"]} for r in records]
ids = [r["id"] for r in records]                              # stable upsert keys

Pricing

Web2RAG uses pay-per-event pricing: you pay only for the RAG chunks you receive.

What you pay for	Price
Each chunk saved to the dataset	$0.0002, which is $0.20 per 1,000 chunks
Run start (Apify's standard fee)	about $0.00005 per run

• Platform usage is included. You aren't billed separately for compute, storage or data transfer.
• You only pay for results. Pages that fail, are skipped (robots.txt, noindex, not HTML), are duplicates or have no text don't create chunks, so they cost nothing.
• Typical cost: with the default chunk size (1000 characters), a page usually yields 4–7 chunks, so 1,000 pages cost about $1. For example, a real 20-page crawl of an e-commerce site produced 133 chunks, which cost $0.027.
• Larger chunks are cheaper. A bigger Chunk size produces fewer chunks per page.
• Stay in control: set Maximum cost per run in the run options. Web2RAG produces only as many chunks as that budget covers, then stops and keeps everything collected so far.
• Try it free: the Apify free plan includes $5 of monthly credit, enough for about 25,000 chunks (roughly 4,000–6,000 pages).

Start with 10–20 pages to see how many chunks a site produces. Any future price change will be announced on this page in advance.

Limitations

• No JavaScript rendering. Pages that build their content in the browser (some single-page apps) come out empty and are counted as empty_pages in the report. Try the site's server-rendered docs, or another URL.
• HTML only. PDFs, Office documents, images and other files are skipped.
• No access bypass. Logins, paywalls, CAPTCHAs and anti-bot protections are never circumvented. Pages answering 401 or 403 are reported as failed.
• One site at a time, politely. Requests are sequential with a delay, so very large sites take longer. The maximum is 5000 pages per run.
• Scope rules. Other subdomains (for example blog.example.com) aren't crawled from example.com. Start a separate run for them. If the start URL redirects (for example to www.), the crawl follows the final address.
• Robots.txt failures are strict. If a site's robots.txt can't be read (server error, 403 or timeout), Web2RAG crawls nothing on that site rather than guess. A missing robots.txt (404) means no restrictions.
• Language detection. The language comes from the page's own markup. Pages that don't declare one get null.
• Query-heavy sites. To avoid crawler traps, at most 50 query-string variants per path and 50 pages per pagination series are visited.

Responsible usage policy

By running Web2RAG you confirm that you are allowed to collect the content you target.

• Only crawl content you're authorized to collect. This means your own websites, sites whose terms permit it, or sites where you have the operator's permission. Follow the site's terms of service and applicable law, including copyright, database rights and data-protection rules such as GDPR and CCPA.
• Respect the site's wishes. Keep Respect robots.txt turned on. Turn it off only for sites you own or have explicit permission to crawl. noindex pages are always excluded.
• Don't overload websites. Keep a reasonable delay. Start with small page limits, and stop if you receive access-denied, rate-limit (429) or anti-bot responses.
• Personal data. Don't use Web2RAG to collect personal or sensitive information without a valid legal basis. You're responsible for how you store and use the collected data.
• Prohibited uses: evading access controls, collecting private or login-protected content, harassment, and any use that violates law or a site's terms.

Built-in safeguards:

• A descriptive User-Agent (Web2RAG/<version>).
• Request throttling and robots.txt compliance on by default.
• Same-site crawling by default.
• Strict page, depth and response-size limits.
• Network protections that block requests to private, local and cloud-metadata addresses, including through redirects.

FAQ and troubleshooting

The run finished with 0 chunks. Open MANIFEST.json.

• failures: the pages may have returned errors.
• skipped_pages.robots_txt: the site's robots.txt disallows crawling.
• empty_pages: the site probably renders its content with JavaScript.

Some pages are missing. They may sit deeper than Maximum crawl depth, beyond Maximum pages, on another subdomain, or be disallowed by robots.txt. Every skip reason is counted in the report.

The run stopped early with "Stopped at your maximum cost per run". Your Maximum cost per run covered only part of the crawl, so Web2RAG stopped instead of doing work you couldn't receive. Everything collected up to that point is saved. Raise the limit in the run options to collect more.

Can I crawl several sites? Start one run per site, or call the Actor from code in a loop.

Can I re-crawl and update my vector store? Yes. IDs are stable for unchanged content, so upserting by id updates changed chunks and leaves unchanged ones alone.

For developers

Web2RAG's crawler and processing pipeline are plain Python and independent of Apify. The Actor (src/web2rag/actor.py) is a thin adapter around src/web2rag/job.py. That module defines the public input contract (the same JSON fields as the form above), validation and error messages, and a single run_job() entry point. A future SaaS dashboard or HTTP API can reuse it unchanged. Run locally with the Apify CLI using apify run.