RAG Web Browser

RAG Web Browser crawls public web pages and returns clean, retrieval-ready text chunks for vector databases, semantic search, question answering, and AI agents. It is designed for developers who need a predictable web-to-RAG data surface without receiving navigation bars, scripts, cookie banners, or other page boilerplate in every chunk.

What it returns

Every dataset item is one homogeneous RAG chunk:

Field	Description
content	Clean page text suitable for embedding or retrieval.
sourceUrl	URL that produced the chunk.
canonicalUrl	Page canonical URL when the source declares one.
title	Page title from Open Graph, the title element, or the first heading.
headingPath	The page headings found near the extracted content.
chunkIndex / totalChunks	Position of the chunk within its source page.
wordCount	Number of whitespace-separated words in the chunk.
crawlDepth	Link depth from the supplied start URL.
fetchedAt	ISO timestamp for the fetch.

The OUTPUT key-value record contains pagesFetched, chunksReturned, failedRequests, and warnings. Diagnostics stay out of the homogeneous dataset so downstream embedding jobs can process every row consistently.

When to use it

Use this Actor when you have one or more public URLs and want clean text for a RAG ingestion pipeline. It works well for documentation pages, knowledge bases, public articles, standards, manuals, product information, and ordinary HTML sites. Set crawlLinks to true when you want to follow links on the same host and use maxPages and maxDepth to bound the crawl.

Do not use it for authenticated applications, private intranets, JavaScript-only applications, sites that require a browser session, or targets where you do not have permission to crawl. It does not bypass login walls, paywalls, robots restrictions, or anti-bot challenges. Use a browser-based Actor or an authorized provider when the target requires rendering or credentials.

Input example: one page

{
  "startUrls": [{ "url": "https://docs.example.com/getting-started" }],
  "crawlLinks": false,
  "maxPages": 1,
  "chunkSize": 1200,
  "chunkOverlap": 150
}

Input example: bounded documentation crawl

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "crawlLinks": true,
  "maxPages": 25,
  "maxDepth": 2,
  "chunkSize": 1200,
  "chunkOverlap": 150,
  "requestTimeoutSecs": 45
}

maxPages is the total page-fetch cap across the run. chunkSize is a character target rather than a token count; chunks prefer paragraph boundaries and long paragraphs are split at the configured bound. chunkOverlap repeats trailing context between adjacent chunks. The Actor clamps runtime values to safe limits even when an API caller bypasses the Console form.

Output example

{
  "chunkId": "aHR0cHM6Ly9kb2NzLmV4YW1wbGUuY29tLw-0",
  "sourceUrl": "https://docs.example.com/getting-started",
  "canonicalUrl": "https://docs.example.com/getting-started",
  "title": "Getting started",
  "description": "Install and configure the SDK.",
  "headingPath": ["Getting started", "Installation"],
  "content": "Getting started\n\nInstall the SDK ...",
  "contentType": "web-page",
  "chunkIndex": 0,
  "totalChunks": 3,
  "wordCount": 184,
  "crawlDepth": 0,
  "fetchedAt": "2026-08-13T18:00:00.000Z"
}

Pricing

Pricing is pay per event: $0.003 per returned RAG chunk plus a $0.00005 run-start event, with any Apify platform usage shown by the platform. The Actor has a $0.30 maximum total event charge per run. A one-page run typically returns a small number of chunks, while long standards or manuals can return more because each chunk is a separate dataset item. Set maxPages and chunkSize to control the result volume and cost.

Reliability and limits

The Actor uses a fast HTTP crawler for static HTML. It retries failed requests twice, limits concurrency, preserves valid partial results, and writes a terminal summary for every run. A malformed URL is rejected by the input schema. A valid page with no readable text completes with a warning and no fabricated chunk. If every request fails before a page is fetched, the run reports a failure rather than presenting an empty dataset as successful data.

For responsible use, crawl only public pages you are authorized to access, follow the target site's terms and robots policies, and avoid collecting personal or restricted information. The Actor is private during development and is not published automatically.