# RAG Web Browser — URL & Website to Markdown for LLM & RAG (`s-r/rag-web-browser`) Actor

Search the web or pass URLs, get clean Markdown, plain text or HTML for LLM and RAG pipelines. Follows links, streams results, no per-run fee.

- **URL**: https://apify.com/s-r/rag-web-browser.md
- **Developed by:** [SR](https://apify.com/s-r) (community)
- **Categories:** AI, Developer tools
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 url converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RAG Web Browser: URL & Website to Markdown for LLM and RAG

Give it a search query or a list of URLs. Get back clean Markdown, plain text or HTML, ready to chunk and embed. It reads the live web for your LLM the way a search tool inside a chat assistant does, without you running browsers, proxies or a bot-detection arms race.

Point it at one page, a whole site, or a question you want answered from current sources.

### What it does

- **Search the web, then read the results.** Pass a query and it fetches the top N results and converts each page to Markdown. One step instead of a SERP scraper plus a content scraper.
- **Convert known URLs in bulk.** Pass a list and it processes them in parallel, streaming rows into the dataset as each finishes rather than making you wait for the slowest page.
- **Crawl a site.** Set a depth and it follows in-content links, stays on the domain unless you say otherwise, and stops at a hard page cap you control.
- **Three output formats.** Markdown, plain text, and HTML, any combination, on the same row.
- **Pay per page, no run fee.** Several actors in this category charge a fixed fee every time a run starts, on top of per-result pricing. This one does not, and it does not charge for pages it failed to fetch.

### Quick start

Search the web and read the top 5 results:

```json
{
  "query": "best vector database for RAG",
  "maxResults": 5
}
```

Convert specific pages, with plain text alongside the Markdown:

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "https://example.com/pricing"
  ],
  "outputFormats": ["markdown", "text"]
}
```

Crawl a documentation site, capped at 50 pages:

```json
{
  "urls": ["https://docs.example.com/getting-started"],
  "crawlDepth": 2,
  "maxCrawlPages": 50,
  "sameDomainOnly": true,
  "excludeUrlPatterns": ["/tag/", "/author/", "?replytocom"]
}
```

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | — | Search the web and convert the top results. A single URL pasted here is fetched directly instead. |
| `urls` | array | — | Specific pages to convert. Bare domains like `example.com` are accepted. Works alongside `query`. |
| `maxResults` | integer | `5` | How many search results to fetch. Search mode only. |
| `searchLanguage` | string | — | Two-letter code (`en`, `de`, `nl`) to narrow results by language. |
| `outputFormats` | array | `["markdown"]` | Any of `markdown`, `text`, `html`. |
| `includeMetadata` | boolean | `true` | Adds `title`, `description`, `publishedTime`, `finalUrl`. |
| `minContentLength` | integer | `200` | Pages thinner than this are flagged, not dropped. |
| `crawlDepth` | integer | `0` | `0` fetches only what you asked for. `1` also follows links found in those pages. Max `3`. |
| `maxCrawlPages` | integer | `10` | Hard ceiling on pages fetched during a crawl. This is your cost cap. |
| `sameDomainOnly` | boolean | `true` | Keep the crawl on the starting domain. |
| `excludeUrlPatterns` | array | — | Skip URLs containing any of these substrings. Case-insensitive. |
| `concurrency` | integer | `10` | Pages in parallel. See the note below before raising it. |
| `requestTimeoutSecs` | integer | `30` | Per-attempt timeout for one page. |
| `hedgeDelaySecs` | integer | `4` | If the first extraction backend has not answered in this many seconds, a second is tried in parallel and the first good result wins. |
| `maxRetries` | integer | `2` | Retries per backend for a rate-limited or failed page. |

Every field has a default. The minimum viable input is a `query` or a single URL.

### Output

One row per page:

```json
{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "finalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation",
  "description": "Retrieval-augmented generation (RAG) is a technique that enables large language models to retrieve and incorporate new information from external data sources.",
  "publishedTime": "2023-11-05T13:19:20Z",
  "markdown": "From Wikipedia, the free encyclopedia\n\n**Retrieval-augmented generation**...",
  "wordCount": 2100,
  "length": 14822,
  "provider": "provider_a",
  "fetchedInSeconds": 0.61
}
```

Search results also carry `searchQuery`, `searchRank`, `searchTitle`, `searchSnippet` and `searchEngine`. Crawled pages carry `crawlDepth` and `seedUrl`. A page that could not be fetched comes back as a row with an `error` explaining why, so a partial failure never silently shrinks your dataset.

Results are pushed as each page completes, so you can start reading the dataset while the run is still going. Download as JSON, JSONL, CSV or Excel, stream via the Apify API, or pipe to a webhook, S3, or BigQuery.

### Notes worth reading before you scale up

**Concurrency.** The default of 10 is a measured throughput peak, not a conservative guess. The extraction backends rate-limit per source IP: a 15-page burst at concurrency 10 from a single IP gets throttled on 14 of 15 requests. This Actor handles that by moving a throttled backend onto rotating egress for the rest of the run, which is why a 30-page batch completes in full rather than losing a handful of pages. Pushing concurrency to 25 made throughput *worse* in testing (3.4 pages/sec at 10, 1.1 at 25). Raise it only if you measure a gain.

**HTML output** is rendered from the extracted Markdown. It is clean, structured HTML, but it is not the origin page's own markup, which the extraction step has already discarded along with the nav bars and cookie banners.

**Crawl links** come from the extracted Markdown, so they are in-content links. Navigation chrome that the extractor stripped is not followed. For a RAG crawl that is usually what you want.

**Where search results come from.** Search mode queries a web index and each row records its `searchEngine`. It is not Google's index, and this Actor does not claim to be. If you need Google specifically, pair a dedicated SERP Actor with this one in URL mode.

### How it compares

| | This Actor | Firecrawl | Tavily | Website Content Crawler |
|---|---|---|---|---|
| Run without your own API key | Yes | No, needs a Firecrawl key | No, needs a Tavily key | Yes |
| Search query as input | Yes | Yes | Yes | No, URL only |
| Markdown, text and HTML | Yes | Yes | Markdown | Yes |
| Crawl with a hard page cap | Yes | Yes | Limited | Yes |
| Per-run start fee | **None** | Monthly credits | Monthly credits | Platform compute |
| Charges for failed pages | **No** | Varies | Varies | Compute is billed regardless |

If you already run jobs on Apify, this keeps web-to-Markdown in the same account, dataset format and billing as everything else.

### FAQ

#### How is this different from a plain URL to Markdown converter?

A converter takes one URL and returns one document. This takes a *question* and returns the current web's answer to it as documents, or takes a site and returns the whole readable surface of it. Search mode and crawl mode are the difference, and both feed the same clean row shape.

#### Why does my page come back with a `contentWarning` instead of content?

It has content, just less than `minContentLength` (200 characters by default). Genuinely short pages, stubs and brief news items are returned flagged rather than thrown away. Lower `minContentLength` to silence it, or raise it to be stricter about thin pages.

#### What happens when a page is behind a bot wall?

The Actor detects interstitials (Cloudflare challenges, "enable JavaScript" stubs, captcha pages) and refuses to pass them off as content. It tries the other extraction backend, and if the wall holds, the row comes back with an `error` naming the reason. You are not charged for it.

#### Can I use it as an HTTP endpoint instead of starting a run?

Yes. Standby mode serves the same pipeline over HTTP, so an agent can request a page mid-conversation without paying Actor start latency each call. Pass the same fields as query parameters, for example `?query=vector+databases&maxResults=3` or `?urls=https://a.com&urls=https://b.com`.

#### How much does a run cost?

Pay-per-event: you are billed per page successfully converted, with no monthly minimum and no fee for starting a run. Failed pages and pages dropped by a plan limit are not billed. A 100-page crawl costs 100 page events. See the Store page for the current per-page price.

#### Can I schedule it?

Yes, via Apify's built-in scheduler. A common setup is a nightly crawl of a docs site with `crawlDepth` set, diffing against the previous run to re-embed only what changed.

#### Is there a free tier?

Free-plan Apify accounts get 10 results per run. Paid plans have no cap. The limit is per run, so free users can still evaluate every feature.

# Actor input Schema

## `query` (type: `string`):

Search the web and convert the top results to Markdown. Leave empty if you already know the URLs. Pasting a single URL here works too — it is fetched directly instead of searched.

## `urls` (type: `array`):

Specific pages to convert. Use this instead of (or together with) a search query. Bare domains like example.com are accepted.

## `maxResults` (type: `integer`):

How many top search results to fetch when a search query is given. Ignored in URL-only mode.

## `searchLanguage` (type: `string`):

Two-letter language code to narrow search results, e.g. en, de, nl, fr. Leave empty for no language filter. Note: this filters by language, not by country.

## `outputFormats` (type: `array`):

Which representations to include on every row. Markdown is the extracted content; text is that with the Markdown syntax stripped; HTML is rendered from the Markdown (it is not the origin page's own markup).

## `includeMetadata` (type: `boolean`):

Add title, description, publishedTime and finalUrl columns to each row.

## `minContentLength` (type: `integer`):

Pages with less extracted text than this are flagged with contentWarning rather than dropped. Raise it to be stricter about thin pages.

## `crawlDepth` (type: `integer`):

0 fetches only the pages you asked for. 1 also follows links found in those pages, 2 follows links from those, and so on.

## `maxCrawlPages` (type: `integer`):

Hard ceiling on total pages fetched when crawl depth is above 0. This is your cost cap.

## `sameDomainOnly` (type: `boolean`):

Only follow links that stay on the starting page's domain. www and the bare domain count as the same site.

## `excludeUrlPatterns` (type: `array`):

Skip any URL containing one of these substrings, e.g. /tag/ or ?replytocom. Case-insensitive.

## `concurrency` (type: `integer`):

Pages fetched in parallel. The default of 10 is the measured throughput peak; going much higher trips upstream rate limits and gets slower, not faster.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for a single page before giving up on that attempt.

## `hedgeDelaySecs` (type: `integer`):

If the first content provider has not answered within this many seconds, a second one is tried in parallel and the first good response wins. Lower is faster but issues more upstream requests.

## `maxRetries` (type: `integer`):

Retries per content provider for a rate-limited or failed page.

## Actor input object example

```json
{
  "query": "best vector database for RAG",
  "maxResults": 5,
  "searchLanguage": "en",
  "outputFormats": [
    "markdown"
  ],
  "includeMetadata": true,
  "minContentLength": 200,
  "crawlDepth": 0,
  "maxCrawlPages": 10,
  "sameDomainOnly": true,
  "concurrency": 10,
  "requestTimeoutSecs": 30,
  "hedgeDelaySecs": 4,
  "maxRetries": 2
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `errors` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "best vector database for RAG",
    "maxResults": 5,
    "searchLanguage": "en",
    "outputFormats": [
        "markdown"
    ],
    "includeMetadata": true,
    "minContentLength": 200,
    "crawlDepth": 0,
    "maxCrawlPages": 10,
    "sameDomainOnly": true,
    "concurrency": 10,
    "requestTimeoutSecs": 30,
    "hedgeDelaySecs": 4,
    "maxRetries": 2
};

// Run the Actor and wait for it to finish
const run = await client.actor("s-r/rag-web-browser").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "best vector database for RAG",
    "maxResults": 5,
    "searchLanguage": "en",
    "outputFormats": ["markdown"],
    "includeMetadata": True,
    "minContentLength": 200,
    "crawlDepth": 0,
    "maxCrawlPages": 10,
    "sameDomainOnly": True,
    "concurrency": 10,
    "requestTimeoutSecs": 30,
    "hedgeDelaySecs": 4,
    "maxRetries": 2,
}

# Run the Actor and wait for it to finish
run = client.actor("s-r/rag-web-browser").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "best vector database for RAG",
  "maxResults": 5,
  "searchLanguage": "en",
  "outputFormats": [
    "markdown"
  ],
  "includeMetadata": true,
  "minContentLength": 200,
  "crawlDepth": 0,
  "maxCrawlPages": 10,
  "sameDomainOnly": true,
  "concurrency": 10,
  "requestTimeoutSecs": 30,
  "hedgeDelaySecs": 4,
  "maxRetries": 2
}' |
apify call s-r/rag-web-browser --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,s-r/rag-web-browser"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/l3TGYATbHT3anhvzQ/builds/aC8yLE9QfglWrcY2K/openapi.json