# RAG Web Browser - Web Search to Markdown for AI (`tidytools/rag-web-browser`) Actor

Search the web (DuckDuckGo, Bing) and read the top pages as Markdown for LLMs and RAG, or read a list of URLs. Searching is free; blocked pages are free.

- **URL**: https://apify.com/tidytools/rag-web-browser.md
- **Developed by:** [Yukai Lin](https://apify.com/tidytools) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RAG Web Browser

### What it does

Search the web and return up to 1–10 readable results as clean Markdown for LLMs, AI agents and RAG pipelines, or read a list of URLs. Search result pages are rendered in a real Chrome (DuckDuckGo first, then Bing) and parsed into ranked results; each candidate page is then read with the TidyTools Smart Fetch backend. Searching is free: you pay only for pages returned as Markdown.

Search engines' terms of use generally do not allow automated querying; reading their result pages is a business risk the operator has accepted. Verification pages (CAPTCHAs, "unusual traffic" checks) are never solved or clicked: that attempt is marked blocked, free of charge, and the next network or engine is tried.

### How to use

1. Type search words in **Search query or URL** (or paste page links in **URLs to read**).
2. Set **Maximum pages** (1–10) and start the run.
3. Read the pages from the Dataset (`markdown` column) or fetch them through the Apify API / MCP in your RAG pipeline.

### Input

```json
{"query":"best open source vector database","maxResults":3}
```

```json
{"urls":["https://docs.python.org/3/library/asyncio.html","realpython.com/async-io-python/"]}
```

| Field | Default | Meaning |
|---|---|---|
| `query` | – | Search words, or an `http(s)` URL to read one page. |
| `maxResults` | 3 | Return up to N readable pages (1–10). Failed pages are followed by the next ranked candidate until N succeed or candidates run out. |
| `urls` | – | Read these pages instead of searching (up to 100; pasted text with one URL per line or comma-separated also works; bare domains get `https://`). `query` is ignored. Invalid lines become free rows with `errorType: "invalid_input"`; duplicates are read once. |
| `url` | – | Read this one page directly (same as one line in `urls`). |
| `engine` | `auto` | `auto` tries DuckDuckGo → Bing; `duckduckgo`, `bing` or `google` uses only that engine. Google answers automated searches with its "unusual traffic" verification page almost every time, so it is not part of `auto`. |
| `searchEgress` | `auto` | Network for search result pages: `auto` (data-center network first, residential network if the engine blocks it), `warp` or `resi`. The residential network is used on paid Apify plans only, within a per-search allowance tied to the pages you can be charged for (at most half of `maxResults` × the page price); once that is used up the search continues on the data-center network or the next engine. On the free plan every option uses the data-center network, so searches are more often blocked there. Never the operator's home connection. |

Only public HTTP(S) URLs are read. Ads, tracking redirects and the engines' own links are dropped; DuckDuckGo (`/l/?uddg=`), Bing (`/ck/a`) and Google (`/url?q=`) redirect wrappers are decoded to the real URL, tracking parameters (`utm_*`, `gclid`, `fbclid`, `msclkid`) are removed, and duplicates (same host without `www.`, path and query) are dropped.

### Output

One Dataset row per page:

```json
{"url":"https://realpython.com/async-io-python/","title":"Python's asyncio: A Hands-On Walkthrough – Real Python","rank":1,"engine":"duckduckgo",
 "searchResult":{"rank":1,"title":"Python's asyncio: A Hands-On Walkthrough – Real Python","snippet":"Python's asyncio library enables you to write concurrent code …","engine":"duckduckgo"},
 "markdown":"# Python's asyncio: A Hands-On Walkthrough …","fetchedAt":"2026-10-22T05:52:10.000Z","success":true,"charged":null,"chargedEvents":null}
```

| Field | Meaning |
|---|---|
| `url` | The page that was read. |
| `title` | The page's own title (the search result title when the page has none). |
| `rank` | Position in the search results (1 = first), `null` for URLs you gave. |
| `engine` | `duckduckgo`, `bing` or `google`; `null` for URLs you gave. |
| `searchResult` | `rank`, `title`, `snippet` and `engine` as the search result page showed them; `null` for URLs you gave. |
| `markdown` | The page's main content as Markdown (navigation, footers and boilerplate removed). `null` when the page could not be read. |
| `fetchedAt` | When the page was read (ISO 8601). |
| `input` | The original line, for rows from `urls`. |
| `success` | `true` when the page returned usable Markdown. |
| `charged`, `chargedEvents` | `false` and `[]` on failed rows. Successful rows are saved before billing, so these are `null` (pending); `SUMMARY.chargedPages` has the result. |
| `errorType`, `error` | On failed rows only, e.g. `blocked`, `robots_disallowed`, `empty_page`, `invalid_input`, `search_blocked`. |

If every engine is blocked, one free row with `errorType: "search_blocked"` is written.

For a search, the Dataset also records free failed page rows. They do not count toward `maxResults`; the Actor tries the next ranked candidate. Successful rows retain their original search rank, so ranks can have gaps. `SUMMARY.pages` counts successful pages, `SUMMARY.failed` counts failed attempts, and `SUMMARY.status` is `SUCCESS` when the requested number of readable pages is reached. URL lists retain their original behavior.

`SUMMARY` (key-value store) has `status` (`SUCCESS`, `PARTIAL`, `NO_RESULTS`, `LIMIT_REACHED`), `pages`, `failed`, `chargedPages` (each saved page URL with its actual charged events), `costPlan`, `notProcessed` when the max charge per run stopped the run, `invalidInputs`/`duplicates` for URL lists and, for searches, `search`: `engine`, `egress`, `costTier`, `results` (number of ranked candidates), `resiBytes` and `attempts` (one entry per result page tried: `engine`, `egress`, `status`, `blockType`, `ms`, `bytes`, `resiBytes`, `results`). Block types: `captcha_unusual_traffic`, `captcha_interactive`, `consent_wall`, `js_challenge_not_passed`, `http_<status>`, `no_results`, `no_results_parsed`, `irrelevant_results`, `too_few_results`, `empty_page`, `limiter_unavailable`.

### Control your cost

No start fee. Searching is free. Each usable page returned as Markdown is one `page-http` event. Verification pages, blocked, empty and failed pages, blocked searches and invalid inputs are free. At the start of a run the log and status show the worst case (number of pages × page price). When your max charge per run is reached, the remaining pages are not read and are listed in `SUMMARY.notProcessed`.

### Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/rag-web-browser) to Claude, Cursor or any MCP client, then ask e.g. "Search for python asyncio tutorials and summarise the top 3 pages." Minimal input: `{"query":"python asyncio tutorial"}`. Failed items are not charged and carry an `errorType`.

### FAQ

**Am I charged for blocked searches or failed pages?** No. Only pages returned as usable Markdown are charged (`page-http`); verification pages, blocked searches, empty and failed pages are free.

**Does it solve CAPTCHAs?** No. A verification page marks that attempt as blocked and the next network or engine is tried.

**Can I read my own list of pages instead of searching?** Yes: put them in `urls` (up to 100) or `url`; `query` is then ignored.

### Limitations

- DuckDuckGo searches over the shared data-center network reserve a cross-run slot at about one query per 7 seconds. When a slot cannot be reserved, auto mode tries the next available network or Bing; no verification is clicked or solved. Search remains free. On 2026-10-23, 20 consecutive live DuckDuckGo/WARP searches through this limiter returned 20/20 usable result pages. The 2026-10-22 baseline without pacing was 7/20; with manual spacing it was 20/20. See `research/rag-serp-measure-2026-10-22.md`.
- Pages are read over plain HTTP from Cloudflare with robots rules respected; sites that refuse that route (for example HTTP 403) return a free failed row.
- Up to 10 successful search pages per query and 100 URLs per run; search candidates are tried in ranked order, while URL lists are read 4 at a time.

# Actor input Schema

## `query` (type: `string`):

Search the public web (DuckDuckGo, then Bing) and read the top results as Markdown, or enter an http(s) URL to read that one page.

## `maxResults` (type: `integer`):

Return up to N readable pages (1–10), trying later search results if a page fails. Ignored for URLs.

## `urls` (type: `array`):

Read these pages as Markdown instead of searching (up to 100; one per line, bare domains get https://). When set, query is ignored. Invalid lines become free rows.

## `url` (type: `string`):

Read this one page directly; query and maxResults are ignored. Same as putting one line in URLs.

## `engine` (type: `string`):

Blocked engines are free (verification pages are never solved) and the next engine is tried.

## `searchEgress` (type: `string`):

Only used for search result pages. The residential network is used on paid Apify plans only, within a per-search allowance (half of maxResults x the page price); on the free plan every option uses the data-center network.

## Actor input object example

```json
{
  "query": "best open source vector database",
  "maxResults": 3,
  "engine": "auto",
  "searchEgress": "auto"
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "best open source vector database"
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidytools/rag-web-browser").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "query": "best open source vector database" }

# Run the Actor and wait for it to finish
run = client.actor("tidytools/rag-web-browser").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "best open source vector database"
}' |
apify call tidytools/rag-web-browser --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidytools/rag-web-browser"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fKOhB9ri7Mo9XsfqC/builds/4XpcLpLVj8mX6MH8Y/openapi.json