# RAG Web Browser (Anti-Bot) - $1.80 / 1000 Pages (`scrapeunblocker/rag-web-browser`) Actor

Search the web and get the top results as clean Markdown, or read any URL, in seconds. Built as a web tool for LLMs, AI agents and RAG. Pages behind Cloudflare, DataDome, PerimeterX or Akamai work. No Google needed. $1.80 per 1000 pages; failed pages are free.

- **URL**: https://apify.com/scrapeunblocker/rag-web-browser.md
- **Developed by:** [Scrapeunblocker](https://apify.com/scrapeunblocker) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.80 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RAG Web Browser (Anti-Bot) - Search to Markdown for LLMs

Give your LLM, AI agent or RAG pipeline live access to the web: send a **search query** and get the **top results already read and converted to clean Markdown**, or send a **URL** and get that page. Every page is loaded through an anti-bot unblocker, so results behind **Cloudflare, DataDome, PerimeterX (HUMAN) or Akamai** come back as real content instead of a challenge screen, which is where ordinary web-browsing tools return nothing.

The input and output use the same field names as the most popular RAG web browser (`query`, `maxResults`, `outputFormats`, `requestTimeoutSecs`; `searchResult`, `metadata`, `markdown`), so existing integrations switch over without code changes.

### Features

- **Search + read in one call** - the top N results of a web search, each as clean Markdown with its title, description and URL.
- **Or read one URL** - put a URL in `query` and get that page's content.
- **Gets through anti-bot walls** - Cloudflare, DataDome, PerimeterX, Akamai.
- **Complete answers** - a result whose page is dead, blocked or empty is replaced by the next search result, so you still get `maxResults` pages.
- **Fast** - search answers in a few seconds and all pages are read in parallel; a typical 3-result run finishes in well under a minute.
- **No Google needed** - searches DuckDuckGo, with Bing and Brave Search started automatically when it is slow or fails; search operators like `site:` and `-site:` work.
- **LLM-friendly Markdown** - main content only, headings, lists, tables, absolute links and fenced code; menus, footers, cookie banners and decorative images removed.
- **Pay only for pages delivered** - skipped results are listed in the `ERRORS` record and never billed.

### Use cases

- **Web search tool for AI agents** (OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, n8n, Make, Zapier).
- **Retrieval-augmented generation** - ground answers in current web pages.
- **Research and fact-checking** - the content behind a search, not just snippets.
- **Reading pages other tools are blocked from** - review sites, marketplaces, job boards, news behind bot protection.

### Input

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `query` | string | `web scraping api` | Search terms (operators like `site:` work), or one `http(s)://` URL to read. |
| `maxResults` | integer | `3` | Search results to read and return (1-20). |
| `outputFormats` | array | `["markdown"]` | Any of `markdown`, `text`, `html` (cleaned content HTML). |
| `requestTimeoutSecs` | integer | `45` | Timeout for one page; a slower page is replaced by the next result. |
| `searchEngine` | string | `auto` | `auto` (DuckDuckGo, with Bing and Brave when it is slow), `duckduckgo`, `bing` or `brave`. |
| `country` | string | US | Country to search from (ISO-2), which sets the results market. |
| `htmlTransformer` | string | `readableText` | `readableText` = main content only, `none` = whole page. |
| `removeElementsCssSelector` | string | nav, footer, scripts, dialogs ... | Elements removed before extraction. |
| `removeCookieWarnings` | boolean | `true` | Remove cookie consent banners. |

#### Example input

```json
{
  "query": "best espresso machine 2026 review",
  "maxResults": 5,
  "outputFormats": ["markdown"]
}
```

Read a single Cloudflare-protected page:

```json
{
  "query": "https://www.glassdoor.com/blog/"
}
```

### Output

One dataset item per page, in search-rank order:

```json
{
  "query": "web scraping api",
  "searchResult": {
    "title": "Web Scraping API for Data Extraction & Web Crawling | Floppydata",
    "description": "Powerful Web Scraping API to extract data from any website...",
    "url": "https://floppydata.com/web-scraping/",
    "position": 1
  },
  "metadata": {
    "title": "Web Scraping API for Data Extraction & Web Crawling | Floppydata",
    "description": "Powerful Web Scraping API to extract data from any website...",
    "languageCode": "en",
    "author": null,
    "url": "https://floppydata.com/web-scraping/",
    "canonicalUrl": "https://floppydata.com/web-scraping/"
  },
  "crawl": {
    "httpStatusCode": 200,
    "loadedAt": "2026-10-12T17:40:12.565916+00:00",
    "requestStatus": "handled"
  },
  "markdown": "# Web Scraping API\n\nNo more scraping blocks, CAPTCHAs, or failed requests...\n\n- Automatically handle blocks, CAPTCHAs, and anti-bot systems\n- ..."
}
```

The key-value store also holds:

- `SEARCH` - the search engine used and all organic results it returned (title, URL, description, position).
- `ERRORS` - results that were skipped (page failed, blocked, not found or empty), with the reason. These are **not billed**.

### Calling it from code

Run it synchronously and get the pages in one HTTP call:

```
POST https://api.apify.com/v2/acts/scrapeunblocker~rag-web-browser/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>
Content-Type: application/json

{ "query": "latest EU AI act news", "maxResults": 3 }
```

### Pricing

**$1.80 per 1000 pages.** You pay only for pages delivered to the dataset; the search itself and skipped results are free.

### Notes

- Only HTML pages are read; PDF and other file results are skipped.
- `country` changes the search market only; pages are always loaded through the most reliable exit.
- Each page is loaded once (no clicking or infinite scrolling).

# Actor input Schema

## `query` (type: `string`):

What to search for, e.g. best espresso machine 2026 (search operators like site:example.com or -site:reddit.com work), or a single page URL starting with http:// or https:// to read just that page.

## `maxResults` (type: `integer`):

How many top search results to read and return (1-20). Results whose page fails or is blocked are replaced by the next search result. Ignored for a URL.

## `outputFormats` (type: `array`):

What to return for every page: Markdown, plain text and/or the cleaned HTML of the content.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one page. A page that takes longer is replaced by the next search result. Raise it for pages behind hard anti-bot challenges.

## `searchEngine` (type: `string`):

Where to search. Automatic asks DuckDuckGo first and, if it is slow or fails, Bing and Brave Search too; the first engine to answer is used.

## `country` (type: `string`):

Optional country (ISO-2, e.g. US, DE, GB) to search from, which sets the results market. Default: US.

## `htmlTransformer` (type: `string`):

Main content keeps the article body and drops menus, sidebars and footers. Whole page keeps everything except the removed elements.

## `removeElementsCssSelector` (type: `string`):

Elements to delete before the content is extracted.

## `removeCookieWarnings` (type: `boolean`):

Drop cookie consent banners from the content.

## Actor input object example

```json
{
  "query": "what is retrieval augmented generation",
  "maxResults": 3,
  "outputFormats": [
    "markdown"
  ],
  "requestTimeoutSecs": 45,
  "searchEngine": "auto",
  "htmlTransformer": "readableText",
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^='data:'],\n[role=\"alert\"],\n[role=\"banner\"],\n[role=\"dialog\"],\n[role=\"alertdialog\"],\n[role=\"region\"][aria-label*=\"skip\" i],\n[aria-modal=\"true\"]",
  "removeCookieWarnings": true
}
```

# Actor output Schema

## `results` (type: `string`):

One item per search result page: the search result, page metadata and its content as Markdown

## `search` (type: `string`):

The engine used and all organic results it returned

## `errors` (type: `string`):

Search results whose page failed, was blocked or had no readable text, with the reason (not billed)

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "what is retrieval augmented generation"
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapeunblocker/rag-web-browser").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "query": "what is retrieval augmented generation" }

# Run the Actor and wait for it to finish
run = client.actor("scrapeunblocker/rag-web-browser").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "what is retrieval augmented generation"
}' |
apify call scrapeunblocker/rag-web-browser --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapeunblocker/rag-web-browser"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/y0qvmW68hLdhgTGHt/builds/sidMTf9gAJnTSzBMt/openapi.json