# RAG Web Browser - Google Search to Markdown for AI (`eiv/rag-web-browser`) Actor

Web search and fetch for AI agents and RAG: searches Google and returns the top results as clean Markdown, or reads any URL. Main content only, token counts, metadata, any country and language, many queries per run. Plain HTTP, no browser: seconds per query.

- **URL**: https://apify.com/eiv/rag-web-browser.md
- **Developed by:** [Eimantas V](https://apify.com/eiv) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 page reads

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**RAG Web Browser** searches **Google** and returns the top results' pages as clean **Markdown**, ready for an LLM, a RAG pipeline or an AI agent. Give it a search, a list of searches, or any URL, and get one row per page with its content, metadata and the Google result it came from. No login, no API key, no browser, from **$0.50 per 1,000 pages**, and searches are free.

```
## What is retrieval augmented generation (RAG)?

Published 21 October 2024

Updated 02 October 2026

By [Ivan Belcic](https://www.ibm.com/think/author/ivan-belcic) and

### What is retrieval augmented generation (RAG)?

Retrieval augmented generation, or RAG, is an architecture for optimizing the performance of an [artificial intelligence (AI)](https://www.ibm.com/think/topics/artificial-intelligence) model by connect...
```

That is the `markdown` of Google's result #2 for "what is retrieval augmented generation" (IBM), with `tokenCount` 3,987 and `wordCount` 2,309 beside it.

**Measured on Apify: one search and its top 3 pages in 3.2 s of run time; 20 searches and 100 pages in 24 s.**

### What does RAG Web Browser do?

It gives AI agents and LLM apps the two things a web browsing tool needs: **search** and **read**. A search goes to Google (any country, any language, any time range) and the top organic results are fetched in parallel, cleaned down to their main content and converted to Markdown. A URL is read directly. One call returns everything, so an agent spends one tool call instead of a search plus N page fetches.

- **Search or read**: a Google search (with operators such as `site:`, `-site:`, `"exact phrase"`, `after:2026-01-01`), a page URL, a bare domain like `docs.apify.com`, or a pasted Google search URL. One or thousands per run.
- **Main content only**: menus, sidebars, footers, cookie banners and comment threads are removed, so the tokens you pay your LLM for are the article, the docs or the post. Switch to "Whole page" when you want everything.
- **Markdown, plain text, cleaned HTML and links**: any combination, at the same price. Tables, code blocks with their language, nested lists and absolute links are kept.
- **Made for LLM context windows**: every page comes with `tokenCount` and `wordCount`, and an optional character cap cuts each page on a paragraph boundary.
- **Always the number of pages you asked for**: when a result cannot be read (a site that blocks automated visitors, a PDF, a dead link), the next Google result is read in its place.
- **Only pay for pages read.** Searches are free; failed pages, skipped sites and empty searches return an explanation row and are never charged.
- **Built for agents and automation**: callable as a tool through [Apify's MCP server](https://mcp.apify.com), the API, schedules, and Make, Zapier or n8n.

### Why it is fast and cheap

There is no browser. Google's lightweight results page is read over plain HTTP, and each result page is fetched with a real browser's network fingerprint, then parsed and converted in the same process. That is why one search with its pages takes about 3 seconds of run time, why the Actor runs on 1 GB instead of 8 GB, and why a page costs a fraction of what browser-based tools charge. A site that does not answer within 4 seconds gets a second request from another IP address, and the first answer wins.

### What data can you extract from the web?

| Field | Example | What it is |
|---|---|---|
| `markdown` | `# What is RAG?\n\nRetrieval augmented...` | The page's content as Markdown (also `text`, `html`, `links` when chosen) |
| `metadata.url` | `https://www.ibm.com/think/topics/retrieval-augmented-generation` | The page's address after redirects |
| `metadata.title` | `What is RAG (Retrieval Augmented Generation)? \| IBM` | The page's title |
| `metadata.description` | `Retrieval augmented generation (RAG) is an architecture...` | Meta description |
| `metadata.publishedAt` | `2024-10-31T00:00:00.000Z` | Publication date (meta tags or JSON-LD), ISO 8601 |
| `metadata.author`, `siteName`, `languageCode`, `canonicalUrl`, `imageUrl`, `openGraph` | `en`, `https://...` | The rest of the page's metadata |
| `searchResult.position` | `2` | The page's rank on Google |
| `searchResult.title`, `description`, `displayedUrl`, `date` | `What is RAG...? - IBM` | The Google result itself, with its snippet |
| `tokenCount` | `3987` | Estimated LLM tokens of the Markdown |
| `wordCount` | `2309` | Words in the content |
| `contentScope` | `main` | `main` when narrowed to the main content, `page` for the whole page |
| `contentTruncated` | `false` | True when your character cap cut the content |
| `crawl.httpStatusCode`, `loadTimeMs`, `contentType` | `200`, `1288`, `text/html` | How the page loaded |
| `input` | `what is retrieval augmented generation` | The search or URL exactly as you entered it |

### How to search Google and get pages as Markdown

1. Click **Try for free** and type a search or a URL into **Search query or URL** (more go under **More queries or URLs**, one per line).
2. Set **Results per search** (3-5 is typical for an AI agent) and pick the **Output formats**.
3. Click **Start**. One search with 3 pages takes a few seconds.
4. Open the **Output** tab, or download the results as JSON, CSV, Excel or HTML.

### How much does it cost?

You pay per page read. Searches are free, and the price drops on higher Apify plans:

| Apify plan | Per 1,000 pages | Per page |
|---|---|---|
| Free | $1.25 | $0.00125 |
| Starter | $1.00 | $0.001 |
| Scale | $0.85 | $0.00085 |
| Business | $0.70 | $0.0007 |
| Enterprise | $0.50-0.60 | $0.0005-0.0006 |

- An AI agent's typical call, one search with 3 pages: **$0.003** on Starter.
- 1,000 searches with 5 pages each (5,000 pages): **$5.00** on Starter, **$3.50** on Business.
- A run also costs **$0.00005** to start. Searches, failed pages, skipped sites and searches with no results cost nothing.
- The Free plan's $5 monthly credit covers about **4,000 pages**.

### Input

```json
{
    "query": "best open source vector database",
    "maxResults": 5,
    "outputFormats": ["markdown"],
    "countryCode": "",
    "timeRange": "y",
    "maxCharsPerPage": 8000
}
```

| Field | Default | Notes |
|---|---|---|
| Search query or URL (`query`) | `what is retrieval augmented generation` | A Google search or a page URL; several lines are several queries |
| More queries or URLs (`queries`) | none | Any number, one per line; duplicates are skipped |
| Start URLs (`startUrls`) | none | Pages in the URL-list shape other Actors output, for chaining |
| Results per search (`maxResults`) | 3 | 1-100; Google's further result pages are fetched as needed |
| Replace pages that fail (`replaceFailedPages`) | on | Reads the next result in place of one that cannot be read |
| Output formats (`outputFormats`) | Markdown | Markdown, plain text, HTML (cleaned) and links, any combination |
| Content to keep (`htmlTransformer`) | Main content only | `readableText` (main content) or `none` (whole page) |
| Max characters per page (`maxCharsPerPage`) | 0 (no cap) | Cuts Markdown and text on a paragraph boundary |
| Remove HTML elements (`removeElementsCssSelector`) | nav, footer, script, ... | CSS selector of extra elements to drop |
| Search from country (`countryCode`) | United States | Google's `gl`: results as a person there sees them |
| Interface language (`languageCode`) | the country's | Google's `hl` |
| Results from (`timeRange`) | any time | Past hour, 24 hours, week, month or year |
| Max pages per run (`maxItems`) | 1000 | Your cost cap across all queries |
| Time per query (`requestTimeoutSecs`) | 40 s | Pages still loading then become `TIMEOUT` rows, so an agent gets its answer in time |
| Retries per page (`maxRequestRetries`) | 1 | Each retry comes from a fresh IP address |
| Queries at a time (`maxConcurrency`) | 5 | Pages of one query are always fetched in parallel |
| Proxy for pages (`proxyConfiguration`) | off | Off: the Actor's own IP, with refused pages retried through Apify datacenter proxy automatically. On: every page through datacenter proxy or your own proxy URLs |
| Debug log (`debugMode`) | off | Logs every request with its route, status and time |

The input names (`query`, `maxResults`, `outputFormats`, `htmlTransformer`, `removeElementsCssSelector`, `requestTimeoutSecs`, `maxRequestRetries`, `debugMode`) and the output fields (`searchResult`, `metadata`, `markdown`, `text`, `html`, `links`, `crawl`) follow the common RAG web browser format, so an existing integration works unchanged.

### Output

Each page read is one row. The **Overview** view shows query, rank, URL, title, size and Markdown; **Search results** shows the Google results; **Page metadata** shows dates and authors; **Errors** lists what could not be read. Every export includes all fields.

```json
{
    "recordType": "item",
    "input": "what is retrieval augmented generation",
    "searchResult": {
        "position": 2,
        "title": "What is RAG (Retrieval Augmented Generation)? - IBM",
        "description": "Retrieval augmented generation (RAG) is an architecture for optimizing the performance of an artificial intelligence (AI) model by ...",
        "url": "https://www.ibm.com/think/topics/retrieval-augmented-generation",
        "displayedUrl": "www.ibm.com › think › topics › retrieval-augmented-generation",
        "date": "2024-10-21T00:00:00.000Z",
        "resultType": "ORGANIC"
    },
    "metadata": {
        "url": "https://www.ibm.com/think/topics/retrieval-augmented-generation",
        "title": "What is RAG (Retrieval Augmented Generation)? | IBM",
        "description": "Retrieval augmented generation (RAG) is an architecture for optimizing the performance of an artificial intelligence (AI) model by connecting it with external knowledge bases.",
        "languageCode": "en",
        "canonicalUrl": "https://www.ibm.com/think/topics/retrieval-augmented-generation",
        "author": null,
        "publishedAt": "2024-10-31T00:00:00.000Z",
        "modifiedAt": "2026-10-02T00:00:00.000Z",
        "siteName": null,
        "imageUrl": "https://www.ibm.com/content/dam/worldwide-content/stock-assets/adb-stk/ul/g/ce/5b/adobestock_451765072.jpeg/_jcr_content/renditions/cq5dam.web.1280.1280.jpeg",
        "openGraph": [{ "property": "og:type", "content": "website" }]
    },
    "markdown": "# What is retrieval augmented generation (RAG)?\n\nPublished 21 October 2024\n\nUpdated 02 October 2026\n\nBy [Ivan Belcic](https://www.ibm.com/think/author/ivan-belcic) ...",
    "wordCount": 2309,
    "tokenCount": 3987,
    "contentTruncated": false,
    "contentScope": "main",
    "crawl": {
        "httpStatusCode": 200,
        "httpStatusMessage": "OK",
        "loadedAt": "2026-10-29T13:42:45.777Z",
        "requestStatus": "handled",
        "loadTimeMs": 1288,
        "contentType": "text/html"
    },
    "scrapedAt": "2026-10-29T13:42:44.186Z"
}
```

A page or search that cannot be read becomes a row with `"recordType": "error"`, the page's `url`, its `searchResult` and an `errorCode`, instead of failing the run:

| `errorCode` | Meaning |
|---|---|
| `BLOCKED` | The site refused automated visitors (HTTP 401/403/429 or a bot check) from every address tried, or is a site known to refuse them (then skipped without a request) |
| `TIMEOUT` | The site did not answer within the time per query |
| `NOT_FOUND` | The page is gone (HTTP 404/410) |
| `UNSUPPORTED_CONTENT` | Not a web page: a PDF, image, video, archive or office file |
| `HTTP_ERROR` / `NETWORK` / `INVALID_URL` | The site answered with an error, the connection failed, or the address is malformed |
| `NO_RESULTS` | Google returned no web results for the search |
| `RATE_LIMITED` | Google refused the search from every address this run tried (retried once at the end of the run) |
| `PARSE_FAILED` | The page loaded but could not be read (please report it) |

Error rows are never charged.

### Tips and limits

- **Agents: keep context small.** Set **Max characters per page** (8,000 characters is about 2,000 tokens) and 3-5 results per search. Use `site:` in the query to stay on documentation you trust.
- **No JavaScript is run.** Pages that build their text in the browser (some single-page apps) return little content. Most news, docs, blogs, wikis and product pages are served as HTML and read fully. Plain Markdown and text files (`llms.txt`, README files) are returned as they are.
- **Some sites refuse automated visitors** from cloud addresses: Reddit, Medium and Stack Overflow always do, so their results are skipped and replaced by the next result. Others (Cloudflare-protected sites) sometimes do; those pages are retried from another IP, then reported as `BLOCKED` and not charged.
- **PDFs and office files are not read.** They are listed as `UNSUPPORTED_CONTENT` and not charged.
- **Web results only.** Google's maps, news carousels and AI Overview are not part of the pages read; for full results-page data use the [Google Search Results Scraper](https://apify.com/eiv/google-search-scraper).
- **Hit your spending limit?** The run stops cleanly at your maximum cost per run and keeps everything it collected. Raise the limit in the run options to get more.

### FAQ

#### Is it legal to scrape Google and websites?

This Actor reads only pages that are publicly available without logging in, the same pages a visitor's browser loads. Pages can still contain personal data, which laws such as the GDPR protect, and some sites' terms restrict automated access. Use the data for a legitimate purpose, respect copyright when you reuse content, and ask a lawyer if you are unsure. Google is a trademark of its owner; this Actor is not affiliated with or endorsed by Google.

#### Can AI agents use it?

Yes, that is what it is built for. Agents connected to [Apify's MCP server](https://mcp.apify.com) can add it as a tool: one call searches, reads the pages and returns Markdown with token counts. The input takes a single `query`, so an agent needs no other parameters.

#### Can I use it through the API or with Make, Zapier and n8n?

Yes. Every run can be started through the [Apify API](https://docs.apify.com/api/v2) (the run-sync endpoint returns the pages in the same call) and returns the dataset as JSON, CSV or Excel. Schedules, webhooks and integrations for Make, Zapier, n8n and Google Sheets are on the **Integrations** tab.

#### Can I switch from another RAG web browser?

Yes. The input fields `query`, `maxResults`, `outputFormats`, `htmlTransformer`, `removeElementsCssSelector`, `requestTimeoutSecs` and `maxRequestRetries`, and the output fields `searchResult`, `metadata`, `markdown`, `text`, `html`, `links` and `crawl`, use the common format. Additions are on top: batch queries, country, language and time range, token counts, the character cap and page replacement.

#### Why did I get fewer pages than I asked for?

Google had fewer results than you asked for, every remaining result failed (the error rows say why), the time per query ran out, or the run reached your maximum cost per run. The log says which, and nothing collected before that is lost.

### Related scrapers

- [Google Search Results Scraper](https://apify.com/eiv/google-search-scraper): the full Google results page as data: organic results, places, top stories, knowledge panel and related searches, without reading the pages.
- [Website Content Crawler for LLMs & RAG](https://apify.com/eiv/llm-content-crawler): crawl a whole website to Markdown, with boilerplate learned across its pages and ready-to-embed chunks.
- [YouTube Transcript Scraper](https://apify.com/eiv/youtube-transcript-probe): video transcripts for the same LLM and RAG pipelines.

### Support

A page that comes back empty or wrong, or a site that should work? Open an issue on the **Issues** tab with the input you used, and it will be looked at quickly.

# Actor input Schema

## `query` (type: `string`):

A Google search, e.g. "best open-source vector databases" (operators work: site:, -site:, "exact phrase", after:2026-01-01), or the URL of one page to read, e.g. https://docs.apify.com. Several lines are read as several queries.

## `queries` (type: `array`):

Any number of further searches or URLs, one per line, read in the same run. Duplicates are skipped.

## `startUrls` (type: `array`):

Pages to read, in the URL-list shape other Actors output, so a previous Actor's results can be fed straight in. Read like URLs in the fields above.

## `maxResults` (type: `integer`):

How many of Google's top organic results to read for each search. Every page read is one charged result. 3-5 is typical for an AI agent; up to 100 (Google's further result pages are fetched as needed). Ignored for URLs.

## `replaceFailedPages` (type: `boolean`):

When a result cannot be read (the site blocks automated visitors, a PDF, a dead link), read the next Google result in its place, so a search returns as many pages as asked for. The failed result is still listed, uncharged.

## `outputFormats` (type: `array`):

What to return for each page: Markdown (best for LLMs), plain text, the cleaned HTML of the content, and the page's links. Any combination; the price per page is the same.

## `htmlTransformer` (type: `string`):

Main content (default): the article, docs or post body, without menus, sidebars, footers, cookie banners and comment threads; a page whose text is spread out keeps everything. Whole page: everything except the elements removed below.

## `maxCharsPerPage` (type: `integer`):

Cut each page's Markdown and text at this many characters, on a paragraph boundary, to keep an LLM's context small (8,000 characters is about 2,000 tokens). Rows say contentTruncated. 0 = no cap.

## `removeElementsCssSelector` (type: `string`):

CSS selector of elements to remove before the content is read, e.g. ".ad, .newsletter". Empty removes nothing extra. Scripts, styles and form controls are always removed.

## `countryCode` (type: `string`):

The country Google searches from (gl): results as a person there sees them. Default: United States.

## `languageCode` (type: `string`):

Google's interface language (hl), which also steers result language. Default: the country's own language.

## `timeRange` (type: `string`):

Only results Google dates within this period, for fresh news and recent posts.

## `maxItems` (type: `integer`):

Stop after this many pages read in total, across all queries. You pay per page read, so this is also your cost cap for the run.

## `requestTimeoutSecs` (type: `integer`):

Time budget for one query: the Google search and all its pages. Pages still loading when it runs out are returned as TIMEOUT rows (not charged), so an agent gets its answer in time. Searches the run retries at the end get a new budget.

## `maxRequestRetries` (type: `integer`):

Retries for a page that is refused (403, 429, a bot check), drops the connection or does not answer within 12 s; each retry comes from a fresh IP, within the time per query. 404s are not retried.

## `maxConcurrency` (type: `integer`):

How many queries run in parallel. The pages of each query are always fetched in parallel.

## `proxyConfiguration` (type: `object`):

Off by default: pages load from the Actor's own IP, and a refused page is retried through Apify datacenter proxy automatically. Turn it on to send every page through datacenter proxy or your own proxy URLs. Residential is not used for pages.

## `debugMode` (type: `boolean`):

Log every search and page request (route, HTTP status, time, error) in the run log, for troubleshooting a site that will not load.

## Actor input object example

```json
{
  "query": "what is retrieval augmented generation",
  "maxResults": 3,
  "replaceFailedPages": true,
  "outputFormats": [
    "markdown"
  ],
  "htmlTransformer": "readableText",
  "maxCharsPerPage": 0,
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^='data:'],\n[role=\"alert\"],\n[role=\"banner\"],\n[role=\"dialog\"],\n[role=\"alertdialog\"],\n[role=\"region\"][aria-label*=\"skip\" i],\n[aria-modal=\"true\"]",
  "countryCode": "",
  "languageCode": "",
  "timeRange": "",
  "maxItems": 1000,
  "requestTimeoutSecs": 40,
  "maxRequestRetries": 1,
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "debugMode": false
}
```

# Actor output Schema

## `results` (type: `string`):

One row per page read (recordType "item"), plus one uncharged row per page or search that could not be read (recordType "error").

## `searchResults` (type: `string`):

The Google results behind the pages: rank, title, URL, snippet and date.

## `summary` (type: `string`):

Totals for the run: queries, pages read, failures, requests and why the run stopped.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "what is retrieval augmented generation"
};

// Run the Actor and wait for it to finish
const run = await client.actor("eiv/rag-web-browser").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "query": "what is retrieval augmented generation" }

# Run the Actor and wait for it to finish
run = client.actor("eiv/rag-web-browser").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "what is retrieval augmented generation"
}' |
apify call eiv/rag-web-browser --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,eiv/rag-web-browser"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/LP1bgp1JLt5UKnxTW/builds/pEwAwz7ZdjohgSFVT/openapi.json