# RAG Web Browser: Web Search to Markdown for LLMs & AI Agents (`pikoulas/rag-web-browser`) Actor

Search the web and get the top results' pages as clean Markdown for LLMs, RAG and AI agents, or fetch one URL: title, URL, snippet, metadata and main content. Standby mode answers agents' and MCP calls over HTTP in seconds. Plain HTTP, no browser, $2.50 per 1,000 searches of 3 pages.

- **URL**: https://apify.com/pikoulas/rag-web-browser.md
- **Developed by:** [Spyridon Pikoulas](https://apify.com/pikoulas) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 searches

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RAG Web Browser

Search the web and get the top results' pages as clean Markdown, ready for LLMs, RAG pipelines,
chatbots and AI agents, or give a URL and get that page:

- **Search results:** each result's title, URL, snippet and rank, from DuckDuckGo, with Brave
  Search for more results or when DuckDuckGo refuses. Search operators work: `"exact phrase"`,
  `site:`, `-site:`.
- **The page's own content, not its chrome:** menus, headers, footers, sidebars, cookie banners
  and share buttons are left out, and headings, lists, tables, code blocks and links are kept, as
  Markdown, plain text or cleaned HTML.
- **Metadata:** the page's title, description, language, canonical URL, author and published
  date.
- **Standby mode:** an HTTP endpoint that answers `GET /search?query=...` with the results as
  JSON, for AI agents, custom GPTs and tools that call the web in the middle of a conversation.
- **Inputs like Apify's RAG Web Browser:** `query`, `maxResults`, `outputFormats`,
  `requestTimeoutSecs`, `removeElementsCssSelector` and the same result shape (`searchResult`,
  `metadata`, `markdown`), so an input or a parser made for it works here.

It is fast: plain HTTP with a real browser's TLS fingerprint and no headless browser. A search
with 3 pages took 1–4 seconds, 10 pages 4 seconds, 20 pages 5 seconds. It costs **$2.50 per 1,000
searches** of 3 pages.

### What people use it for

- **Web search for AI agents:** a tool an agent calls through MCP, LangChain, LlamaIndex or an
  OpenAI function to look things up and read the pages, not only the snippets.
- **RAG with fresh content:** the current pages about a topic, as Markdown for a prompt or a
  vector database.
- **Grounding and fact-checking:** the sources behind an answer, with their URLs and dates.
- **Research and content briefs:** what the top pages for a query say, side by side.
- **SEO:** which pages rank for a query, their titles, descriptions and word counts.

### How to search the web and get the pages as Markdown

1. Click **Try for free**; Apify's free plan covers a first run.
2. Enter your **Search terms or URL** and how many results you want.
3. Click **Start**, then download the results as JSON, CSV, Excel or HTML, or read them through
   the API.

An input with more options:

```json
{
  "query": "function calling site:openai.com",
  "maxResults": 5,
  "outputFormats": ["markdown", "text"],
  "includeImages": false,
  "removeElementsCssSelector": ".newsletter, #comments",
  "requestTimeoutSecs": 30,
  "maxRequestRetries": 1
}
```

A query that is a URL (`https://docs.python.org/3/library/asyncio-task.html`, or
`example.com/pricing`) is fetched as it is, without a search.

### Output

One item per result, in rank order (Markdown shortened here):

```json
{
  "query": "retrieval augmented generation",
  "searchResult": {
    "title": "Retrieval-augmented generation - Wikipedia",
    "description": "Retrieval-augmented generation (RAG) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources. [1][2] With RAG, LLMs first refer to a specified set of documents, then respond to user queries.",
    "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "rank": 1,
    "engine": "duckduckgo"
  },
  "crawl": {
    "httpStatusCode": 200,
    "loadedUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "loadedAt": "2026-09-29T14:50:06Z",
    "requestStatus": "handled",
    "loadMs": 267
  },
  "metadata": {
    "title": "Retrieval-augmented generation - Wikipedia",
    "description": null,
    "languageCode": "en",
    "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "canonicalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "author": "Contributors to Wikimedia projects",
    "publishedAt": "2023-11-05T13:19:20Z"
  },
  "markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique that enables [large language models](https://en.wikipedia.org/wiki/Large_language_model) (LLMs) to retrieve and incorporate new information from external data sources.[1][2] With RAG, LLMs first refer to a specified set of documents, then respond to user queries. ...",
  "wordCount": 2406,
  "needsJavaScript": false,
  "warning": null
}
```

`text` and `html` come instead of or besides `markdown` when asked for. A result from Brave
Search can carry the page's `date`. A page that can't be read keeps its `searchResult` and comes
with an `error` ("Couldn't load the page: HTTP 404.", "Couldn't load the page: the site's bot
check refused it.", "Not a web page (application/pdf): nothing to extract."), and isn't charged.

### Use it as an API

One call runs it and returns the results as JSON. Apify's Python and JavaScript clients,
LangChain and LlamaIndex loaders, and its Make, Zapier and n8n integrations work the same way.

```bash
curl -X POST "https://api.apify.com/v2/acts/pikoulas~rag-web-browser/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "retrieval augmented generation", "maxResults": 3}'
```

In Standby mode it answers straight away, without starting a run for each call; the input's
fields are query parameters (`outputFormats=markdown,text`):

```bash
curl "https://pikoulas--rag-web-browser.apify.actor/search?query=retrieval%20augmented%20generation&maxResults=3" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN"
```

AI agents can call it as a tool through Apify's MCP server: `https://mcp.apify.com?tools=pikoulas/rag-web-browser`.

### How much does it cost to search the web with it?

You pay **$0.001 per search** and **$0.0005 per page**: a search with 3 pages is $0.0025, which
is $2.50 per 1,000. A URL given as the query is only the page, $0.0005. Pages that fail, and pages
with next to no text (apps that need JavaScript), aren't charged, and neither is a search with no
results. The same prices apply in Standby mode. Runs that go through Apify's residential proxy also
pay its traffic, $0.0012 per 100 KB; a result page is 50–150 KB on the wire (about 70 KB on
average) and a search 7–60 KB. There are no monthly fees.

### Good to know

- **Search engines:** DuckDuckGo's first page gives up to about 10 results; Brave Search adds
  more, 20 a page. Google isn't used: it answers plain HTTP only with a JavaScript page. So the
  results are close to, but not the same as, Google's.
- **No JavaScript.** It reads the HTML a site sends, which for most sites (news, blogs, docs,
  Wikipedia, shops, forums) is the whole content. Apps that build their page in the browser come
  with `needsJavaScript: true`, next to no text and a warning, and aren't charged.
- **Bot checks:** sites behind Cloudflare's or DataDome's challenge, and Reddit's threads, refuse
  plain HTTP clients: those results come with an error and aren't charged. A proxy doesn't get
  past a challenge.
- **Timeout:** `requestTimeoutSecs` is the whole request's, the search included; the pages not
  loaded by then come with an error and the others are returned.
- **Files:** PDFs, images and other files aren't read; plain-text and Markdown files are given as
  they are.
- **Standby:** a Standby run serves requests until a minute passes without one; the request after
  that waits a few seconds for a new one to start. `proxyConfiguration` can't be set per request.
- Input fields of Apify's RAG Web Browser that don't apply here (`scrapingTool`,
  `htmlTransformer`, `serpProxyGroup` and the like) are ignored.

### FAQ

#### Is it legal to scrape search results and web pages?

It fetches only public pages, without logging in. Scraping public data is generally legal, but
personal data is protected by laws such as the GDPR and the CCPA, and a site's terms may limit
how you use its content. If you're unsure, ask a lawyer; Apify's
[Is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/) is a good start.

#### Something missing or not working?

Open an issue on the **Issues** tab with the input you used, or ask there for a field you need.

# Actor input Schema

## `query` (type: `string`):

What to search the web for, or the URL of one page to fetch. Search operators work: "exact phrase", site:openai.com, -site:reddit.com. A URL (https://... or example.com/page) is fetched as it is, without a search.

## `maxResults` (type: `integer`):

How many of the top organic results to fetch, in rank order. Ignored when the query is a URL.

## `outputFormats` (type: `array`):

Each page's main content as Markdown (headings, lists, tables, code blocks and links kept), plain text, and the cleaned HTML of that content.

## `includeImages` (type: `boolean`):

Keep the content's images in the Markdown, as ![alt](url). Off saves tokens.

## `removeElementsCssSelector` (type: `string`):

Elements to leave out of the content besides the page's menus, header, footer, sidebars and banners, which are left out anyway, e.g. .newsletter, #comments, figure.

## `requestTimeoutSecs` (type: `integer`):

The time the whole request has, the search included. A page not loaded by then comes with an error, and the others are returned.

## `maxRequestRetries` (type: `integer`):

Times a page is asked again after a network error, a server error (5xx) or a 429.

## `proxyConfiguration` (type: `object`):

Not needed for most sites. The result pages are fetched through this proxy; datacenter IPs cost nothing extra, residential ones are charged per 100 KB fetched. A search engine refusing the run's IP is asked again through Apify's residential proxy by itself.

## Actor input object example

```json
{
  "query": "retrieval augmented generation",
  "maxResults": 3,
  "outputFormats": [
    "markdown"
  ],
  "includeImages": false,
  "requestTimeoutSecs": 40,
  "maxRequestRetries": 1,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `content` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "retrieval augmented generation"
};

// Run the Actor and wait for it to finish
const run = await client.actor("pikoulas/rag-web-browser").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "query": "retrieval augmented generation" }

# Run the Actor and wait for it to finish
run = client.actor("pikoulas/rag-web-browser").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "retrieval augmented generation"
}' |
apify call pikoulas/rag-web-browser --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pikoulas/rag-web-browser"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/eI2EfPnsN48XDMAh3/builds/3UM2B1nBXHtvKxx6j/openapi.json