RAG Web Browser - Google Search to Markdown for AI

RAG Web Browser searches Google and returns the top results' pages as clean Markdown, ready for an LLM, a RAG pipeline or an AI agent. Give it a search, a list of searches, or any URL, and get one row per page with its content, metadata and the Google result it came from. No login, no API key, no browser, from $0.50 per 1,000 pages, and searches are free.

# What is retrieval augmented generation (RAG)?

Published 21 October 2024

Updated 02 October 2026

By [Ivan Belcic](https://www.ibm.com/think/author/ivan-belcic) and

## What is retrieval augmented generation (RAG)?

Retrieval augmented generation, or RAG, is an architecture for optimizing the performance of an [artificial intelligence (AI)](https://www.ibm.com/think/topics/artificial-intelligence) model by connect...

That is the markdown of Google's result #2 for "what is retrieval augmented generation" (IBM), with tokenCount 3,987 and wordCount 2,309 beside it.

Measured on Apify: one search and its top 3 pages in 3.2 s of run time; 20 searches and 100 pages in 24 s.

What does RAG Web Browser do?

It gives AI agents and LLM apps the two things a web browsing tool needs: search and read. A search goes to Google (any country, any language, any time range) and the top organic results are fetched in parallel, cleaned down to their main content and converted to Markdown. A URL is read directly. One call returns everything, so an agent spends one tool call instead of a search plus N page fetches.

• Search or read: a Google search (with operators such as site:, -site:, "exact phrase", after:2026-01-01), a page URL, a bare domain like docs.apify.com, or a pasted Google search URL. One or thousands per run.
• Main content only: menus, sidebars, footers, cookie banners and comment threads are removed, so the tokens you pay your LLM for are the article, the docs or the post. Switch to "Whole page" when you want everything.
• Markdown, plain text, cleaned HTML and links: any combination, at the same price. Tables, code blocks with their language, nested lists and absolute links are kept.
• Made for LLM context windows: every page comes with tokenCount and wordCount, and an optional character cap cuts each page on a paragraph boundary.
• Always the number of pages you asked for: when a result cannot be read (a site that blocks automated visitors, a PDF, a dead link), the next Google result is read in its place.
• Only pay for pages read. Searches are free; failed pages, skipped sites and empty searches return an explanation row and are never charged.
• Built for agents and automation: callable as a tool through Apify's MCP server, the API, schedules, and Make, Zapier or n8n.

Why it is fast and cheap

There is no browser. Google's lightweight results page is read over plain HTTP, and each result page is fetched with a real browser's network fingerprint, then parsed and converted in the same process. That is why one search with its pages takes about 3 seconds of run time, why the Actor runs on 1 GB instead of 8 GB, and why a page costs a fraction of what browser-based tools charge. A site that does not answer within 4 seconds gets a second request from another IP address, and the first answer wins.

What data can you extract from the web?

Field	Example	What it is
markdown	# What is RAG?\n\nRetrieval augmented...	The page's content as Markdown (also text, html, links when chosen)
metadata.url	https://www.ibm.com/think/topics/retrieval-augmented-generation	The page's address after redirects
metadata.title	What is RAG (Retrieval Augmented Generation)? | IBM	The page's title
metadata.description	Retrieval augmented generation (RAG) is an architecture...	Meta description
metadata.publishedAt	2024-10-31T00:00:00.000Z	Publication date (meta tags or JSON-LD), ISO 8601
metadata.author, siteName, languageCode, canonicalUrl, imageUrl, openGraph	en, https://...	The rest of the page's metadata
searchResult.position	2	The page's rank on Google
searchResult.title, description, displayedUrl, date	What is RAG...? - IBM	The Google result itself, with its snippet
tokenCount	3987	Estimated LLM tokens of the Markdown
wordCount	2309	Words in the content
contentScope	main	main when narrowed to the main content, page for the whole page
contentTruncated	false	True when your character cap cut the content
crawl.httpStatusCode, loadTimeMs, contentType	200, 1288, text/html	How the page loaded
input	what is retrieval augmented generation	The search or URL exactly as you entered it

How to search Google and get pages as Markdown

1. Click Try for free and type a search or a URL into Search query or URL (more go under More queries or URLs, one per line).
2. Set Results per search (3-5 is typical for an AI agent) and pick the Output formats.
3. Click Start. One search with 3 pages takes a few seconds.
4. Open the Output tab, or download the results as JSON, CSV, Excel or HTML.

How much does it cost?

You pay per page read. Searches are free, and the price drops on higher Apify plans:

Apify plan	Per 1,000 pages	Per page
Free	$1.25	$0.00125
Starter	$1.00	$0.001
Scale	$0.85	$0.00085
Business	$0.70	$0.0007
Enterprise	$0.50-0.60	$0.0005-0.0006

• An AI agent's typical call, one search with 3 pages: $0.003 on Starter.
• 1,000 searches with 5 pages each (5,000 pages): $5.00 on Starter, $3.50 on Business.
• A run also costs $0.00005 to start. Searches, failed pages, skipped sites and searches with no results cost nothing.
• The Free plan's $5 monthly credit covers about 4,000 pages.

Input

{
    "query": "best open source vector database",
    "maxResults": 5,
    "outputFormats": ["markdown"],
    "countryCode": "",
    "timeRange": "y",
    "maxCharsPerPage": 8000
}

Field	Default	Notes
Search query or URL (query)	what is retrieval augmented generation	A Google search or a page URL; several lines are several queries
More queries or URLs (queries)	none	Any number, one per line; duplicates are skipped
Start URLs (startUrls)	none	Pages in the URL-list shape other Actors output, for chaining
Results per search (maxResults)	3	1-100; Google's further result pages are fetched as needed
Replace pages that fail (replaceFailedPages)	on	Reads the next result in place of one that cannot be read
Output formats (outputFormats)	Markdown	Markdown, plain text, HTML (cleaned) and links, any combination
Content to keep (htmlTransformer)	Main content only	readableText (main content) or none (whole page)
Max characters per page (maxCharsPerPage)	0 (no cap)	Cuts Markdown and text on a paragraph boundary
Remove HTML elements (removeElementsCssSelector)	nav, footer, script, ...	CSS selector of extra elements to drop
Search from country (countryCode)	United States	Google's gl: results as a person there sees them
Interface language (languageCode)	the country's	Google's hl
Results from (timeRange)	any time	Past hour, 24 hours, week, month or year
Max pages per run (maxItems)	1000	Your cost cap across all queries
Time per query (requestTimeoutSecs)	40 s	Pages still loading then become TIMEOUT rows, so an agent gets its answer in time
Retries per page (maxRequestRetries)	1	Each retry comes from a fresh IP address
Queries at a time (maxConcurrency)	5	Pages of one query are always fetched in parallel
Proxy for pages (proxyConfiguration)	off	Off: the Actor's own IP, with refused pages retried through Apify datacenter proxy automatically. On: every page through datacenter proxy or your own proxy URLs
Debug log (debugMode)	off	Logs every request with its route, status and time

The input names (query, maxResults, outputFormats, htmlTransformer, removeElementsCssSelector, requestTimeoutSecs, maxRequestRetries, debugMode) and the output fields (searchResult, metadata, markdown, text, html, links, crawl) follow the common RAG web browser format, so an existing integration works unchanged.

Output

Each page read is one row. The Overview view shows query, rank, URL, title, size and Markdown; Search results shows the Google results; Page metadata shows dates and authors; Errors lists what could not be read. Every export includes all fields.

{
    "recordType": "item",
    "input": "what is retrieval augmented generation",
    "searchResult": {
        "position": 2,
        "title": "What is RAG (Retrieval Augmented Generation)? - IBM",
        "description": "Retrieval augmented generation (RAG) is an architecture for optimizing the performance of an artificial intelligence (AI) model by ...",
        "url": "https://www.ibm.com/think/topics/retrieval-augmented-generation",
        "displayedUrl": "www.ibm.com › think › topics › retrieval-augmented-generation",
        "date": "2024-10-21T00:00:00.000Z",
        "resultType": "ORGANIC"
    },
    "metadata": {
        "url": "https://www.ibm.com/think/topics/retrieval-augmented-generation",
        "title": "What is RAG (Retrieval Augmented Generation)? | IBM",
        "description": "Retrieval augmented generation (RAG) is an architecture for optimizing the performance of an artificial intelligence (AI) model by connecting it with external knowledge bases.",
        "languageCode": "en",
        "canonicalUrl": "https://www.ibm.com/think/topics/retrieval-augmented-generation",
        "author": null,
        "publishedAt": "2024-10-31T00:00:00.000Z",
        "modifiedAt": "2026-10-02T00:00:00.000Z",
        "siteName": null,
        "imageUrl": "https://www.ibm.com/content/dam/worldwide-content/stock-assets/adb-stk/ul/g/ce/5b/adobestock_451765072.jpeg/_jcr_content/renditions/cq5dam.web.1280.1280.jpeg",
        "openGraph": [{ "property": "og:type", "content": "website" }]
    },
    "markdown": "# What is retrieval augmented generation (RAG)?\n\nPublished 21 October 2024\n\nUpdated 02 October 2026\n\nBy [Ivan Belcic](https://www.ibm.com/think/author/ivan-belcic) ...",
    "wordCount": 2309,
    "tokenCount": 3987,
    "contentTruncated": false,
    "contentScope": "main",
    "crawl": {
        "httpStatusCode": 200,
        "httpStatusMessage": "OK",
        "loadedAt": "2026-10-29T13:42:45.777Z",
        "requestStatus": "handled",
        "loadTimeMs": 1288,
        "contentType": "text/html"
    },
    "scrapedAt": "2026-10-29T13:42:44.186Z"
}

A page or search that cannot be read becomes a row with "recordType": "error", the page's url, its searchResult and an errorCode, instead of failing the run:

errorCode	Meaning
BLOCKED	The site refused automated visitors (HTTP 401/403/429 or a bot check) from every address tried, or is a site known to refuse them (then skipped without a request)
TIMEOUT	The site did not answer within the time per query
NOT_FOUND	The page is gone (HTTP 404/410)
UNSUPPORTED_CONTENT	Not a web page: a PDF, image, video, archive or office file
HTTP_ERROR / NETWORK / INVALID_URL	The site answered with an error, the connection failed, or the address is malformed
NO_RESULTS	Google returned no web results for the search
RATE_LIMITED	Google refused the search from every address this run tried (retried once at the end of the run)
PARSE_FAILED	The page loaded but could not be read (please report it)

Error rows are never charged.

Tips and limits

• Agents: keep context small. Set Max characters per page (8,000 characters is about 2,000 tokens) and 3-5 results per search. Use site: in the query to stay on documentation you trust.
• No JavaScript is run. Pages that build their text in the browser (some single-page apps) return little content. Most news, docs, blogs, wikis and product pages are served as HTML and read fully. Plain Markdown and text files (llms.txt, README files) are returned as they are.
• Some sites refuse automated visitors from cloud addresses: Reddit, Medium and Stack Overflow always do, so their results are skipped and replaced by the next result. Others (Cloudflare-protected sites) sometimes do; those pages are retried from another IP, then reported as BLOCKED and not charged.
• PDFs and office files are not read. They are listed as UNSUPPORTED_CONTENT and not charged.
• Web results only. Google's maps, news carousels and AI Overview are not part of the pages read; for full results-page data use the Google Search Results Scraper.
• Hit your spending limit? The run stops cleanly at your maximum cost per run and keeps everything it collected. Raise the limit in the run options to get more.

FAQ

Is it legal to scrape Google and websites?

This Actor reads only pages that are publicly available without logging in, the same pages a visitor's browser loads. Pages can still contain personal data, which laws such as the GDPR protect, and some sites' terms restrict automated access. Use the data for a legitimate purpose, respect copyright when you reuse content, and ask a lawyer if you are unsure. Google is a trademark of its owner; this Actor is not affiliated with or endorsed by Google.

Can AI agents use it?

Yes, that is what it is built for. Agents connected to Apify's MCP server can add it as a tool: one call searches, reads the pages and returns Markdown with token counts. The input takes a single query, so an agent needs no other parameters.

Can I use it through the API or with Make, Zapier and n8n?

Yes. Every run can be started through the Apify API (the run-sync endpoint returns the pages in the same call) and returns the dataset as JSON, CSV or Excel. Schedules, webhooks and integrations for Make, Zapier, n8n and Google Sheets are on the Integrations tab.

Can I switch from another RAG web browser?

Yes. The input fields query, maxResults, outputFormats, htmlTransformer, removeElementsCssSelector, requestTimeoutSecs and maxRequestRetries, and the output fields searchResult, metadata, markdown, text, html, links and crawl, use the common format. Additions are on top: batch queries, country, language and time range, token counts, the character cap and page replacement.

Why did I get fewer pages than I asked for?

Google had fewer results than you asked for, every remaining result failed (the error rows say why), the time per query ran out, or the run reached your maximum cost per run. The log says which, and nothing collected before that is lost.

Related scrapers

• Google Search Results Scraper: the full Google results page as data: organic results, places, top stories, knowledge panel and related searches, without reading the pages.
• Website Content Crawler for LLMs & RAG: crawl a whole website to Markdown, with boilerplate learned across its pages and ready-to-embed chunks.
• YouTube Transcript Scraper: video transcripts for the same LLM and RAG pipelines.

Support

A page that comes back empty or wrong, or a site that should work? Open an issue on the Issues tab with the input you used, and it will be looked at quickly.