RAG Web Browser

What it does

Search the web and return up to 1–10 readable results as clean Markdown for LLMs, AI agents and RAG pipelines, or read a list of URLs. Search result pages are rendered in a real Chrome (DuckDuckGo first, then Bing) and parsed into ranked results; each candidate page is then read with the TidyTools Smart Fetch backend. Searching is free: you pay only for pages returned as Markdown.

Search engines' terms of use generally do not allow automated querying; reading their result pages is a business risk the operator has accepted. Verification pages (CAPTCHAs, "unusual traffic" checks) are never solved or clicked: that attempt is marked blocked, free of charge, and the next network or engine is tried.

How to use

1. Type search words in Search query or URL (or paste page links in URLs to read).
2. Set Maximum pages (1–10) and start the run.
3. Read the pages from the Dataset (markdown column) or fetch them through the Apify API / MCP in your RAG pipeline.

Input

{"query":"best open source vector database","maxResults":3}

{"urls":["https://docs.python.org/3/library/asyncio.html","realpython.com/async-io-python/"]}

Field	Default	Meaning
query	–	Search words, or an http(s) URL to read one page.
maxResults	3	Return up to N readable pages (1–10). Failed pages are followed by the next ranked candidate until N succeed or candidates run out.
urls	–	Read these pages instead of searching (up to 100; pasted text with one URL per line or comma-separated also works; bare domains get https://). query is ignored. Invalid lines become free rows with errorType: "invalid_input"; duplicates are read once.
url	–	Read this one page directly (same as one line in urls).
engine	auto	auto tries DuckDuckGo → Bing; duckduckgo, bing or google uses only that engine. Google answers automated searches with its "unusual traffic" verification page almost every time, so it is not part of auto.
searchEgress	auto	Network for search result pages: auto (data-center network first, residential network if the engine blocks it), warp or resi. The residential network is used on paid Apify plans only, within a per-search allowance tied to the pages you can be charged for (at most half of maxResults × the page price); once that is used up the search continues on the data-center network or the next engine. On the free plan every option uses the data-center network, so searches are more often blocked there. Never the operator's home connection.

Only public HTTP(S) URLs are read. Ads, tracking redirects and the engines' own links are dropped; DuckDuckGo (/l/?uddg=), Bing (/ck/a) and Google (/url?q=) redirect wrappers are decoded to the real URL, tracking parameters (utm_*, gclid, fbclid, msclkid) are removed, and duplicates (same host without www., path and query) are dropped.

Output

One Dataset row per page:

{"url":"https://realpython.com/async-io-python/","title":"Python's asyncio: A Hands-On Walkthrough – Real Python","rank":1,"engine":"duckduckgo",
 "searchResult":{"rank":1,"title":"Python's asyncio: A Hands-On Walkthrough – Real Python","snippet":"Python's asyncio library enables you to write concurrent code …","engine":"duckduckgo"},
 "markdown":"# Python's asyncio: A Hands-On Walkthrough …","fetchedAt":"2026-10-22T05:52:10.000Z","success":true,"charged":null,"chargedEvents":null}

Field	Meaning
url	The page that was read.
title	The page's own title (the search result title when the page has none).
rank	Position in the search results (1 = first), null for URLs you gave.
engine	duckduckgo, bing or google; null for URLs you gave.
searchResult	rank, title, snippet and engine as the search result page showed them; null for URLs you gave.
markdown	The page's main content as Markdown (navigation, footers and boilerplate removed). null when the page could not be read.
fetchedAt	When the page was read (ISO 8601).
input	The original line, for rows from urls.
success	true when the page returned usable Markdown.
charged, chargedEvents	false and [] on failed rows. Successful rows are saved before billing, so these are null (pending); SUMMARY.chargedPages has the result.
errorType, error	On failed rows only, e.g. blocked, robots_disallowed, empty_page, invalid_input, search_blocked.

If every engine is blocked, one free row with errorType: "search_blocked" is written.

For a search, the Dataset also records free failed page rows. They do not count toward maxResults; the Actor tries the next ranked candidate. Successful rows retain their original search rank, so ranks can have gaps. SUMMARY.pages counts successful pages, SUMMARY.failed counts failed attempts, and SUMMARY.status is SUCCESS when the requested number of readable pages is reached. URL lists retain their original behavior.

SUMMARY (key-value store) has status (SUCCESS, PARTIAL, NO_RESULTS, LIMIT_REACHED), pages, failed, chargedPages (each saved page URL with its actual charged events), costPlan, notProcessed when the max charge per run stopped the run, invalidInputs/duplicates for URL lists and, for searches, search: engine, egress, costTier, results (number of ranked candidates), resiBytes and attempts (one entry per result page tried: engine, egress, status, blockType, ms, bytes, resiBytes, results). Block types: captcha_unusual_traffic, captcha_interactive, consent_wall, js_challenge_not_passed, http_<status>, no_results, no_results_parsed, irrelevant_results, too_few_results, empty_page, limiter_unavailable.

Control your cost

No start fee. Searching is free. Each usable page returned as Markdown is one page-http event. Verification pages, blocked, empty and failed pages, blocked searches and invalid inputs are free. At the start of a run the log and status show the worst case (number of pages × page price). When your max charge per run is reached, the remaining pages are not read and are listed in SUMMARY.notProcessed.

Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/rag-web-browser) to Claude, Cursor or any MCP client, then ask e.g. "Search for python asyncio tutorials and summarise the top 3 pages." Minimal input: {"query":"python asyncio tutorial"}. Failed items are not charged and carry an errorType.

FAQ

Am I charged for blocked searches or failed pages? No. Only pages returned as usable Markdown are charged (page-http); verification pages, blocked searches, empty and failed pages are free.

Does it solve CAPTCHAs? No. A verification page marks that attempt as blocked and the next network or engine is tried.

Can I read my own list of pages instead of searching? Yes: put them in urls (up to 100) or url; query is then ignored.

Limitations

• DuckDuckGo searches over the shared data-center network reserve a cross-run slot at about one query per 7 seconds. When a slot cannot be reserved, auto mode tries the next available network or Bing; no verification is clicked or solved. Search remains free. On 2026-10-23, 20 consecutive live DuckDuckGo/WARP searches through this limiter returned 20/20 usable result pages. The 2026-10-22 baseline without pacing was 7/20; with manual spacing it was 20/20. See research/rag-serp-measure-2026-10-22.md.
• Pages are read over plain HTTP from Cloudflare with robots rules respected; sites that refuse that route (for example HTTP 403) return a free failed row.
• Up to 10 successful search pages per query and 100 URLs per run; search candidates are tried in ranked order, while URL lists are read 4 at a time.