RAG Web Browser (Anti-Bot) - Search to Markdown for LLMs

Give your LLM, AI agent or RAG pipeline live access to the web: send a search query and get the top results already read and converted to clean Markdown, or send a URL and get that page. Every page is loaded through an anti-bot unblocker, so results behind Cloudflare, DataDome, PerimeterX (HUMAN) or Akamai come back as real content instead of a challenge screen, which is where ordinary web-browsing tools return nothing.

The input and output use the same field names as the most popular RAG web browser (query, maxResults, outputFormats, requestTimeoutSecs; searchResult, metadata, markdown), so existing integrations switch over without code changes.

Features

• Search + read in one call - the top N results of a web search, each as clean Markdown with its title, description and URL.
• Or read one URL - put a URL in query and get that page's content.
• Gets through anti-bot walls - Cloudflare, DataDome, PerimeterX, Akamai.
• Complete answers - a result whose page is dead, blocked or empty is replaced by the next search result, so you still get maxResults pages.
• Fast - search answers in a few seconds and all pages are read in parallel; a typical 3-result run finishes in well under a minute.
• No Google needed - searches DuckDuckGo, with Bing and Brave Search started automatically when it is slow or fails; search operators like site: and -site: work.
• LLM-friendly Markdown - main content only, headings, lists, tables, absolute links and fenced code; menus, footers, cookie banners and decorative images removed.
• Pay only for pages delivered - skipped results are listed in the ERRORS record and never billed.

Use cases

• Web search tool for AI agents (OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, n8n, Make, Zapier).
• Retrieval-augmented generation - ground answers in current web pages.
• Research and fact-checking - the content behind a search, not just snippets.
• Reading pages other tools are blocked from - review sites, marketplaces, job boards, news behind bot protection.

Input

Field	Type	Default	Description
query	string	web scraping api	Search terms (operators like site: work), or one http(s):// URL to read.
maxResults	integer	3	Search results to read and return (1-20).
outputFormats	array	["markdown"]	Any of markdown, text, html (cleaned content HTML).
requestTimeoutSecs	integer	45	Timeout for one page; a slower page is replaced by the next result.
searchEngine	string	auto	auto (DuckDuckGo, with Bing and Brave when it is slow), duckduckgo, bing or brave.
country	string	US	Country to search from (ISO-2), which sets the results market.
htmlTransformer	string	readableText	readableText = main content only, none = whole page.
removeElementsCssSelector	string	nav, footer, scripts, dialogs ...	Elements removed before extraction.
removeCookieWarnings	boolean	true	Remove cookie consent banners.

Example input

{
  "query": "best espresso machine 2026 review",
  "maxResults": 5,
  "outputFormats": ["markdown"]
}

Read a single Cloudflare-protected page:

{
  "query": "https://www.glassdoor.com/blog/"
}

Output

One dataset item per page, in search-rank order:

{
  "query": "web scraping api",
  "searchResult": {
    "title": "Web Scraping API for Data Extraction & Web Crawling | Floppydata",
    "description": "Powerful Web Scraping API to extract data from any website...",
    "url": "https://floppydata.com/web-scraping/",
    "position": 1
  },
  "metadata": {
    "title": "Web Scraping API for Data Extraction & Web Crawling | Floppydata",
    "description": "Powerful Web Scraping API to extract data from any website...",
    "languageCode": "en",
    "author": null,
    "url": "https://floppydata.com/web-scraping/",
    "canonicalUrl": "https://floppydata.com/web-scraping/"
  },
  "crawl": {
    "httpStatusCode": 200,
    "loadedAt": "2026-10-12T17:40:12.565916+00:00",
    "requestStatus": "handled"
  },
  "markdown": "# Web Scraping API\n\nNo more scraping blocks, CAPTCHAs, or failed requests...\n\n- Automatically handle blocks, CAPTCHAs, and anti-bot systems\n- ..."
}

The key-value store also holds:

• SEARCH - the search engine used and all organic results it returned (title, URL, description, position).
• ERRORS - results that were skipped (page failed, blocked, not found or empty), with the reason. These are not billed.

Calling it from code

Run it synchronously and get the pages in one HTTP call:

POST https://api.apify.com/v2/acts/scrapeunblocker~rag-web-browser/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>
Content-Type: application/json
{ "query": "latest EU AI act news", "maxResults": 3 }

Pricing

$1.80 per 1000 pages. You pay only for pages delivered to the dataset; the search itself and skipped results are free.

Notes

• Only HTML pages are read; PDF and other file results are skipped.
• country changes the search market only; pages are always loaded through the most reliable exit.
• Each page is loaded once (no clicking or infinite scrolling).