RAG Web Browser

Search the web and get the top results' pages as clean Markdown, ready for LLMs, RAG pipelines, chatbots and AI agents, or give a URL and get that page:

• Search results: each result's title, URL, snippet and rank, from DuckDuckGo, with Brave Search for more results or when DuckDuckGo refuses. Search operators work: "exact phrase", site:, -site:.
• The page's own content, not its chrome: menus, headers, footers, sidebars, cookie banners and share buttons are left out, and headings, lists, tables, code blocks and links are kept, as Markdown, plain text or cleaned HTML.
• Metadata: the page's title, description, language, canonical URL, author and published date.
• Standby mode: an HTTP endpoint that answers GET /search?query=... with the results as JSON, for AI agents, custom GPTs and tools that call the web in the middle of a conversation.
• Inputs like Apify's RAG Web Browser: query, maxResults, outputFormats, requestTimeoutSecs, removeElementsCssSelector and the same result shape (searchResult, metadata, markdown), so an input or a parser made for it works here.

It is fast: plain HTTP with a real browser's TLS fingerprint and no headless browser. A search with 3 pages took 1–4 seconds, 10 pages 4 seconds, 20 pages 5 seconds. It costs $2.50 per 1,000 searches of 3 pages.

What people use it for

• Web search for AI agents: a tool an agent calls through MCP, LangChain, LlamaIndex or an OpenAI function to look things up and read the pages, not only the snippets.
• RAG with fresh content: the current pages about a topic, as Markdown for a prompt or a vector database.
• Grounding and fact-checking: the sources behind an answer, with their URLs and dates.
• Research and content briefs: what the top pages for a query say, side by side.
• SEO: which pages rank for a query, their titles, descriptions and word counts.

How to search the web and get the pages as Markdown

1. Click Try for free; Apify's free plan covers a first run.
2. Enter your Search terms or URL and how many results you want.
3. Click Start, then download the results as JSON, CSV, Excel or HTML, or read them through the API.

An input with more options:

{
  "query": "function calling site:openai.com",
  "maxResults": 5,
  "outputFormats": ["markdown", "text"],
  "includeImages": false,
  "removeElementsCssSelector": ".newsletter, #comments",
  "requestTimeoutSecs": 30,
  "maxRequestRetries": 1
}

A query that is a URL (https://docs.python.org/3/library/asyncio-task.html, or example.com/pricing) is fetched as it is, without a search.

Output

One item per result, in rank order (Markdown shortened here):

{
  "query": "retrieval augmented generation",
  "searchResult": {
    "title": "Retrieval-augmented generation - Wikipedia",
    "description": "Retrieval-augmented generation (RAG) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources. [1][2] With RAG, LLMs first refer to a specified set of documents, then respond to user queries.",
    "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "rank": 1,
    "engine": "duckduckgo"
  },
  "crawl": {
    "httpStatusCode": 200,
    "loadedUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "loadedAt": "2026-09-29T14:50:06Z",
    "requestStatus": "handled",
    "loadMs": 267
  },
  "metadata": {
    "title": "Retrieval-augmented generation - Wikipedia",
    "description": null,
    "languageCode": "en",
    "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "canonicalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "author": "Contributors to Wikimedia projects",
    "publishedAt": "2023-11-05T13:19:20Z"
  },
  "markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique that enables [large language models](https://en.wikipedia.org/wiki/Large_language_model) (LLMs) to retrieve and incorporate new information from external data sources.[1][2] With RAG, LLMs first refer to a specified set of documents, then respond to user queries. ...",
  "wordCount": 2406,
  "needsJavaScript": false,
  "warning": null
}

text and html come instead of or besides markdown when asked for. A result from Brave Search can carry the page's date. A page that can't be read keeps its searchResult and comes with an error ("Couldn't load the page: HTTP 404.", "Couldn't load the page: the site's bot check refused it.", "Not a web page (application/pdf): nothing to extract."), and isn't charged.

Use it as an API

One call runs it and returns the results as JSON. Apify's Python and JavaScript clients, LangChain and LlamaIndex loaders, and its Make, Zapier and n8n integrations work the same way.

curl -X POST "https://api.apify.com/v2/acts/pikoulas~rag-web-browser/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "retrieval augmented generation", "maxResults": 3}'

In Standby mode it answers straight away, without starting a run for each call; the input's fields are query parameters (outputFormats=markdown,text):

curl "https://pikoulas--rag-web-browser.apify.actor/search?query=retrieval%20augmented%20generation&maxResults=3" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN"

AI agents can call it as a tool through Apify's MCP server: https://mcp.apify.com?tools=pikoulas/rag-web-browser.

How much does it cost to search the web with it?

You pay $0.001 per search and $0.0005 per page: a search with 3 pages is $0.0025, which is $2.50 per 1,000. A URL given as the query is only the page, $0.0005. Pages that fail, and pages with next to no text (apps that need JavaScript), aren't charged, and neither is a search with no results. The same prices apply in Standby mode. Runs that go through Apify's residential proxy also pay its traffic, $0.0012 per 100 KB; a result page is 50–150 KB on the wire (about 70 KB on average) and a search 7–60 KB. There are no monthly fees.

Good to know

• Search engines: DuckDuckGo's first page gives up to about 10 results; Brave Search adds more, 20 a page. Google isn't used: it answers plain HTTP only with a JavaScript page. So the results are close to, but not the same as, Google's.
• No JavaScript. It reads the HTML a site sends, which for most sites (news, blogs, docs, Wikipedia, shops, forums) is the whole content. Apps that build their page in the browser come with needsJavaScript: true, next to no text and a warning, and aren't charged.
• Bot checks: sites behind Cloudflare's or DataDome's challenge, and Reddit's threads, refuse plain HTTP clients: those results come with an error and aren't charged. A proxy doesn't get past a challenge.
• Timeout: requestTimeoutSecs is the whole request's, the search included; the pages not loaded by then come with an error and the others are returned.
• Files: PDFs, images and other files aren't read; plain-text and Markdown files are given as they are.
• Standby: a Standby run serves requests until a minute passes without one; the request after that waits a few seconds for a new one to start. proxyConfiguration can't be set per request.
• Input fields of Apify's RAG Web Browser that don't apply here (scrapingTool, htmlTransformer, serpProxyGroup and the like) are ignored.

FAQ

Is it legal to scrape search results and web pages?

It fetches only public pages, without logging in. Scraping public data is generally legal, but personal data is protected by laws such as the GDPR and the CCPA, and a site's terms may limit how you use its content. If you're unsure, ask a lawyer; Apify's Is web scraping legal? is a good start.

Something missing or not working?

Open an issue on the Issues tab with the input you used, or ask there for a field you need.