Web Search API for AI Agents & RAG | Markdown | $1 / 1K pages

Web search API for AI agents and RAG: query in, clean Markdown out

Give your LLM, AI agent or RAG pipeline live web results without an API key. Send a question, and this Actor searches the web, opens the top result pages and returns each page as clean Markdown (or plain text / HTML) together with the search title, URL and snippet. Think of it as a pay-per-page alternative to Tavily, Exa or a search API plus a scraper, built for grounding answers with fresh sources.

• Search + content in one call: top 1-20 result pages per query, main content only (no menus, ads, cookie banners or footers).
• No API key, no browser: searches DuckDuckGo, and switches to Brave Search and then Yahoo when an engine is busy. It does not use Google.
• Pay only for pages with real content: blocked, JavaScript-only, empty or failed pages are returned as free status rows, and by default the next search result is opened instead so you still get the number of pages you asked for.
• Search-results-only mode: just rank, title, URL and snippet, billed per query.
• Direct URLs: put a URL in the list and it is fetched and converted to Markdown without searching.
• Agent friendly: small input, one flat row per page, works through the Apify MCP server, API, LangChain, n8n, Make and Zapier.

What people use it for

• Grounding an AI agent or chatbot with up-to-date web pages (RAG, "search the web" tool for ChatGPT / Claude / custom agents)
• Research assistants that read the top 3-10 sources for a question and summarise them
• Collecting web content for a vector database or knowledge base on a topic
• Fact checking and citation finding (title + URL + content for each source)
• Monitoring what ranks for a set of keywords (search-results-only mode, scheduled)
• Searching inside one website with the site filter (docs, blogs, help centers)

What data you get

One row per result page:

Field	What it is
query	Your query (or URL)
rank	Position in the search results (1 = top)
searchEngine	DuckDuckGo, Brave Search or Yahoo Search
searchTitle, searchSnippet	Title and snippet from the results page
url, loadedUrl	Result URL and final URL after redirects
fetchStatus	ok (billed) or a free status: blocked, needs-javascript, thin, http-error, timeout, not-html, search-failed, no-results; not-fetched in search-results-only mode
title, description, language, publishedTime	Page metadata
markdown / text / html	Main content in the formats you picked
wordCount, textLength, truncated	Size of the content, and whether it was cut to your character limit
charged	Whether the row was billed

Input

Field	Default	Notes
queries	required	Search queries, one per line (up to 100). A line that is a URL is fetched directly.
maxResults	3	Pages per query (1-20).
outputFormats	markdown	Any of markdown, text, html.
searchResultsOnly	false	Only list results (title, URL, snippet), billed per query.
site	empty	Restrict results to one domain, e.g. docs.python.org.
country	any	Region and language of results (DuckDuckGo and Brave; the Yahoo fallback is US only).
searchEngine	auto	auto, duckduckgo, brave or yahoo.
replaceFailedPages	true	Open the next result when a page cannot be read.
requestTimeoutSecs	15	Per page, 5-30 seconds.
maxCharactersPerPage	0 (no limit)	Cut content to save LLM tokens.
includeImages	false	Keep image links in the Markdown.
maxConcurrency	5	Parallel page downloads.

Example input

{
  "queries": ["what is retrieval augmented generation", "https://docs.apify.com/platform/actors"],
  "maxResults": 3,
  "outputFormats": ["markdown"],
  "maxCharactersPerPage": 8000
}

Output example

A real row from a run on 21 October 2026 (maxCharactersPerPage: 400):

{
  "query": "what is retrieval augmented generation",
  "rank": 1,
  "searchEngine": "DuckDuckGo",
  "searchTitle": "What is Retrieval-Augmented Generation (RAG) - GeeksforGeeks",
  "searchSnippet": "Retrieval-Augmented Generation (RAG) is a way to make AI answers more reliable by combining searching for relevant information and then generating a response. Instead of guessing based only on old training data, it first finds useful data from external sources (like documents or databases) and then uses it to give a better answer.",
  "url": "https://www.geeksforgeeks.org/nlp/what-is-retrieval-augmented-generation-rag/",
  "loadedUrl": "https://www.geeksforgeeks.org/nlp/what-is-retrieval-augmented-generation-rag/",
  "httpStatus": 200,
  "fetchStatus": "ok",
  "title": "What is Retrieval-Augmented Generation (RAG) - GeeksforGeeks",
  "language": "en",
  "publishedTime": "2024-03-27 08:55:19+00:00",
  "markdown": "Retrieval-Augmented Generation (RAG) is a way to make AI answers more reliable by combining searching for relevant information and then generating a response. Instead of guessing based only on old training data, it first finds useful data from external sources (like documents or databases) and then uses it to give a better answer.\n\n-   Fetches up-to-date data and reduces incorrect or made-up answe",
  "wordCount": 909,
  "textLength": 6333,
  "truncated": true,
  "error": null,
  "charged": true
}

Search-results-only row (from a real run):

{
  "query": "vector database",
  "rank": 1,
  "searchEngine": "DuckDuckGo",
  "searchTitle": "Vector database - Wikipedia",
  "searchSnippet": "A vector database, vector store or vector search engine is a database that stores and retrieves embeddings of data in vector space. ...",
  "url": "https://en.wikipedia.org/wiki/Vector_database",
  "fetchStatus": "not-fetched",
  "charged": false
}

Pricing

Pay per event, no subscription:

• $0.003 per run (start fee)
• $0.001 per page returned with real content ($1 per 1,000 pages)
• $0.001 per query in search-results-only mode (up to 20 results per query)
• Free: failed searches, queries with no results, and pages that were blocked, need JavaScript, were empty or failed to load

Worked examples:

Run	Cost
1 question, top 3 pages	$0.003 + 3 x $0.001 = $0.006
100 questions, top 3 pages each	$0.003 + 300 x $0.001 = $0.303
1,000 queries, search results only	$0.003 + 1,000 x $0.001 = $1.003

For comparison, Apify's RAG Web Browser (checked 21 October 2026) charges a search event per query ($0.00348 on the Free plan, $0.00203 on Gold) plus $0.00228 (Free) to $0.00133 (Gold) per page. For 3 pages per query that is about $0.0103 (Free) to $0.0060 (Gold) per query, against $0.003 here. You can cap any run with Apify's maximum cost per run setting; the Actor stops fetching pages it could not bill.

How it works

1. Each query goes to DuckDuckGo's HTML search through a fresh Apify datacenter proxy session. If DuckDuckGo shows its bot check or rate limit, the query moves to Brave Search, then Yahoo. Requests to each engine are spaced out and an engine that just refused is rested for a while. No CAPTCHA solving.
2. The top result pages are downloaded over plain HTTP (no browser), the main content is picked out (article/main element, else Mozilla Readability) and converted to GitHub-flavoured Markdown with headings, lists, tables, links and code blocks.
3. Pages behind a bot wall, JavaScript-only apps and empty pages are detected and returned as free rows; the next result is opened instead.

In our tests inside Apify on 21 October 2026, 158 out of 158 varied queries (English plus German, French, Spanish, Portuguese, Italian, Dutch, Polish, Turkish, Japanese, Chinese, Korean and Russian) returned relevant results, and 90 of 90 requested pages came back with content for a 30-query run (9 blocked pages were replaced automatically). That run took 75 seconds.

Use it from AI agents, code and no-code tools

• Apify MCP server: add https://mcp.apify.com?tools=hridayrungta/rag-web-search to Claude, Cursor, VS Code or any MCP client, and the agent can call it as a web search tool.
• API (one call, results in the response): POST https://api.apify.com/v2/acts/hridayrungta~rag-web-search/run-sync-get-dataset-items?token=<YOUR_TOKEN> with the input JSON as the body.
• Python / JavaScript: the apify-client package (client.actor("hridayrungta/rag-web-search").call(run_input={...})), or LangChain's Apify integration to load the rows as documents.
• n8n, Make, Zapier, Google Sheets, webhooks: use the Apify app or the run-sync endpoint above.
• Schedules: use Apify Schedules to run a list of queries daily or weekly, for example to track what ranks for your keywords (search-results-only mode).

FAQ

Is this a Tavily or Exa alternative?

It does the same core job (search plus page content for LLMs) with pay-per-page pricing and no API key or monthly plan. It does not generate an AI answer or rank pages with its own neural index: you get the search engine's ranking and the clean page content, and your own LLM does the answering.

Which search engine powers the results? Is it Google?

Not Google. Results come from DuckDuckGo by default, with Brave Search and Yahoo as fallbacks, and every row says which engine answered. Rankings therefore differ from Google's. If you need Google results specifically, use a Google SERP Actor such as Apify's Google Search Results Scraper.

Why not use an official search API?

Microsoft retired the Bing Search APIs in August 2025. Brave and Tavily offer APIs that need a key and a plan, and they return snippets rather than the full page as Markdown. This Actor needs no key and you pay only for pages that came back with content.

How fast is it?

A run with one query and three pages took about 5-6 seconds in our tests, including start-up. Searches are spaced out per engine, so large lists run at roughly 2-3 seconds per query. There is no Standby (always-on HTTP) mode yet.

What does it not do?

• No JavaScript rendering: pages that only render in a browser come back as free needs-javascript rows (and are replaced by the next result).
• Some sites with strong bot protection (for example certain Cloudflare-protected sites) return free blocked rows.
• No PDF or Word parsing (returned as free not-html rows).
• No ads, featured snippets, AI overviews or other SERP extras: organic results only.
• Up to 20 results per query; the region setting is applied by DuckDuckGo and Brave, the Yahoo fallback uses the US index.

Is it legal?

It reads public search result pages and public web pages, like a browser would. You are responsible for how you use the content, including the terms of the sites you fetch and copyright of their text.

Related Actors

• Website to Markdown Crawler: crawl a whole site (sitemap, llms.txt) into Markdown for RAG
• Contact Info Scraper: emails, phones and social links from websites
• YouTube Transcript Scraper: video transcripts for LLMs
• RAG Web Browser: Apify's Google-based search + browser tool, with JavaScript rendering
• Website Content Crawler: Apify's full crawler with a headless browser option
• Google Search Results Scraper: when you need Google's own results