# Web Search API for AI Agents & RAG | Markdown | $1 / 1K pages (`hridayrungta/rag-web-search`) Actor

Web search API for AI agents and RAG: send a query, get the top result pages as clean Markdown with title, URL and snippet. DuckDuckGo with Brave and Yahoo fallback, no API key, no browser. $1 per 1,000 pages, failed pages free.

- **URL**: https://apify.com/hridayrungta/rag-web-search.md
- **Developed by:** [Hriday Rungta](https://apify.com/hridayrungta) (community)
- **Categories:** AI, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 page fetcheds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Web search API for AI agents and RAG: query in, clean Markdown out

Give your LLM, AI agent or RAG pipeline live web results without an API key. Send a question, and this Actor searches the web, opens the top result pages and returns each page as **clean Markdown** (or plain text / HTML) together with the search **title, URL and snippet**. Think of it as a pay-per-page alternative to Tavily, Exa or a search API plus a scraper, built for grounding answers with fresh sources.

- **Search + content in one call:** top 1-20 result pages per query, main content only (no menus, ads, cookie banners or footers).
- **No API key, no browser:** searches **DuckDuckGo**, and switches to **Brave Search** and then **Yahoo** when an engine is busy. It does **not** use Google.
- **Pay only for pages with real content:** blocked, JavaScript-only, empty or failed pages are returned as free status rows, and by default the next search result is opened instead so you still get the number of pages you asked for.
- **Search-results-only mode:** just rank, title, URL and snippet, billed per query.
- **Direct URLs:** put a URL in the list and it is fetched and converted to Markdown without searching.
- **Agent friendly:** small input, one flat row per page, works through the Apify MCP server, API, LangChain, n8n, Make and Zapier.

#### What people use it for

- Grounding an AI agent or chatbot with up-to-date web pages (RAG, "search the web" tool for ChatGPT / Claude / custom agents)
- Research assistants that read the top 3-10 sources for a question and summarise them
- Collecting web content for a vector database or knowledge base on a topic
- Fact checking and citation finding (title + URL + content for each source)
- Monitoring what ranks for a set of keywords (search-results-only mode, scheduled)
- Searching inside one website with the site filter (docs, blogs, help centers)

### What data you get

One row per result page:

| Field | What it is |
|---|---|
| `query` | Your query (or URL) |
| `rank` | Position in the search results (1 = top) |
| `searchEngine` | DuckDuckGo, Brave Search or Yahoo Search |
| `searchTitle`, `searchSnippet` | Title and snippet from the results page |
| `url`, `loadedUrl` | Result URL and final URL after redirects |
| `fetchStatus` | `ok` (billed) or a free status: `blocked`, `needs-javascript`, `thin`, `http-error`, `timeout`, `not-html`, `search-failed`, `no-results`; `not-fetched` in search-results-only mode |
| `title`, `description`, `language`, `publishedTime` | Page metadata |
| `markdown` / `text` / `html` | Main content in the formats you picked |
| `wordCount`, `textLength`, `truncated` | Size of the content, and whether it was cut to your character limit |
| `charged` | Whether the row was billed |

### Input

| Field | Default | Notes |
|---|---|---|
| `queries` | required | Search queries, one per line (up to 100). A line that is a URL is fetched directly. |
| `maxResults` | 3 | Pages per query (1-20). |
| `outputFormats` | markdown | Any of markdown, text, html. |
| `searchResultsOnly` | false | Only list results (title, URL, snippet), billed per query. |
| `site` | empty | Restrict results to one domain, e.g. `docs.python.org`. |
| `country` | any | Region and language of results (DuckDuckGo and Brave; the Yahoo fallback is US only). |
| `searchEngine` | auto | auto, duckduckgo, brave or yahoo. |
| `replaceFailedPages` | true | Open the next result when a page cannot be read. |
| `requestTimeoutSecs` | 15 | Per page, 5-30 seconds. |
| `maxCharactersPerPage` | 0 (no limit) | Cut content to save LLM tokens. |
| `includeImages` | false | Keep image links in the Markdown. |
| `maxConcurrency` | 5 | Parallel page downloads. |

#### Example input

```json
{
  "queries": ["what is retrieval augmented generation", "https://docs.apify.com/platform/actors"],
  "maxResults": 3,
  "outputFormats": ["markdown"],
  "maxCharactersPerPage": 8000
}
```

### Output example

A real row from a run on 21 October 2026 (`maxCharactersPerPage: 400`):

```json
{
  "query": "what is retrieval augmented generation",
  "rank": 1,
  "searchEngine": "DuckDuckGo",
  "searchTitle": "What is Retrieval-Augmented Generation (RAG) - GeeksforGeeks",
  "searchSnippet": "Retrieval-Augmented Generation (RAG) is a way to make AI answers more reliable by combining searching for relevant information and then generating a response. Instead of guessing based only on old training data, it first finds useful data from external sources (like documents or databases) and then uses it to give a better answer.",
  "url": "https://www.geeksforgeeks.org/nlp/what-is-retrieval-augmented-generation-rag/",
  "loadedUrl": "https://www.geeksforgeeks.org/nlp/what-is-retrieval-augmented-generation-rag/",
  "httpStatus": 200,
  "fetchStatus": "ok",
  "title": "What is Retrieval-Augmented Generation (RAG) - GeeksforGeeks",
  "language": "en",
  "publishedTime": "2024-03-27 08:55:19+00:00",
  "markdown": "Retrieval-Augmented Generation (RAG) is a way to make AI answers more reliable by combining searching for relevant information and then generating a response. Instead of guessing based only on old training data, it first finds useful data from external sources (like documents or databases) and then uses it to give a better answer.\n\n-   Fetches up-to-date data and reduces incorrect or made-up answe",
  "wordCount": 909,
  "textLength": 6333,
  "truncated": true,
  "error": null,
  "charged": true
}
```

Search-results-only row (from a real run):

```json
{
  "query": "vector database",
  "rank": 1,
  "searchEngine": "DuckDuckGo",
  "searchTitle": "Vector database - Wikipedia",
  "searchSnippet": "A vector database, vector store or vector search engine is a database that stores and retrieves embeddings of data in vector space. ...",
  "url": "https://en.wikipedia.org/wiki/Vector_database",
  "fetchStatus": "not-fetched",
  "charged": false
}
```

### Pricing

Pay per event, no subscription:

- **$0.003** per run (start fee)
- **$0.001 per page** returned with real content ($1 per 1,000 pages)
- **$0.001 per query** in search-results-only mode (up to 20 results per query)
- Free: failed searches, queries with no results, and pages that were blocked, need JavaScript, were empty or failed to load

Worked examples:

| Run | Cost |
|---|---|
| 1 question, top 3 pages | $0.003 + 3 x $0.001 = **$0.006** |
| 100 questions, top 3 pages each | $0.003 + 300 x $0.001 = **$0.303** |
| 1,000 queries, search results only | $0.003 + 1,000 x $0.001 = **$1.003** |

For comparison, Apify's RAG Web Browser (checked 21 October 2026) charges a search event per query ($0.00348 on the Free plan, $0.00203 on Gold) plus $0.00228 (Free) to $0.00133 (Gold) per page. For 3 pages per query that is about $0.0103 (Free) to $0.0060 (Gold) per query, against $0.003 here. You can cap any run with Apify's maximum cost per run setting; the Actor stops fetching pages it could not bill.

### How it works

1. Each query goes to DuckDuckGo's HTML search through a fresh Apify datacenter proxy session. If DuckDuckGo shows its bot check or rate limit, the query moves to Brave Search, then Yahoo. Requests to each engine are spaced out and an engine that just refused is rested for a while. No CAPTCHA solving.
2. The top result pages are downloaded over plain HTTP (no browser), the main content is picked out (article/main element, else Mozilla Readability) and converted to GitHub-flavoured Markdown with headings, lists, tables, links and code blocks.
3. Pages behind a bot wall, JavaScript-only apps and empty pages are detected and returned as free rows; the next result is opened instead.

In our tests inside Apify on 21 October 2026, 158 out of 158 varied queries (English plus German, French, Spanish, Portuguese, Italian, Dutch, Polish, Turkish, Japanese, Chinese, Korean and Russian) returned relevant results, and 90 of 90 requested pages came back with content for a 30-query run (9 blocked pages were replaced automatically). That run took 75 seconds.

### Use it from AI agents, code and no-code tools

- **Apify MCP server:** add `https://mcp.apify.com?tools=hridayrungta/rag-web-search` to Claude, Cursor, VS Code or any MCP client, and the agent can call it as a web search tool.
- **API (one call, results in the response):** `POST https://api.apify.com/v2/acts/hridayrungta~rag-web-search/run-sync-get-dataset-items?token=<YOUR_TOKEN>` with the input JSON as the body.
- **Python / JavaScript:** the `apify-client` package (`client.actor("hridayrungta/rag-web-search").call(run_input={...})`), or LangChain's Apify integration to load the rows as documents.
- **n8n, Make, Zapier, Google Sheets, webhooks:** use the Apify app or the run-sync endpoint above.
- **Schedules:** use Apify Schedules to run a list of queries daily or weekly, for example to track what ranks for your keywords (search-results-only mode).

### FAQ

#### Is this a Tavily or Exa alternative?

It does the same core job (search plus page content for LLMs) with pay-per-page pricing and no API key or monthly plan. It does not generate an AI answer or rank pages with its own neural index: you get the search engine's ranking and the clean page content, and your own LLM does the answering.

#### Which search engine powers the results? Is it Google?

Not Google. Results come from DuckDuckGo by default, with Brave Search and Yahoo as fallbacks, and every row says which engine answered. Rankings therefore differ from Google's. If you need Google results specifically, use a Google SERP Actor such as Apify's Google Search Results Scraper.

#### Why not use an official search API?

Microsoft retired the Bing Search APIs in August 2025. Brave and Tavily offer APIs that need a key and a plan, and they return snippets rather than the full page as Markdown. This Actor needs no key and you pay only for pages that came back with content.

#### How fast is it?

A run with one query and three pages took about 5-6 seconds in our tests, including start-up. Searches are spaced out per engine, so large lists run at roughly 2-3 seconds per query. There is no Standby (always-on HTTP) mode yet.

#### What does it not do?

- No JavaScript rendering: pages that only render in a browser come back as free `needs-javascript` rows (and are replaced by the next result).
- Some sites with strong bot protection (for example certain Cloudflare-protected sites) return free `blocked` rows.
- No PDF or Word parsing (returned as free `not-html` rows).
- No ads, featured snippets, AI overviews or other SERP extras: organic results only.
- Up to 20 results per query; the region setting is applied by DuckDuckGo and Brave, the Yahoo fallback uses the US index.

#### Is it legal?

It reads public search result pages and public web pages, like a browser would. You are responsible for how you use the content, including the terms of the sites you fetch and copyright of their text.

### Related Actors

- [Website to Markdown Crawler](https://apify.com/hridayrungta/website-to-markdown-crawler): crawl a whole site (sitemap, llms.txt) into Markdown for RAG
- [Contact Info Scraper](https://apify.com/hridayrungta/contact-info-scraper): emails, phones and social links from websites
- [YouTube Transcript Scraper](https://apify.com/hridayrungta/youtube-transcript-scraper): video transcripts for LLMs
- [RAG Web Browser](https://apify.com/apify/rag-web-browser): Apify's Google-based search + browser tool, with JavaScript rendering
- [Website Content Crawler](https://apify.com/apify/website-content-crawler): Apify's full crawler with a headless browser option
- [Google Search Results Scraper](https://apify.com/apify/google-search-scraper): when you need Google's own results

# Actor input Schema

## `queries` (type: `array`):

What to search for, one query per line (for example "best vector database for RAG"). A line that is a full URL (https://...) is fetched directly instead of searched. Up to 100 per run.

## `maxResults` (type: `integer`):

How many top search results to open and return as Markdown for each query (1-20). In search-results-only mode, how many results to list.

## `outputFormats` (type: `array`):

Which content fields to return for each page: Markdown (best for LLMs), plain text, and/or the cleaned main-content HTML.

## `searchResultsOnly` (type: `boolean`):

Return only the search results (rank, title, URL, snippet) without opening the pages. Billed per query instead of per page. Faster and cheaper when you only need links.

## `site` (type: `string`):

Optional domain to restrict results to, for example docs.python.org. Added to every query as the site: operator.

## `country` (type: `string`):

Region of the search results (also picks the result language). Applied by DuckDuckGo and Brave; the Yahoo fallback always searches the US index. Leave empty for no region.

## `searchEngine` (type: `string`):

Auto uses DuckDuckGo first and switches to Brave Search, then Yahoo, when an engine is rate limiting. Pick one engine to use only that one.

## `replaceFailedPages` (type: `boolean`):

If a top result cannot be read (blocked, needs JavaScript, error), open the next search result instead so you still get the number of pages you asked for. Failed pages are returned as free status rows either way.

## `requestTimeoutSecs` (type: `integer`):

Maximum time to wait for one page to download (5-30). Slow pages are returned as free timeout rows and, if enabled, replaced by the next result.

## `maxCharactersPerPage` (type: `integer`):

Cut the Markdown / text / HTML of each page to this many characters to save LLM tokens (0 = no limit). Rows that were cut have truncated = true.

## `includeImages` (type: `boolean`):

Keep image links in the Markdown. Off by default because most LLM pipelines do not need them.

## `maxConcurrency` (type: `integer`):

How many result pages to download at the same time (1-10).

## Actor input object example

```json
{
  "queries": [
    "what is retrieval augmented generation",
    "https://docs.apify.com/platform/actors"
  ],
  "maxResults": 3,
  "outputFormats": [
    "markdown"
  ],
  "searchResultsOnly": false,
  "country": "",
  "searchEngine": "auto",
  "replaceFailedPages": true,
  "requestTimeoutSecs": 15,
  "maxCharactersPerPage": 0,
  "includeImages": false,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "what is retrieval augmented generation"
    ],
    "maxResults": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("hridayrungta/rag-web-search").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["what is retrieval augmented generation"],
    "maxResults": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("hridayrungta/rag-web-search").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "what is retrieval augmented generation"
  ],
  "maxResults": 3
}' |
apify call hridayrungta/rag-web-search --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hridayrungta/rag-web-search"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4yLbZIZG7jMLbTebI/builds/XB2bUcfcLuDmklXTn/openapi.json