# SERP Scraper - Google, Bing & DuckDuckGo for LLM/RAG (`get_anything/serp-scraper`) Actor

Scrape search engine results by keyword: organic results (title, URL, snippet, position) and Related Searches. Bing (default, reliable) plus DuckDuckGo and best-effort Google, all rendered in a hardened browser. Any country & language, pagination, no API key. Ideal for SEO, RAG and LLM grounding.

- **URL**: https://apify.com/get_anything/serp-scraper.md
- **Developed by:** [Get Anything](https://apify.com/get_anything) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## SERP Scraper — Google, Bing & DuckDuckGo for LLM / RAG

Scrape **search engine results** by keyword and get clean, structured data — no API key, any country and language. Built as the reliable **"search" half of a RAG web browser**: feed live search results into ChatGPT / Claude, do rank tracking, or mine keyword ideas.

### Engines

| Engine | Reliability | Notes |
|--------|-------------|-------|
| **Bing** *(default)* | ✅ High | High-quality organic results, rendered in a hardened browser. Recommended for production. |
| **DuckDuckGo** | ✅ High | Privacy-friendly results, browser-rendered. |
| **Google** | ⚠️ Best-effort | Google aggressively CAPTCHA-walls cloud/datacenter IPs. This engine renders the basic (`gbv=1`) page and retries on a fresh residential IP, but success is **not guaranteed** — use Bing for reliable results. |

> Why not Google-only? Google serves a `/sorry` CAPTCHA to virtually all cloud IPs. No scraper can reliably beat that without a paid CAPTCHA-solving service, so this actor defaults to **Bing**, which returns comparable results on every run.

### What you get

One item per organic result:

| Field | Description |
|-------|-------------|
| `query` | The search query |
| `engine` | Engine used (`bing` / `duckduckgo` / `google`) |
| `position` | Rank on the SERP (1-based, across pages) |
| `title` | Result title |
| `url` | Destination URL |
| `displayedUrl` | Breadcrumb URL shown by the engine (when available) |
| `domain` | Hostname of the result |
| `snippet` | Result snippet text |
| `relatedSearches` | *(first result only)* related search phrases |
| `scrapedAt` | ISO 8601 timestamp |

### Input

```json
{
  "queries": ["best web scraping tools", "apify alternatives"],
  "engine": "bing",
  "maxResults": 20,
  "country": "us",
  "language": "en",
  "includeRelatedSearches": true,
  "maxRetries": 3,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}
```

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `queries` | array | — | One or more search queries (supports `site:`, `intitle:`, quotes) |
| `engine` | string | `bing` | `bing`, `duckduckgo`, or `google` |
| `maxResults` | integer | 10 | Organic results per query (paginates to reach it) |
| `country` | string | `us` | Two-letter country code |
| `language` | string | `en` | Two-letter language code |
| `includeRelatedSearches` | boolean | true | Collect "Related searches" |
| `maxRetries` | integer | 3 | Retries on a fresh IP when blocked (mainly Google) |
| `proxyConfiguration` | object | Residential | Proxy — residential strongly recommended |

### Use cases

- **LLM / RAG grounding** — feed live search results into ChatGPT / Claude pipelines.
- **SEO & rank tracking** — monitor rankings for target keywords across countries.
- **Market & competitor research** — capture Related Searches for content and keyword ideas.
- **Content & lead discovery** — harvest ranking pages for a topic at scale.

### 🤖 Use with Claude or ChatGPT (MCP)

Run this actor straight from Claude, ChatGPT, Cursor or any MCP client via the [Apify MCP server](https://mcp.apify.com). In **Claude Desktop**: Settings → Connectors → Add custom connector → `https://mcp.apify.com`, then ask it to search the web. Or expose just this tool:

```json
{ "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=get_anything/serp-scraper" } } }
```

Full guide: [Connect Apify actors to Claude & ChatGPT](https://dev.to/get_anything/connect-any-apify-scraper-to-claude-or-chatgpt-in-2-minutes-mcp-37he).

### Notes

- Only public search-results-page data is collected.
- Residential proxy is strongly recommended for all engines and is what makes Google's retry logic effective.

# Actor input Schema

## `queries` (type: `array`):

One or more search queries, e.g. 'best CRM for startups', 'apify actors'. Supports operators like site:, intitle:, quotes.

## `engine` (type: `string`):

Bing (default) returns high-quality organic results reliably. DuckDuckGo is a reliable privacy-friendly alternative. Google is best-effort: it aggressively CAPTCHA-walls cloud IPs, so use Bing for production.

## `maxResults` (type: `integer`):

Number of organic results to return per query. The actor paginates (\&start=) until it reaches this many.

## `language` (type: `string`):

Two-letter UI language code, e.g. 'en', 'de', 'ar'.

## `country` (type: `string`):

Two-letter country code to localise results, e.g. 'us', 'gb', 'ae'.

## `includeRelatedSearches` (type: `boolean`):

Collect the 'Related searches' phrases shown at the bottom of the SERP.

## `maxRetries` (type: `integer`):

If an engine blocks the request (mainly Google's /sorry CAPTCHA wall), retry the query with a fresh browser context on a new proxy IP up to this many times. Bing/DuckDuckGo rarely block.

## `proxyConfiguration` (type: `object`):

Proxy used to render the SERP. Residential proxies are strongly recommended - they are what makes the CAPTCHA retry work.

## Actor input object example

```json
{
  "queries": [
    "web scraping tools"
  ],
  "engine": "bing",
  "maxResults": 10,
  "language": "en",
  "country": "us",
  "includeRelatedSearches": true,
  "maxRetries": 3,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Structured SERP results in the default dataset - one item per organic result, with People Also Ask and Related Searches attached to the first result of each query.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "web scraping tools"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("get_anything/serp-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "queries": ["web scraping tools"] }

# Run the Actor and wait for it to finish
run = client.actor("get_anything/serp-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "web scraping tools"
  ]
}' |
apify call get_anything/serp-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,get_anything/serp-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/KjwrAdrWRPIlk2UbO/builds/Mk262Q1VjqZhJHuNl/openapi.json