# AI Search Scraper — ChatGPT, Perplexity & Google AI Overviews (`memo23/ai-search-scraper`) Actor

Scrape ChatGPT Search, Perplexity and Google AI Overview answers for any query: the AI answer, cited sources and source URLs, in one run. One schema across engines. No keys or login. Bulk queries. JSON, CSV, Excel.

- **URL**: https://apify.com/memo23/ai-search-scraper.md
- **Developed by:** [Muhamed Didovic](https://apify.com/memo23) (community)
- **Categories:** AI, SEO tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 chatgpt answers

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Search Scraper — ChatGPT, Perplexity & Google AI Overviews

Send any query and get back the **AI-generated answer**, its **cited sources** and the **source URLs** — from **ChatGPT Search**, **Perplexity** and **Google AI Overviews** in a single run. One consistent schema across every engine. No API keys, no login. JSON / CSV / Excel out.

![How the AI Search Scraper works](https://raw.githubusercontent.com/muhamed-didovic/muhamed-didovic.github.io/main/assets/how-it-works-ai-search.png)

### Why use the AI Search Scraper?

- **Three AI engines, one run.** ChatGPT Search, Perplexity and Google AI Overviews for the same query in a single call — not three separate actors, three datasets, three schemas.
- **The full answer, not just a snippet.** Complete AI answer text, and every cited source with its URL — plus title, date and the snippet the engine grounded on, where it gives them.
- **One unified schema.** Every engine returns the same row shape — `answer`, `sources`, `sourceUrls` — so downstream code doesn't branch per engine.
- **No keys, no login.** All three engines run out of the box on a paid Apify plan. On a free plan ChatGPT Search and Google AI Overviews still run; Perplexity calls a paid API per query, so it needs a paid plan or your own Perplexity key.
- **Built for scale.** Bulk queries, parallel workers, and success-gated billing — you only pay for answers that actually come back.

### Overview

AI assistants are the new search box. When a buyer asks ChatGPT "what's the best CRM?", or reads Google's AI Overview, the answer they see — and the sources it cites — decides who gets discovered. This actor captures that answer layer at scale: give it any query and it returns exactly what each AI engine said and which web pages it drew from, ready for GEO/AEO monitoring, competitive research, or feeding a RAG pipeline.

It is a **raw per-query search scraper** — the actual answer and sources for any query — not a brand-visibility score. (If you want an aggregated citation-share metric across engines, see the AI Visibility Tracker.)

### Supported inputs

- **Queries** — any prompts, one per line. Bulk-friendly.
- **Engines** — pick any of `ChatGPT Search`, `Google AI Overviews`, `Perplexity`. ChatGPT Search + Google AI Overviews are on by default.
- **Perplexity options** — recency filter (day/week/month/year), related questions, images. Requires your own Perplexity API key.
- **Google options** — country (`gl`) and language (`hl`) for the AI Overview engine.
- **Proxy** — advanced override for the exit used by the scrape engines (sensible defaults are built in).

### Use cases

- **GEO / AEO monitoring** — track what ChatGPT and Google's AI Overview answer for your category queries, and which domains they cite, over time.
- **Competitive answer tracking** — see whether you or a rival shows up in the AI answer for high-intent prompts.
- **Source / citation mining** — harvest the third-party pages (Reddit, review sites, docs) that feed AI answers, so you know where to earn mentions.
- **Content & PR strategy** — compare which sources each engine cites for the same question and cover the gaps.
- **RAG & datasets** — build a clean, per-engine answer+sources dataset to ground your own LLM app.

### How it works

1. **Input** — provide queries (one per line) and choose your engines.
2. **Engines** — each (query × engine) task runs on its own worker; answers are pushed the instant they land. None of the three needs any setup from you; if you supply your own Perplexity key it is used instead of ours.
3. **Output** — every answer becomes one flat row in the same shape, tagged by `engine`, ready to export.

### Input configuration

| Field | Type | Description |
|---|---|---|
| `queries` | string | Prompts to send, one per line. |
| `engines` | array | Which engines to query: `chatgpt`, `perplexity`, `google-ai-overview`. Default `["chatgpt","google-ai-overview"]`. |
| `perplexityApiKey` | string | Optional. Bill Perplexity usage to your own account instead — and enable the engine on a free Apify plan. |
| `perplexityModel` | string | Perplexity depth: `fast` (default), `low`, `medium`, `high`, `xhigh`. |
| `googleCountryCode` | string | Google country for AI Overview, e.g. `us`. |
| `googleLanguageCode` | string | Google language for AI Overview, e.g. `en`. |
| `maxItems` | integer | Hard cap on total answer rows. |
| `maxConcurrency` | integer | Parallel `(query × engine)` workers. |
| `proxy` | object | Advanced: override the exit for the scrape engines. |

#### Example input

```json
{
  "queries": "best CRM tools\nbest project management software",
  "engines": ["chatgpt", "google-ai-overview"]
}
```

### Output overview

One row per query per engine, identical shape across engines, discriminated by `engine` (`chatgpt` · `google-ai-overview` · `perplexity`).

### Output samples

**ChatGPT Search** (`engine: "chatgpt"`)

```json
{
  "query": "best CRM tools",
  "engine": "chatgpt",
  "answer": "# Best CRM tools in 2026\n\nThe best CRM depends on your team size and workflow. Top picks: Salesforce, HubSpot, Zoho CRM, Pipedrive…",
  "sources": [
    { "url": "https://www.techradar.com/best/the-best-crm-software", "domain": "techradar.com", "title": "The best CRM software" },
    { "url": "https://www.capterra.com/resources/best-crm-software-reviews-analysis/", "domain": "capterra.com", "title": "Best CRM Software 2026" }
  ],
  "sourceUrls": ["https://www.techradar.com/best/the-best-crm-software", "https://www.capterra.com/resources/best-crm-software-reviews-analysis/"],
  "followUpQuestions": [],
  "model": "gpt-6",
  "scrapedAt": "2026-10-31T06:00:00.000Z"
}
```

**Google AI Overview** (`engine: "google-ai-overview"`)

```json
{
  "query": "best CRM tools",
  "engine": "google-ai-overview",
  "answer": "Here are the top CRM software platforms for different business needs: Salesforce (best for large enterprises), HubSpot (best all-in-one), Zoho CRM, Pipedrive…",
  "sources": [
    { "url": "https://www.salesforce.com/crm/best-crm/", "domain": "salesforce.com", "title": "Salesforce CRM Overview" },
    { "url": "https://zapier.com/blog/best-crm-app/", "domain": "zapier.com", "title": "The 11 best CRM software in 2026 | Zapier" }
  ],
  "sourceUrls": ["https://www.salesforce.com/crm/best-crm/", "https://zapier.com/blog/best-crm-app/"],
  "followUpQuestions": [],
  "model": "google-ai-overview",
  "scrapedAt": "2026-10-31T06:00:00.000Z"
}
```

**Perplexity** (`engine: "perplexity"`, requires your API key)

```json
{
  "query": "best CRM tools",
  "engine": "perplexity",
  "answer": "The top CRMs are Salesforce, HubSpot and Pipedrive…",
  "sources": [
    { "title": "Salesforce CRM", "url": "https://www.salesforce.com/crm/", "domain": "salesforce.com", "date": "2025-02-01" }
  ],
  "sourceUrls": ["https://www.salesforce.com/crm/"],
  "followUpQuestions": [],
  "model": "openai/gpt-6-luna",
  "scrapedAt": "2026-10-31T06:00:00.000Z"
}
```

### Key output fields

| Field | Type | Description |
|---|---|---|
| `query` | string | The query that was sent. |
| `engine` | string | `chatgpt` · `google-ai-overview` · `perplexity`. |
| `answer` | string | null | The AI answer text. `null` only when an engine returned sources but no prose for that query. |
| `sources` | array | Cited sources: `url` + `domain`, plus `title` / `snippet` / `date` where available. |
| `sourceUrls` | array | Flat, ordered list of the cited source URLs. |
| `followUpQuestions` | array | Reserved. The engines no longer publish related questions, so this is currently always empty; the field stays so the row shape does not change under you. |
| `images` | array | Images returned alongside the answer (Perplexity, when requested). |
| `model` | string | The model / engine version that produced the answer. |
| `scrapedAt` | string | ISO-8601 timestamp. |

### FAQ

**Do I need a ChatGPT / OpenAI / Google API key?** No. All three engines run with no key and no login on a paid Apify plan. On a free plan, Perplexity is the one engine that needs either a paid plan or your own Perplexity key, because it calls a paid API for every query.

**What does the ChatGPT answer look like?** The same public web-search answer you'd read in the browser — the full answer text plus the web pages it cited — returned as clean, structured data.

**Does Google AI Overview return the answer text?** Yes — both the AI Overview answer and its cited sources. Not every query triggers an AI Overview on Google; when one isn't shown for a query, that query simply returns no row for this engine.

**How am I billed?** Pay-per-result: a small actor-start fee plus a per-answer charge for each engine, only when an answer is actually returned. Failed or empty calls are not charged.

**Can I run many queries at once?** Yes — paste as many as you like, one per line, and raise `maxConcurrency` for throughput.

**Are answers deterministic?** No — AI engines vary their wording between runs. The cited sources are generally more stable than the exact prose.

### Support

Found a bug or want a field added? Open an issue on the actor's **Issues** tab and it gets picked up.

### Additional services

Need lead data, SERPs, or brand visibility too? See **Google Search Results Scraper**, **AI Visibility Tracker**, and the rest of the portfolio on the Apify Store.

### Explore more scrapers

- **Google Search Results Scraper** — organic, ads, People Also Ask, knowledge panel, AI Overviews.
- **AI Visibility Tracker** — brand citation share across ChatGPT, Gemini, Perplexity, Copilot, Grok & Google AI.

### 🤖 For AI agents & LLM apps

This actor is agent-ready. It takes a plain list of queries and returns clean, typed JSON (`answer`, `sources`, `sourceUrls`) with one consistent schema across engines — ideal as a **web-grounded answer tool** for an agent, or as a **retrieval source for a RAG pipeline**. Call it over the Apify API or via [Apify MCP](https://docs.apify.com/platform/integrations/mcp) and let your agent run and schedule it.

### ⚠️ Disclaimer

This actor collects publicly accessible AI-generated answers and their cited sources for research, monitoring, and analysis. It does not access private accounts or bypass logins. AI answers are probabilistic and vary between runs; verify anything you rely on. You are responsible for using the output in compliance with the source services' terms and with applicable law (including copyright and data-protection rules). No affiliation with OpenAI, Google, or Perplexity is implied.

### SEO Keywords

ChatGPT search scraper, AI search scraper, AI answer scraper, scrape ChatGPT answers, Google AI Overview scraper, Google AI Overviews, Perplexity scraper, GEO tool, AEO tool, generative engine optimization, answer engine optimization, AI citations, AI search monitoring, LLM answer scraper, cited sources scraper, AI SERP, ChatGPT sources, RAG data source, AI visibility, AI search results API.

# Actor input Schema

## `queries` (type: `string`):

The prompts to send to the AI engines. Enter one query per line.

## `engines` (type: `array`):

Which answer engines to query. <b>ChatGPT</b> and <b>Google AI Overviews</b> are included on every plan. <b>Perplexity</b> calls a paid API per query, so it needs a paid Apify plan — or your own Perplexity key in the field below, which works on any plan. On a free plan without a key, Perplexity is skipped and the other engines still run.

## `perplexityModel` (type: `string`):

How hard Perplexity works on each query: <code>fast</code> (default), <code>low</code>, <code>medium</code>, <code>high</code> or <code>xhigh</code>. Deeper settings search more and take longer. Older Sonar model names are still accepted and mapped to the nearest depth.

## `perplexityApiKey` (type: `string`):

Leave empty on a paid plan and this Actor supplies the key. Paste your own key from console.perplexity.ai to bill Perplexity usage to your own account instead — this also enables the Perplexity engine on a free Apify plan.

## `googleCountryCode` (type: `string`):

Google country code for the AI Overview engine, e.g. `us`, `gb`, `de`. Default `us`.

## `googleLanguageCode` (type: `string`):

Google UI language for the AI Overview engine, e.g. `en`. Default `en`.

## `maxItems` (type: `integer`):

Hard cap on total answer rows across all queries and engines.

## `maxConcurrency` (type: `integer`):

How many (query × engine) tasks run in parallel.

## `proxy` (type: `object`):

Advanced: override the exit used by ChatGPT (and Google AI Overview citation resolution). By default ChatGPT uses Apify Residential and Google AI Overview uses the Apify Google SERP proxy. Perplexity runs direct via API.

## Actor input object example

```json
{
  "queries": "best SEO tools\napify web scraping",
  "engines": [
    "chatgpt",
    "google-ai-overview"
  ],
  "googleCountryCode": "us",
  "googleLanguageCode": "en",
  "maxItems": 10000,
  "maxConcurrency": 4,
  "proxy": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": "What are the best CRM tools",
    "proxy": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("memo23/ai-search-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": "What are the best CRM tools",
    "proxy": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("memo23/ai-search-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": "What are the best CRM tools",
  "proxy": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}' |
apify call memo23/ai-search-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,memo23/ai-search-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/1TNOV9QRMdpl04D77/builds/V4IzRQyZuex4Mc9Mm/openapi.json