# Universal AI GPT Scraper (`louisdeconinck/ai-gpt-scraper`) Actor

Transform any website into structured data with AI-powered extraction. This versatile tool combines advanced web scraping with intelligent content analysis to deliver clean, customized JSON output - perfect for automating data collection from any web source.

- **URL**: https://apify.com/louisdeconinck/ai-gpt-scraper.md
- **Developed by:** [Louis Deconinck](https://apify.com/louisdeconinck) (community)
- **Categories:** AI, Automation, Developer tools
- **Stats:** 193 total users, 0 monthly users, 100.0% runs succeeded, 9 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $50.00 / 1,000 actor starts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Transform any website into structured data effortlessly! This powerful Apify actor revolutionizes web scraping by combining AI with precise data extraction. Simply specify what data you need, and watch as advanced AI models intelligently parse web content into clean, structured JSON - saving you countless hours of manual data collection and processing. Perfect for businesses and developers who need reliable, automated data extraction without complex coding or maintenance.

### Use Cases

- Extract product information from e-commerce sites
- Gather pricing data from service providers
- Collect structured data from blog posts or articles
- Extract specific fields from documentation pages
- Convert any web content into structured JSON data

### Features

- 🎯 **Custom Field Extraction**: Define exactly what fields you want to extract
- 🤖 **AI-Powered**: Uses advanced language models to understand and extract content
- 📊 **Structured Output**: Get clean JSON or CSV data with your specified fields
- 🔄 **Type Support**: Specify the data type for each field (string, number, boolean, etc.)
- 🎛️ **Model Selection**: Choose from predefined AI models or use your own
- 🎯 **CSS Selector Support**: Target specific page elements using CSS selectors
- 🔒 **Secure**: Support for secret API keys and proxy configuration

### Input Configuration

#### Required Fields

- **URLs** (`array`): List of web pages to scrape
- **Fields** (`array`): Specification of fields to extract, each containing:
  - `name`: Field name in the output
  - `description`: Description to guide the AI, be as specific and descriptive as possible
  - `type`: Data type (`string`, `number`, `boolean`, `array`, `object`)

#### Content Extraction Options

- **CSS Selector** (`string`, optional): CSS selector to target specific elements on the page. This can greatly reduce the AI cost, by reducing the number of input tokens. It can also have a positive impact on accuracy. If provided, only text from elements matching this selector will be extracted. If not provided, the default content extraction will be used. This is an advanced option, if you are not familiar with CSS selectors, please do not provide one. Inspect the HTML of a page to find the correct CSS selector.

Example CSS selectors:

- `main`: selects elements with tag "main".
- `#price`: selects elements with id "price".
- `.product-details-container .price, .product-details-container .description`: selects elements with class "price" and "description" that are descendants of elements with class "product-details-container".
- `article.main-story, .article-body > p`: selects elements with tag "article" and class "main-story", as well as direct child "p" elements under elements with class "article-body".
- `.documentation-content h2, .documentation-content .method-signature`: selects "h2" elements and elements with class "method-signature" that are descendants of elements with class "documentation-content".
- `.post-container[data-type="user-post"] .content`: selects elements with class "content" that are descendants of elements with both class "post-container" and data-type attribute "user-post".
- `#product-listing div.item:not(.ad) .details h3, .price-info span.current-price`: selects "h3" elements under elements with class "details" that are descendants of "div" elements with class "item" but not class "ad" under element with ID "product-listing", as well as "span" elements with class "current-price" under elements with class "price-info".

#### AI Model Configuration

You can either use one of our predfined models which we verified that work well. Or you could specify your own model from [OpenRouter](https://openrouter.ai/models?supported_parameters=structured_outputs). If you use a predefined model, you don't have to bring your own API key we will cover the AI cost and you will be charged for it through Apify usage. If you bring your own OpenRouter API key you will not be charged for the AI cost. Your API key is stored securly and encrypted with Apify.

After some testing we found Google Gemini Flash 2.0 to give the best quality for the lowest price.

Free Apify users can only process 1 URL every 24 hours using predefined models to test out this functionality. If you are a free user you will have to [upgrade your Apify account](https://apify.com/pricing?fpr=7p4wu) to a paying subscription tier to use predefined models or bring your own OpenRouter API key.

##### Option 1: Predefined Models

- **Predefined Model** (`string`): Choose from supported models:
  - Google Gemini Flash 1.5
  - Google Gemini Flash 2.0 (recommended)
  - OpenAI GPT-4o-mini
  - Google Gemini Pro 1.5
  - OpenAI GPT-4o

##### Option 2: Custom Model

- **Use Custom Model** (`boolean`): Toggle to use your own model
- **Custom Model Name** (`string`): OpenRouter model identifier e.g. google/gemini-2.0-flash-001
- **OpenRouter API Key** (`string`): Your API key for custom model access (is stored encrypted)

Make sure your model supports structured outputs. Check model compatibility at: https://openrouter.ai/models?supported_parameters=structured_outputs

#### Proxies

- **Proxy Configuration** (`object`): Configure proxy settings for web scraping

#### Example input

```json
{
    "urls": [
        "https://apify.com/clockworks/free-tiktok-scraper"
    ],
    "fields": [
        {
            "name": "name",
            "description": "The name/title of the scraper tool",
            "type": "string"
        },
        {
            "name": "price",
            "description": "The price per 1000 results, only the number",
            "type": "number"
        },
        {
            "name": "author",
            "description": "The author or maintainer of the scraper",
            "type": "string"
        }
    ],
    "cssSelector": "main > article",
    "useCustomModel": false,
    "predefinedModel": "google/gemini-2.0-flash-001",
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    }
}
```

### Output

The actor outputs a dataset where each item contains:

- `url`: The source URL
- Custom fields as specified in your input configuration

Example output:

```json
{
    "url": "https://apify.com/clockworks/free-tiktok-scraper",
    "author": "Clockworks",
    "name": "TikTok Data Extractor",
    "price": 4
}
```

### Cost

There are 3 costs to using this model: startup cost, cost per result and AI cost. We split it up like this to make our pricing as competitive as possible.

- There's a one time charge of $0.05 (5 cents) every time you start an actor run. This cost is to cover server startup times.
- Every result pushed to the dataset (= every input URL) is charged at $0.001 (1/10th of a cent).
- If you use a predefined model you will be charged for every 1,000 tokens depending on the AI model used. If you bring your own API key you will not be charged this.
  - Google Gemini Flash 1.5: $0.0006 / 1,000 tokens (6/100th of a cent)
  - Google Gemini Flash 2.0: $0.0008 / 1,000 tokens (8/100th of a cent, best value)
  - Open AI GPT-4o mini: $0.0012 / 1,000 tokens (12/100th of a cent)
  - Google Gemini Pro 1.5: $0.02 / 1,000 tokens (2 cents)
  - Open AI GPT4o: $0.04 / 1,000 tokens (4 cents)

You can check how many tokens are in a given text by using the Open AI Tokenizer: https://platform.openai.com/tokenizer. Generally speaking 1 token = 1 word.

### Limitations

- The AI models require clear, well-structured content for best results
- Some models may have token limits affecting the amount of text they can process
- Custom models must support structured output format
- Rate limits may apply based on the chosen AI provider

### Cost of Usage

- When using predefined models, AI costs are covered
- Custom model usage requires your own OpenRouter API key and credits
- Standard Apify platform charges apply (proxy usage if enabled)

### Tips for Best Results

1. Be specific in your field descriptions
2. Choose appropriate data types for each field
3. Test with a small number of URLs first
4. Use the model that best fits your needs (faster models for simple extraction, more powerful models for complex tasks)
5. Consider using proxies when scraping at scale
6. Use CSS selectors when you know exactly which elements contain the relevant information
7. Test your CSS selectors first in browser DevTools to ensure they match the desired elements

### Technical Details

- Built with TypeScript
- Uses Crawlee for web scraping
- Integrates with OpenRouter for AI processing
- Supports structured output with JSON schema validation
- Includes automatic error handling and retries
- Supports both default content extraction and CSS selector-based extraction

# Actor input Schema

## `urls` (type: `array`):

URLs to extract data from

## `fields` (type: `array`):

Fields to extract and their descriptions

## `cssSelector` (type: `string`):

Optional CSS selector to extract text from. If provided, only text from elements matching this selector will be extracted. If not provided, the default content extraction will be used.

## `predefinedModel` (type: `string`):

Select one of our supported AI models, we cover the AI costs for you

## `useCustomModel` (type: `boolean`):

Whether to use a custom model with your own API key

## `customModelName` (type: `string`):

The OpenRouter identifier of your model. Make sure your model supports structured outputs and all required parameters. Check model compatibility at: https://openrouter.ai/models?supported_parameters=structured_outputs

## `customApiKey` (type: `string`):

Your OpenRouter API key for using the custom model

## `proxyConfiguration` (type: `object`):

Select proxies to be used by the crawler

## Actor input object example

```json
{
  "urls": [
    "https://apify.com/louisdeconinck/ai-gpt-scraper"
  ],
  "fields": [
    {
      "name": "name",
      "description": "The name/title of the scraper tool",
      "type": "string"
    },
    {
      "name": "price",
      "description": "The price per 1000 results, only the number",
      "type": "number"
    },
    {
      "name": "author",
      "description": "The author or maintainer of the scraper",
      "type": "string"
    }
  ],
  "cssSelector": "main",
  "predefinedModel": "google/gemini-2.0-flash-001",
  "useCustomModel": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://apify.com/louisdeconinck/ai-gpt-scraper"
    ],
    "fields": [
        {
            "name": "name",
            "description": "The name/title of the scraper tool",
            "type": "string"
        },
        {
            "name": "price",
            "description": "The price per 1000 results, only the number",
            "type": "number"
        },
        {
            "name": "author",
            "description": "The author or maintainer of the scraper",
            "type": "string"
        }
    ],
    "cssSelector": "main",
    "predefinedModel": "google/gemini-2.0-flash-001",
    "useCustomModel": false,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("louisdeconinck/ai-gpt-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://apify.com/louisdeconinck/ai-gpt-scraper"],
    "fields": [
        {
            "name": "name",
            "description": "The name/title of the scraper tool",
            "type": "string",
        },
        {
            "name": "price",
            "description": "The price per 1000 results, only the number",
            "type": "number",
        },
        {
            "name": "author",
            "description": "The author or maintainer of the scraper",
            "type": "string",
        },
    ],
    "cssSelector": "main",
    "predefinedModel": "google/gemini-2.0-flash-001",
    "useCustomModel": False,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("louisdeconinck/ai-gpt-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://apify.com/louisdeconinck/ai-gpt-scraper"
  ],
  "fields": [
    {
      "name": "name",
      "description": "The name/title of the scraper tool",
      "type": "string"
    },
    {
      "name": "price",
      "description": "The price per 1000 results, only the number",
      "type": "number"
    },
    {
      "name": "author",
      "description": "The author or maintainer of the scraper",
      "type": "string"
    }
  ],
  "cssSelector": "main",
  "predefinedModel": "google/gemini-2.0-flash-001",
  "useCustomModel": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call louisdeconinck/ai-gpt-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,louisdeconinck/ai-gpt-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/cilbtHPS4zwTX5hv3/builds/sfhtl0NDOZDKYZMUU/openapi.json