Scrape GPT - Universal AI Web Scraper Agent

Pricing

from $3.00 / 1,000 results

Try for free

Go to Apify Store

Scrape GPT - Universal AI Web Scraper Agent

Try for free

AI-powered universal web scraper that works on ANY website without configuration. Extract data from e-commerce, news sites, social media, and more using intelligent LLM-based field mapping. Features JSON-first extraction, automatic pagination, anti-bot bypass, and cost-effective caching.

Pricing

from $3.00 / 1,000 results

Rating

5.0

(1)

Developer

Paradox Analytics

Maintained by Community

Actor stats

Bookmarked

Total users

Monthly active users

2 hours ago

Last modified

Categories

Agents

Automation

You can access the Scrape GPT - Universal AI Web Scraper Agent programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "2.0",
    "x-build-id": "F8WUMcGKrfpfiEkbd"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/paradox-analytics~scrape-gpt---universal-ai-web-scraper-agent/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-paradox-analytics-scrape-gpt---universal-ai-web-scraper-agent",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/paradox-analytics~scrape-gpt---universal-ai-web-scraper-agent/runs": {
      "post": {
        "operationId": "runs-sync-paradox-analytics-scrape-gpt---universal-ai-web-scraper-agent",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/paradox-analytics~scrape-gpt---universal-ai-web-scraper-agent/run-sync": {
      "post": {
        "operationId": "run-sync-paradox-analytics-scrape-gpt---universal-ai-web-scraper-agent",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "required": [
          "startUrls",
          "fields"
        ],
        "properties": {
          "startUrls": {
            "title": "URLs to Scrape",
            "type": "array",
            "description": "List of URLs to scrape. The system will automatically detect patterns and extract data.",
            "items": {
              "type": "object",
              "required": [
                "url"
              ],
              "properties": {
                "url": {
                  "type": "string",
                  "title": "URL of a web page",
                  "format": "uri"
                }
              }
            }
          },
          "fields": {
            "title": "Fields to Extract",
            "type": "array",
            "description": "List of fields to extract (e.g., title, price, description). The LLM will generate semantic extraction patterns for these fields.",
            "items": {
              "type": "string"
            }
          },
          "openaiApiKey": {
            "title": "🔑 OpenAI API Key",
            "type": "string",
            "description": "Your OpenAI API key (required for pattern generation). Cost: ~$0.02 per new domain, then $0.0001 for cached patterns. You can also set OPENAI_API_KEY as an environment variable."
          },
          "useDirectLLM": {
            "title": "🤖 Use Direct LLM Extraction",
            "type": "boolean",
            "description": "Enable Direct LLM extraction. Extracts data directly without pattern generation. Recommended for maximum accuracy!",
            "default": true
          },
          "directLLMQualityMode": {
            "title": "Direct LLM Quality Mode",
            "enum": [
              "conservative",
              "balanced",
              "aggressive"
            ],
            "type": "string",
            "description": "Quality mode for DirectLLM: 'conservative' (50% fields required, highest quality), 'balanced' (33% fields, default), 'aggressive' (10% fields, maximum items)",
            "default": "balanced"
          },
          "hybridConfig": {
            "title": "⚙️ Hybrid System Configuration (Advanced)",
            "type": "object",
            "description": "Advanced configuration for the hybrid system",
            "properties": {
              "similarityThreshold": {
                "title": "Pattern Similarity Threshold",
                "type": "number",
                "description": "Threshold for pattern reuse (0.0-1.0). Higher = more strict matching. Recommended: 0.75",
                "default": 0.75,
                "minimum": 0,
                "maximum": 1
              },
              "enableLLM": {
                "title": "Enable LLM Pattern Generation",
                "type": "boolean",
                "description": "Use LLM to generate semantic patterns. If false, uses fallback patterns (free but less accurate).",
                "default": true
              },
              "cachePatterns": {
                "title": "Cache Generated Patterns",
                "type": "boolean",
                "description": "Save patterns to cache for future reuse. Highly recommended for cost savings!",
                "default": true
              },
              "maxContainers": {
                "title": "Max Containers per Page",
                "type": "integer",
                "description": "Maximum number of repeating items to extract per page",
                "default": 20,
                "minimum": 1,
                "maximum": 100
              }
            }
          },
          "proxyConfiguration": {
            "title": "🔗 Apify Proxy Configuration",
            "type": "object",
            "description": "Configure Apify residential proxies (recommended for production). Leave empty to use external proxy or no proxy."
          },
          "useExternalProxy": {
            "title": "🌐 Use External Proxy (Bright Data, etc.)",
            "type": "boolean",
            "description": "Enable external proxy instead of Apify proxy. Requires externalProxyServer, externalProxyUsername, and externalProxyPassword.",
            "default": false
          },
          "externalProxyServer": {
            "title": "External Proxy Server",
            "type": "string",
            "description": "External proxy server URL (e.g., http://brd.superproxy.io:33335). Required if useExternalProxy is true."
          },
          "externalProxyUsername": {
            "title": "External Proxy Username",
            "type": "string",
            "description": "External proxy username. Required if useExternalProxy is true."
          },
          "externalProxyPassword": {
            "title": "External Proxy Password",
            "type": "string",
            "description": "External proxy password. Required if useExternalProxy is true."
          },
          "webUnblockerApiKey": {
            "title": "🌐 Web Unblocker API Key",
            "type": "string",
            "description": "Web Unblocker API key (e.g., Bright Data Web Unblocker) for bypassing Kasada/Cloudflare. Used as fallback when standard proxies are blocked. You can also set BRIGHT_DATA_API_KEY as an environment variable."
          },
          "webUnblockerZone": {
            "title": "Web Unblocker Zone",
            "type": "string",
            "description": "Bright Data Web Unblocker zone name (default: web_unlocker1)",
            "default": "web_unlocker1"
          },
          "enableAutoPagination": {
            "title": "📄 Enable Auto-Pagination",
            "type": "boolean",
            "description": "Automatically scrape pages when pagination is detected. WARNING: Some sites have thousands of pages!",
            "default": false
          },
          "maxPages": {
            "title": "Max Pages to Scrape",
            "minimum": 0,
            "maximum": 10000,
            "type": "integer",
            "description": "Maximum number of pages to scrape when auto-pagination is enabled. Leave blank or set to 0 to scrape all pages. Only applies when enableAutoPagination is true.",
            "default": 0
          },
          "headless": {
            "title": "Headless Browser",
            "type": "boolean",
            "description": "Run browser in headless mode (recommended)",
            "default": true
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Scrape GPT - Universal AI Web Scraper Agent OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for Scrape GPT - Universal AI Web Scraper Agent from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

Scrape GPT - Universal AI Web Scraper Agent API in Python

Scrape GPT - Universal AI Web Scraper Agent API in JavaScript

Scrape GPT - Universal AI Web Scraper Agent API through CLI

Scrape GPT - Universal AI Web Scraper Agent API

Universal Web Extractor V8

motivational_nickel/my-actor

Flexible web extractor using Python + Playwright or HTTP. Supports CSS-based field extraction, HTML snapshots, screenshots, metadata, monitoring mode, and link-following. Ideal for scraping product pages, listings, news articles, tech profiles, or universal structured data from any website.

Leoncio Jr Coronado

5.0

Universal Contact Extractor

rl1987/universal-contact-extractor

Extract phone numbers, emails and social media URLs from web pages.

R.L.

Universal AI GPT Scraper

louisdeconinck/ai-gpt-scraper

Transform any website into structured data with AI-powered extraction. This versatile tool combines advanced web scraping with intelligent content analysis to deliver clean, customized JSON output - perfect for automating data collection from any web source.

Louis Deconinck

145

5.0

Intelligent Website Scrapper

happitap/intelligent-website-scrapper

An intelligent website scraper that uses LangChain and LLM to extract and process content based on high-level goals like summarization, product extraction, service extraction, and FAQ extraction.

HappiTap

127

5.0

Universal Website Scraper

jowi/universal-website-scraper

The Universal Website Scraper is a powerful all-purpose crawler that extracts structured data from ANY website, including dynamic pages, product listings, articles, tables, blogs, and more. No coding needed just enter a URL and let the actor do the rest.

Fred

Extract-any-webpage-content-for-llm

ai-developer/extract-any-webpage-content-for-llm

Fast and easy way to extract data from any webpage and are LLM friendly. The tool lets you easily extract content from any website. Ideal for researchers, marketers, and developers.

aideveloper

575

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

210

5.0

Website Contact Scraper - AI-Powered Lead Finder

timo.sieber/website-lead-scraper

AI-powered website scraper that extracts real contact data from company sites! Finds people, positions, emails & phone numbers using LLM technology. Scans team pages, contact sections & company info. Perfect for B2B lead generation and sales research.

Timo Sieber

Universal Website to API – Any Site → JSON

inquisitive_zeppelin/universal-website-to-api---any-site---json

Multi-URL Web Scraper is a fast, reliable, and highly flexible scraping tool designed to extract structured data from multiple web pages in a single run.

Hamza Ahmed

Universal Article Scraper

universal_scraping/universal-article-scraper

Universal article scraper for news websites, blogs, etc. It can scrape articles from multiple websites simultaneously, including metadata such as title, content, publication date, image, and author.