Pricing

from $1.00 / 1,000 results

Universal Web Scraper API

Scrape any URL through a global, anti-bot-resistant Scraper API. JS rendering, geo-targeting, residential proxies, CSS-selector extraction, and optional LLM processing.

Pricing

from $1.00 / 1,000 results

Rating

0.0

(0)

Developer

youssef farhan

Actor stats

Bookmarked

Total users

Monthly active users

10 hours ago

Last modified

Universal Web Scraper API — Extract Any URL With Anti-Bot Proxies + AI

A universal web scraper API that turns any URL into clean HTML, plain text, and structured JSON, using global anti-bot proxies in 46 countries. Built for developers, researchers, growth teams, and AI agents that need reliable web data without managing proxies or browsers. Add CSS selectors or an LLM prompt and get structured output in one run.

What you get

Every URL produces one dataset record with these fields:

url — the page that was scraped.
statusCode — HTTP status returned by the target.
ok — true when the status was 2xx.
error — error message for failed URLs (null on success).
scrapedAt — ISO 8601 UTC timestamp of the scrape.
html — full raw HTML (toggle off with storeHtml).
text — readable text with scripts, styles, and markup stripped.
textLength — character count of text.
extractedFields — your CSS-selector fields as a key/value object.
llm — optional AI result: output (text or parsed JSON), model, usage.
options — echo of render, geoCode, and superProxy used per request.

Sample output

{
  "url": "https://quotes.toscrape.com",
  "statusCode": 200,
  "ok": true,
  "html": "<!DOCTYPE html><html>...</html>",
  "text": "Quotes to Scrape. The world as we have created it is a process of our thinking.",
  "textLength": 4213,
  "extractedFields": {
    "title": "Quotes to Scrape",
    "firstQuote": "The world as we have created it is a process of our thinking.",
    "firstAuthor": "Albert Einstein"
  },
  "llm": {
    "model": "google/gemini-3.1-flash-lite",
    "output": { "topic": "inspirational quotes", "itemCount": 10 },
    "usage": { "prompt_tokens": 1820, "completion_tokens": 24, "total_tokens": 1844 }
  },
  "options": { "render": true, "geoCode": "us", "superProxy": false },
  "scrapedAt": "2026-06-25T12:00:00.000000+00:00",
  "error": null
}

Use cases

✓ Data engineers feeding clean page text and JSON into RAG and LLM pipelines.
✓ Price researchers monitoring competitor product pages across multiple countries.
✓ Growth and lead-gen teams pulling structured data from directories at scale.
✓ AI agent builders calling this web scraper API over MCP for live web data.
✓ SEO and content teams auditing pages behind anti-bot protection.
✓ Analysts extracting named fields from any site with simple CSS selectors.

Pricing

This Actor is pay-per-result — you only pay for records pushed to the dataset.

Event	Price per 1,000	Real example
Scraped result (one URL)	`[PLACEHOLDER: $X.XX per 1,000 results]`	Scraping 10,000 URLs ≈ `[PLACEHOLDER: $XX]`

Free tier: your first [PLACEHOLDER: N] results are free — no credit card needed to try it. You also use your own Scraper API key, and you can get 1,000 free monthly scraping credits here to cover the underlying requests.

How it works

Input: a list of startUrls plus your Scraper API token.
Fetch: each URL is routed through rotating anti-bot proxies, with optional JavaScript rendering and geo-targeting.
Process: apply CSS extractRules and/or an LLM prompt to each page.
Output: structured records land in the Apify dataset (export as JSON, CSV, Excel, or XML).
Automate: run on a schedule and trigger webhooks on finish — built into the Apify platform.

Why this web scraper API

Use your own key: pay the scraping provider directly, with 1,000 free monthly credits to start.
Global anti-bot coverage: residential/premium proxies and geo-targeting across 46 countries.
Built-in AI: summarize, classify, or extract JSON via OpenRouter — pay with Apify credits or your own OpenRouter key.
Resumable, migration-safe runs: state is persisted automatically, so a migrated run resumes without re-scraping or duplicating data.
Auto-scaled concurrency: parallelism is sized to a percentage of your live plan limit, so you never trip rate limits.

Input example

{
  "apiKey": "YOUR_API_TOKEN",
  "startUrls": [{ "url": "https://quotes.toscrape.com" }],
  "render": true,
  "geoCode": "us",
  "extractRules": { "title": "h1", "firstQuote": ".quote .text" }
}

Omit extractRules and llm* fields to just return clean HTML and text. Set concurrencyPercentage to control speed.

FAQ

Does it handle anti-bot protection and proxies? Yes. Every request is routed through a rotating proxy network, with optional residential/premium proxies and per-country geo-targeting.

What output formats are supported? The dataset exports to JSON, CSV, Excel, and XML, or via the Apify API.

Can I scrape one URL or many? Both. Pass a single URL or thousands in startUrls; they are scraped concurrently.

Does it support scheduling and webhooks? Yes — use Apify schedules to run it automatically and webhooks to push results downstream.

Is the data live or cached? Live. Each run fetches the current page in real time.

How do I extract specific fields? Provide extractRules as a { "fieldName": "css selector" } map, or set an llmPrompt for AI extraction.

Can AI agents call it? Yes — it's available via the Apify REST API and as an MCP server for Claude, ChatGPT, and Cursor (see below).

Can I request custom fields or a different site? Yes — see the custom scraper section at the bottom.

Use via API or MCP

Run it programmatically via the Apify REST API:

POST https://api.apify.com/v2/acts/fayoussef~universal-scraper-api/runs?token=YOUR_TOKEN

Or connect it as an MCP server so AI agents can call it directly:

https://mcp.apify.com/actors/fayoussef~universal-scraper-api

Need a custom scraper?

Need different fields, a specific site, or a fully managed pipeline? Visit automationbyexperts.

PLACEHOLDERS TO FILL IN:
- Pay-per-result price per 1,000 results
- Real-run cost example (e.g. 10,000 URLs ≈ $XX)
- Free-tier amount (number of free results)
- Confirm custom-scraper CTA URL (automationbyexperts.com)

Universal Content Extractor

red.cars/universal-content-extractor

Enterprise-grade media extraction platform. Download high-quality videos and metadata from 30+ major social media platforms with 99.9% uptime. Optimized for professional workflows.

AutomateLab

159

3.9

(2)

Universal Web Scraper & Data Extractor – Fast No-Code Tool

motivational_nickel/my-actor

Universal web scraper that extracts structured data from almost any website. Detect and scrape webpage content into clean datasets (CSV, Excel, JSON) without coding. Ideal for web scraping, research, lead generation, automation pipelines, and large-scale data extraction.

Leoncio Jr Coronado

5.0

(1)

Scrape GPT - Universal AI Web Scraper Agent

paradox-analytics/scrape-gpt---universal-ai-web-scraper-agent

AI-powered universal web scraper that works on ANY website without configuration. Extract data from e-commerce, news sites, social media, and more using intelligent LLM-based field mapping. Features JSON-first extraction, automatic pagination, anti-bot bypass, and cost-effective caching.

Paradox Analytics

Universal Web Scraper - Extract Any URL

lazymac/web-scraper-toolkit

Pay-per-result web scraper with JS rendering, CSS selector / XPath / regex extraction, schema validation, retry on failure. Use for product catalogs, competitor pricing, news aggregation, lead generation. Fast (<2s/page), respects robots.txt by default.

2x lazymac

Universal Web Scraper with Playwright

sepiropht/my-actor

Powerful web scraper using Playwright to extract data from any website. Define custom CSS selectors, handle JavaScript-rendered pages, support pagination, and collect multiple data fields per page. Perfect for price monitoring, news extraction, and lead generation.

William Mbotta

universal-structured-web-extractor-base

pierremrd29/universal-structured-web-extractor-base

Autonomous web change detection Actor for AI agents. Monitors URLs, detects meaningful content changes, computes diffs, and outputs structured signals for automation, alerts, and decision-making workflows.

Pierre Morandeau

Universal Web Scraper — Custom Data Extraction Starter Template

apricot_blackberry/my-actor

Universal Web Scraper — Custom Data Extraction Starter Template helps teams get quick, high-signal results with reliable output, clear fields, and fast setup.

Creator Fusion

Universal AI Web Scraper

stanvanrooy6/universal-ai-web-scraper

Turn any website into an API. Extract structured data using plain English. Features anti-bot bypass, dynamic rendering, and web search. No coding needed.

Stan Van Rooy

1.5

(2)

Universal RAG Web Scraper

express_kingfisher/rag-web-scraper

Turn any website into clean, LLM-ready Markdown. Automatically strips ads, navigation, and noise using Mozilla Readability. Perfect for feeding data to ChatGPT, Claude, or Vector Databases (RAG).

Prince Raj

Universal Web Snapshot - HTML, Text, Markdown Capture

fetchcraft/universal-web-snapshot

Capture a clean HTML, plain-text, or markdown snapshot of any URL. Built for archival, change detection, and downstream LLM input. Stores rendered title, final URL, status, fetched-at timestamp. $0.003 per snapshot. Free preview run.

Emily Ward

OmniStruct Cost-Optimized Scraper

subhaajit/Apify-scraper

Need clean, structured data from a website without burning through your Apify compute credits? OmniStruct Cost-Optimized Scraper is built from the ground up to be the most efficient, budget-friendly universal scraper on the platform.