AI Universal Scraper — scrape any website with GPT or Claude, no selectors needed

An AI web scraper that turns any page into clean, structured JSON. Paste a URL, list the fields you want (title, price, author…), plug in your own OpenAI or Anthropic API key, and a language model reads the page like a human and returns exactly those fields. No CSS selectors to maintain, no per-site configuration, no broken parser every time a site changes its HTML — which is exactly where traditional scrapers fall over.

What it does

• Extracts structured data from any URL — you provide a list of pages and either a list of field names, a plain-English instruction, or both.
• Uses your own LLM key — supports OpenAI (default model gpt-4o-mini) and Anthropic (default model claude-haiku-4-5-20251001), with an optional model override so you can trade cost for quality.
• Cleans pages before sending them to the model — scripts, styles and markup are stripped so you only pay tokens for actual content, capped at a character limit you control (maxChars, default 12,000).
• Returns strict JSON — the model is instructed to answer with pure JSON, and the reply is parsed defensively (code fences and stray prose are handled).
• One row per item — a single-object answer becomes one dataset row per URL; when the page clearly lists multiple items (a news feed, a product listing), the model returns an array and you get one row per item, each tagged with the source url.
• Fails soft — an unreachable URL or a failed extraction is pushed as a row with an error field and the run continues with the remaining URLs.

Use cases

• Competitor product monitoring — pull name, price, availability from competitor product pages without writing a scraper per site.
• Content and news pipelines — turn any article or feed page into {title, summary, author} rows, ready for a newsletter digest or a research database.
• AI data extraction pipelines — use it as the "scrape any website with GPT" step in a larger workflow: feed the dataset into Zapier, Make, or your own code via the Apify API.
• Lead and directory extraction — extract names, roles and links from listing pages that have no API and no purpose-built scraper.
• One-off datasets — when a client asks for structured data from an obscure site, describe the fields instead of reverse-engineering the HTML.

Input

Field	Type	Default	Description
startUrls	array	— (required)	One or more page URLs to extract data from.
fields	array	—	The data points you want, e.g. "title", "price", "author", "rating". Leave empty if you use Instructions instead.
instructions	string	—	Natural-language guidance, e.g. "Extract every news item as a separate object with title and points."
llmProvider	string	openai	Which model provider to use: openai or anthropic.
apiKey	string	— (optional)	Your OpenAI or Anthropic API key. Stored encrypted; used only to call the model. Leave empty to run in no-LLM preview mode.
model	string	—	Override the model. Defaults: OpenAI → gpt-4o-mini, Anthropic → claude-haiku-4-5.
maxChars	integer	12000	How much page text to send to the model (controls cost). Minimum 1,000.

You must set startUrls; for full AI extraction also provide apiKey plus at least one of fields or instructions. The apiKey is optional — without it the Actor runs in a no-LLM preview mode that uses built-in heuristics to return each page's title, meta description, and a short text preview. That's handy for a free test drive, but the structured field extraction described below needs your OpenAI or Anthropic key.

{
  "startUrls": ["https://news.ycombinator.com"],
  "fields": ["title", "summary", "author"],
  "llmProvider": "openai",
  "apiKey": "sk-...",
  "maxChars": 12000
}

Where to get an API key

• OpenAI — create a key at platform.openai.com/api-keys.
• Anthropic — create a key at console.anthropic.com under API Keys.

The key is marked secret in the input schema, so Apify stores it encrypted and it never appears in logs or shared run links.

Output

Each dataset row is the JSON object the model extracted, plus the source url. The field names are the ones you asked for — the schema of the output is effectively yours to define. With the example input above, a row looks like:

{
  "url": "https://news.ycombinator.com",
  "title": "Show HN: I built a self-hosted analytics tool in a weekend",
  "summary": "A lightweight, privacy-first alternative to hosted analytics, written in Go.",
  "author": "jdoe42"
}

If a URL can't be fetched or the model's reply can't be parsed, you get a row like {"url": "https://…", "error": "extraction failed: …"} instead, so nothing disappears silently. Export everything as JSON, CSV, or Excel from the run's Dataset tab, or pull it via the Apify API.

Pricing

This actor uses Apify pay-per-event pricing charged per result — you pay only for the rows actually delivered to your dataset, never for setup or empty runs. See the actor's Pricing tab for the current per-result rate. A free Apify plan is enough to try it.

Separately, you pay your LLM provider directly for tokens on your own key. With the default gpt-4o-mini or claude-haiku-4-5 and the 12,000-character cap, a page typically costs a fraction of a cent; lower maxChars for cheaper runs or raise it for long pages.

Tips / FAQ

Does it work on JavaScript-heavy sites? It fetches the raw HTML over HTTP (no headless browser). Server-rendered pages — most articles, listings, product pages, docs — work well. A single-page app that renders everything client-side may return little usable text; check the row's content before scaling up.

Which provider and model should I pick? Extraction quality depends on the model you choose. The defaults (gpt-4o-mini, claude-haiku-4-5) are fast and cheap and handle most pages. For messy or ambiguous pages, set model to a stronger one (e.g. gpt-4o or a Claude Sonnet model) — same input, better judgment, higher token cost.

How do I get multiple rows from one page? Use instructions, e.g. "Extract every news item as a separate object with title and points." When the model returns an array, each object becomes its own dataset row.

How do I control cost? Only the first maxChars characters of cleaned page text are sent to the model (default 12,000). That cap, plus your choice of model, is your cost lever.

Does it follow links or crawl a whole site? No — it processes exactly the URLs you provide, one LLM call per page. For crawling, generate the URL list first (e.g. from a sitemap) and pass it in.

What happens on errors? Failed fetches and failed extractions are logged and pushed as {url, error} rows; the run continues with the next URL rather than aborting.

Related actors

• Smart Scraper by Example — show it one example of the data you want and it scrapes the rest of the site to match.
• AI Web Scraper — a sibling AI extraction actor for structured web data.
• Company & Domain Enricher — enrich the domains you scrape into full company records.

---

Found a bug or need a feature? Open an Issue on this actor's Issues tab — typical response within 1 business day.