AI Web Scraper — Extract Structured Data From Any URL

Extract structured data from web pages with an LLM and your own field schema—without writing CSS selectors or maintaining site-specific extraction code. Give this AI web scraper URLs and field descriptions; receive clean JSON records for articles, products, jobs, directories, listings, research, and RAG pipelines.

Export results, run via API, schedule and monitor runs, or integrate with other tools and AI agents.

---

How it works

1. You provide one or more URLs and a list of fields (name + short description).
2. The actor fetches each page, converts it to clean text, and asks an LLM to return JSON matching your fields.
3. You get one row per record (or one row per repeating item in list mode).

Because extraction follows semantic field instructions instead of fixed selectors, it can tolerate many layout changes. Results still depend on the text the actor can fetch and the selected model's interpretation.

What can the AI web scraper extract?

Define string, number, boolean, object, or array-like fields for the page content you need. Common extraction targets include:

• Product names, prices, availability, SKUs, and descriptions
• Job titles, companies, locations, salary text, and requirements
• Article titles, authors, dates, summaries, and topics
• Directory listings, business details, categories, and profile URLs
• Research facts, tables represented in page text, and source metadata
• Repeating search-result or catalog cards with listMode

---

Input

Field	Type	Description
startUrls	array	The page URLs to extract from.
fields	array	What to extract — [{ "name": "title", "description": "the product title", "type": "string" }].
listMode	boolean	ON = one row per repeating item on the page (grids, listings). OFF = one row per page.
maxItems	integer	Cap on total output rows.
maxCrawlPages	integer	Cap on pages fetched.
maxContentChars	integer	How much page text to send to the model (cost control).
proxyConfiguration	object	Apify proxy settings (datacenter by default).

Example input

{
  "startUrls": [{ "url": "https://quotes.toscrape.com" }],
  "fields": [
    { "name": "text", "description": "the full quote text" },
    { "name": "author", "description": "who said it" },
    { "name": "tags", "description": "list of tag labels", "type": "array" }
  ],
  "listMode": true
}

---

API key (required)

Extraction runs through OpenRouter — set a single environment variable on the actor (Console → Settings → Environment variables):

OPENROUTER_API_KEY = sk-or-...

The extraction model is managed internally for a predictable public input surface. You pay OpenRouter directly for model usage; the actor's PPE events cover the extraction layer.

---

Output

Every row contains source_url, scraped_at, error, plus your fields:

{
  "text": "The world as we have created it is a process of our thinking.",
  "author": "Albert Einstein",
  "tags": ["change", "deep-thoughts", "thinking", "world"],
  "source_url": "https://quotes.toscrape.com",
  "scraped_at": "2026-06-07T12:00:00.000Z",
  "error": null
}

---

Pricing (Pay Per Event)

Event	When
actor-start	Once per run.
page-processed	Each page successfully fetched and extracted (one LLM call).

Failed pages (fetch error, model error, missing key) are not charged.

The maximum number of chargeable page events is controlled by maxCrawlPages; maxItems limits the total rows returned. OpenRouter usage is billed separately through your own account. Review the live Store pricing panel for current Apify event prices.

Run the AI web scraper through the API

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('USERNAME/ai-web-extractor').call({
  startUrls: [{ url: 'https://quotes.toscrape.com' }],
  fields: [
    { name: 'text', description: 'Full quote text', type: 'string' },
    { name: 'author', description: 'Name of the quote author', type: 'string' },
    { name: 'tags', description: 'Tag labels', type: 'array' },
  ],
  listMode: true,
  maxItems: 100,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Use the Apify API, schedules, webhooks, dataset exports, or Apify MCP to connect the records to a spreadsheet, database, RAG workflow, or AI agent.

---

Use cases

• RAG / AI pipelines — turn arbitrary pages into clean structured records.
• Long-tail sites — scrape sites with no dedicated actor.
• Listings & directories — pull every item from a results page with listMode.
• Monitoring — schedule extraction of the same fields over time.

When to use—and when not to use—this actor

Use it for a small or medium collection of known URLs when you need flexible, schema-guided extraction and a dedicated site actor does not exist. It is particularly useful for prototypes, long-tail websites, changing page layouts, and agent workflows where the requested fields vary.

Do not use it for authenticated pages, actions such as form submission, pixel-perfect browser automation, or very large commodity crawls where deterministic selectors are cheaper. It fetches the URLs you provide; it is not a general search engine or unlimited site crawler. JavaScript-heavy or strongly protected pages may require a dedicated browser-based actor.

---

Tips

• Write clear field descriptions — they're the instructions the model follows.
• Use listMode for pages with many repeating records; turn it off for single detail pages.
• If a JavaScript-heavy page returns little text, a larger maxContentChars cannot recover content that was never present in the fetched response; use a dedicated browser actor instead.
• Use explicit field names such as productPriceUsd rather than vague names such as value.
• State desired formats in descriptions, for example “ISO 8601 date” or “number without currency symbol.”
• Start with one representative URL before scaling to a larger batch.

Output quality and limitations

LLM extraction is probabilistic. Validate required fields, spot-check high-value records, and inspect error before downstream use. Pages can contain misleading text or prompt-like content; treat all extracted content as untrusted data. The actor does not guarantee completeness, factual accuracy, or successful access to every website.

FAQ

Can this scrape any website?

It can process many publicly accessible pages, but “any URL” does not mean every page is reachable. Login walls, bot protection, JavaScript-only rendering, robots restrictions, and unsupported content can limit extraction.

What is list mode?

Enable listMode when one page contains repeated records such as product cards or search results. Disable it for a single article, product detail, company profile, or other one-record page.

Is this suitable for RAG data extraction?

Yes, when the RAG pipeline needs concise structured metadata or facts from known pages. Keep source URLs with every record and validate important fields before indexing.

Responsible scraping

Only process content you are authorized to access. Respect website terms, robots policies where applicable, copyright, privacy, and data-protection laws. Do not use the actor to bypass access controls or collect sensitive personal data. Keep OPENROUTER_API_KEY in a secret environment variable.