AI Web Scraper - scrape any website without selectors or code

AI Web Scraper turns web pages into clean, structured data. Name the columns you want, or describe them in plain English, and it reads each page and fills them in. No CSS selectors, no XPath, no code, and nothing to rewrite when a site changes its design. Export to JSON, CSV, Excel or XML, or pull the results from the Apify API.

Point it at one URL or a thousand. It works on pages you have never seen, on sites that share no common markup, and on lists of URLs that all look different.

What is AI Web Scraper?

It is a no-code web scraper that uses AI to find your data on the page instead of asking you to describe where it sits. A traditional scraper needs a selector for every field and breaks when the site is redesigned. This one needs the name of the field.

Give it:

• URLs to scrape, and optionally a crawl depth so it follows links itself
• Fields, one per line, such as product_name, price, in_stock
• Instructions in plain English, when the field names need context

Get back one row per page, or one row per item when the page is a listing, with your columns first and the source of each row beside them.

How do I scrape a website without code?

1. Paste your URLs.
2. Type the columns you want. Add a colon and a short hint where a name alone is ambiguous: price: the current price as a number, without the currency symbol.
3. Set a page budget, and a crawl depth if you want links followed.
4. Run it and download the dataset as JSON, CSV or Excel.

There is no step where you inspect the page or write a selector.

What data can AI Web Scraper extract?

Anything visible on the page as text or as a link. Common uses:

You want	Fields you would name
Product data	product_name, price, currency, in_stock, rating, image_url
Job listings	job_title, company, location, salary_range, remote: true or false
Company directories	company_name, website, industry, employees, city
Articles and news	headline, author, published: date in YYYY-MM-DD, summary
Real estate	address, price, bedrooms, square_feet, listing_url
Reviews	reviewer, rating, review_text, date

A field the page does not answer comes back as null, never as a guess.

How do I write fields that give clean data?

The difference between a clean dataset and a messy one is usually four or five extra words.

Name the format you want.

• Good: price: current price as a number, no currency symbol
• Weak: price, which gets you "$129.99" on one page and 129.99 on the next

Disambiguate anything that appears twice.

• Good: author: the writer of the article, not the person quoted in it
• Weak: author

Pin down dates and true or false values.

• Good: published: date in YYYY-MM-DD, remote: true if the job can be done from home

Ask for a list when you want a list. Set a field's type to list and you get every match, such as every tag on a post, rather than the first one.

Keep one row about one thing. On a listing page the scraper writes one row per item by itself. Name the fields of a single item, not "all the product names".

What does the output look like?

{
  "product_name": "Ninja Pro 4-in-1 Air Fryer",
  "price": 129.99,
  "in_stock": true,
  "url": "https://example.com/products/ninja-pro-air-fryer",
  "pageTitle": "Ninja Pro 4-in-1 Air Fryer | Example Store",
  "statusCode": 200,
  "depth": 1,
  "extractedAt": "2026-09-20T14:31:07.881Z"
}

Your columns come first, in the order you typed them, on every row of every run. That matters when the data feeds a spreadsheet or a database table: next month's run still lines up with today's.

If you name a column that clashes with a source field, such as url, your column wins and the page's own address moves to pageUrl. Nothing is silently overwritten.

Can it crawl a whole website?

Yes. Set a crawl depth above 0 and it follows links from your start URLs.

• Depth 0 scrapes only the URLs you listed
• Depth 1 also scrapes what they link to, and so on
• Stay on the same site is on by default, so one stray link cannot spend the budget elsewhere
• Only follow URLs matching and Never follow URLs matching take wildcards such as https://example.com/blog/* or */tag/*
• Maximum pages is a hard cap for the whole run

Links are followed breadth first, so the pages nearest your start URLs are scraped first.

Which websites does it work on?

Most of them, including sites that turn away ordinary scrapers and sites that build their pages in the browser. Pages that resist are handled with more effort automatically, and each row records whether it took the standard path or the harder one, so you can see what a site is costing you.

Set Effort per page to Fast when a site scrapes easily and you want the lowest cost, or to Thorough when you know it will fight. Auto is the default and steps up only for the pages that need it.

Pages behind a login are out of scope, and a site that shows a full image puzzle rather than a simple check will not be worked through.

How much does AI Web Scraper cost?

You pay per page, plus Apify platform usage, so Maximum pages is the dial that controls your bill. The run stops the moment it is reached.

Large runs are cheaper per page than small ones. The scraper works out a site's structure the first time it sees it and reuses that across the rest of the site, so a thousand pages on one site costs far less than a thousand pages on a thousand different sites. When a page does not match what it worked out, that page is read in full and the structure is worked out again: an unusual page costs more, but it never comes back empty.

Characters read per page also moves the cost. Lower it when your data sits near the top of a page, raise it for long documents. The default is 12,000 characters, about 2,000 words, and the ceiling is 60,000.

Can I call it as an API instead of starting a run?

Yes. The Actor stays warm and answers HTTP requests, so a single page does not need a run. Send your Apify API token as a bearer token.

curl "https://automly--ai-web-scraper.apify.actor/extract?url=https://quotes.toscrape.com/&fields=quote,author" \
  -H "Authorization: Bearer <YOUR_APIFY_API_TOKEN>"

Several pages at once, with a typed column:

curl -X POST "https://automly--ai-web-scraper.apify.actor/extract" \
  -H "Authorization: Bearer <YOUR_APIFY_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://quotes.toscrape.com/", "https://quotes.toscrape.com/page/2/"],
       "fields": ["quote: the quote text", "author", {"name": "tags", "type": "array"}]}'

What a site taught it stays available between requests, so the first call to a site is the slow one and the rest come back in under a second. Every response carries modelCalls, so you can watch that number stop rising. The Endpoints tab on this page lists the same endpoints and can send requests from your browser.

Input options

Option	What it does
startUrls	Pages to scrape, and where a crawl starts
fields	The columns to extract, one per line, with optional hints
instructions	Plain-English description of what to pull out
maxPages	Hard budget for the run
maxCrawlDepth	How far to follow links, 0 for no crawling
stayOnSite	Keep the crawl on the starting site
includeUrlPatterns, excludeUrlPatterns	Wildcard filters for links
extractionMode	Reuse a site's structure, or read every page in full
effort	How hard to work on pages that resist
solveChallenges	Work through anti-bot checks
maxCharactersPerPage	How much of each page is read
includePageContent	Add the page text to each row
concurrency	Pages worked on at once
proxyConfiguration	Optional proxy settings

AI Web Scraper compared with a traditional scraper

	Traditional scraper	AI Web Scraper
Setup	A selector per field, per site	Name the fields
A new site	A new scraper	Same input, new URL
Site redesign	Breaks, needs fixing	Works out the new structure
Skills needed	CSS or XPath, usually code	None
Listing pages	Extra work	One row per item automatically

What can you build with it?

• Price and catalogue monitoring across stores that share no common markup
• Lead lists from directories: name, role, company, location, contact page
• Job market datasets normalised across boards that each format things differently
• Competitor tracking on pricing pages, feature tables and changelogs, diffed week over week
• Research and news datasets from sources with nothing in common but the topic
• RAG pipelines, using the page text alongside the extracted fields

Using AI Web Scraper with the Apify API

curl -X POST "https://api.apify.com/v2/acts/automly~ai-web-scraper/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://example.com/products"}],
    "fields": ["product_name", "price: current price as a number", "in_stock: true or false"],
    "maxPages": 25,
    "maxCrawlDepth": 1
  }'

It also works with the Apify Python and JavaScript clients, scheduled runs, webhooks, and integrations such as Zapier, Make, n8n, LangChain, Google Sheets and Slack.

What does a run leave behind?

Two things in the run's storage, each on its own tab:

• Run report: pages scraped, how many needed extra work, how many anti-bot checks were cleared, and every page that failed with its reason
• Learned page layouts, one record per site, showing how each of your columns was located. Read these when a column comes back empty, to tell a miss apart from a page that genuinely lacks the field

Is it legal to scrape websites with AI?

Scraping publicly available pages is broadly lawful in the US and the EU, and this Actor reads only what any visitor can see. It does not log in and does not bypass authentication. You are responsible for what you collect: check the site's terms, and take care with personal data, which is regulated under the GDPR and similar laws. This is not legal advice.

Frequently asked questions

Do I need to write CSS selectors or XPath? No. That is the point of this Actor. You name the fields and it finds them.

Do I need my own AI API key or a ChatGPT subscription? No. Everything is included in the price per page.

Will my columns stay the same between runs? Yes. They are the field names you typed, in your order, on every row. Tools that let the AI invent its own keys hand you a different spreadsheet each time.

Can it extract a list of products from one page? Yes. When a page lists many things, it writes one row per item rather than cramming the page into a single row. Nothing to configure.

What happens when a page does not have one of my fields? That cell is null. Values come from the page, never from the model's imagination.

Does it work on sites that block scrapers? Usually. Pages that resist are retried with more effort, and you can set that effort per run. Sites behind a login, or behind a full image puzzle, are out of scope.

How accurate is it? Accurate on facts plainly on the page: titles, prices, dates, names, links. Weaker on judgement calls and on values that exist only inside images. Switch on the page text in the output and spot-check your first run.

Why did a page come back empty? Either the site refused every attempt, or the page genuinely has none of your fields. Failed pages are listed in the log and in the run report with a reason for each.

Can it scrape pages in other languages? Yes. Pages are read in the language they are written in, and you can ask in the instructions for values to come back in another language.

How do I keep the cost down on a big crawl? Set a firm page budget, narrow the crawl with URL patterns, keep effort on Auto, and lower the characters read per page when your data sits near the top.

Can I run it on a schedule? Yes. Schedule the Actor and each run appends to a dataset you can diff against the last one.

How is this different from a website content crawler? A content crawler hands you the page. This hands you the fields, already named, typed and ready for a spreadsheet.

Support

Something not extracting the way you expect? Open an issue on the Issues tab with the URL and the fields you asked for.