# AI Web Scraper - Describe What You Want, Get Structured Data (`automly/ai-web-scraper`) Actor Scrape any website without writing selectors. Name the fields you want or describe them in plain English, and the Actor reads each page and returns clean structured data as JSON, CSV or Excel. - **URL**: https://apify.com/automly/ai-web-scraper.md - **Developed by:** [Automly](https://apify.com/automly) (community) - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $5.50 / 1,000 scraped results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## AI Web Scraper - scrape any website without selectors or code **AI Web Scraper** turns web pages into clean, structured data. Name the columns you want, or describe them in plain English, and it reads each page and fills them in. No CSS selectors, no XPath, no code, and nothing to rewrite when a site changes its design. Export to JSON, CSV, Excel or XML, or pull the results from the Apify API. Point it at one URL or a thousand. It works on pages you have never seen, on sites that share no common markup, and on lists of URLs that all look different. ### What is AI Web Scraper? It is a no-code web scraper that uses AI to find your data on the page instead of asking you to describe where it sits. A traditional scraper needs a selector for every field and breaks when the site is redesigned. This one needs the name of the field. Give it: - **URLs** to scrape, and optionally a crawl depth so it follows links itself - **Fields**, one per line, such as `product_name`, `price`, `in_stock` - **Instructions** in plain English, when the field names need context Get back one row per page, or one row per item when the page is a listing, with your columns first and the source of each row beside them. ### How do I scrape a website without code? 1. Paste your URLs. 2. Type the columns you want. Add a colon and a short hint where a name alone is ambiguous: `price: the current price as a number, without the currency symbol`. 3. Set a page budget, and a crawl depth if you want links followed. 4. Run it and download the dataset as JSON, CSV or Excel. There is no step where you inspect the page or write a selector. ### What data can AI Web Scraper extract? Anything visible on the page as text or as a link. Common uses: | You want | Fields you would name | |---|---| | Product data | `product_name`, `price`, `currency`, `in_stock`, `rating`, `image_url` | | Job listings | `job_title`, `company`, `location`, `salary_range`, `remote: true or false` | | Company directories | `company_name`, `website`, `industry`, `employees`, `city` | | Articles and news | `headline`, `author`, `published: date in YYYY-MM-DD`, `summary` | | Real estate | `address`, `price`, `bedrooms`, `square_feet`, `listing_url` | | Reviews | `reviewer`, `rating`, `review_text`, `date` | A field the page does not answer comes back as `null`, never as a guess. ### How do I write fields that give clean data? The difference between a clean dataset and a messy one is usually four or five extra words. **Name the format you want.** - Good: `price: current price as a number, no currency symbol` - Weak: `price`, which gets you `"$129.99"` on one page and `129.99` on the next **Disambiguate anything that appears twice.** - Good: `author: the writer of the article, not the person quoted in it` - Weak: `author` **Pin down dates and true or false values.** - Good: `published: date in YYYY-MM-DD`, `remote: true if the job can be done from home` **Ask for a list when you want a list.** Set a field's type to list and you get every match, such as every tag on a post, rather than the first one. **Keep one row about one thing.** On a listing page the scraper writes one row per item by itself. Name the fields of a single item, not "all the product names". ### What does the output look like? ```json { "product_name": "Ninja Pro 4-in-1 Air Fryer", "price": 129.99, "in_stock": true, "url": "https://example.com/products/ninja-pro-air-fryer", "pageTitle": "Ninja Pro 4-in-1 Air Fryer | Example Store", "statusCode": 200, "depth": 1, "extractedAt": "2026-09-20T14:31:07.881Z" } ``` Your columns come first, in the order you typed them, on every row of every run. That matters when the data feeds a spreadsheet or a database table: next month's run still lines up with today's. If you name a column that clashes with a source field, such as `url`, your column wins and the page's own address moves to `pageUrl`. Nothing is silently overwritten. ### Can it crawl a whole website? Yes. Set a crawl depth above 0 and it follows links from your start URLs. - **Depth 0** scrapes only the URLs you listed - **Depth 1** also scrapes what they link to, and so on - **Stay on the same site** is on by default, so one stray link cannot spend the budget elsewhere - **Only follow URLs matching** and **Never follow URLs matching** take wildcards such as `https://example.com/blog/*` or `*/tag/*` - **Maximum pages** is a hard cap for the whole run Links are followed breadth first, so the pages nearest your start URLs are scraped first. ### Which websites does it work on? Most of them, including sites that turn away ordinary scrapers and sites that build their pages in the browser. Pages that resist are handled with more effort automatically, and each row records whether it took the standard path or the harder one, so you can see what a site is costing you. Set **Effort per page** to Fast when a site scrapes easily and you want the lowest cost, or to Thorough when you know it will fight. Auto is the default and steps up only for the pages that need it. Pages behind a login are out of scope, and a site that shows a full image puzzle rather than a simple check will not be worked through. ### How much does AI Web Scraper cost? You pay per page, plus Apify platform usage, so **Maximum pages** is the dial that controls your bill. The run stops the moment it is reached. Large runs are cheaper per page than small ones. The scraper works out a site's structure the first time it sees it and reuses that across the rest of the site, so a thousand pages on one site costs far less than a thousand pages on a thousand different sites. When a page does not match what it worked out, that page is read in full and the structure is worked out again: an unusual page costs more, but it never comes back empty. **Characters read per page** also moves the cost. Lower it when your data sits near the top of a page, raise it for long documents. The default is 12,000 characters, about 2,000 words, and the ceiling is 60,000. ### Can I call it as an API instead of starting a run? Yes. The Actor stays warm and answers HTTP requests, so a single page does not need a run. Send your Apify API token as a bearer token. ```bash curl "https://automly--ai-web-scraper.apify.actor/extract?url=https://quotes.toscrape.com/&fields=quote,author" \ -H "Authorization: Bearer " ``` Several pages at once, with a typed column: ```bash curl -X POST "https://automly--ai-web-scraper.apify.actor/extract" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"urls": ["https://quotes.toscrape.com/", "https://quotes.toscrape.com/page/2/"], "fields": ["quote: the quote text", "author", {"name": "tags", "type": "array"}]}' ``` What a site taught it stays available between requests, so the first call to a site is the slow one and the rest come back in under a second. Every response carries `modelCalls`, so you can watch that number stop rising. The Endpoints tab on this page lists the same endpoints and can send requests from your browser. ### Input options | Option | What it does | |---|---| | `startUrls` | Pages to scrape, and where a crawl starts | | `fields` | The columns to extract, one per line, with optional hints | | `instructions` | Plain-English description of what to pull out | | `maxPages` | Hard budget for the run | | `maxCrawlDepth` | How far to follow links, 0 for no crawling | | `stayOnSite` | Keep the crawl on the starting site | | `includeUrlPatterns`, `excludeUrlPatterns` | Wildcard filters for links | | `extractionMode` | Reuse a site's structure, or read every page in full | | `effort` | How hard to work on pages that resist | | `solveChallenges` | Work through anti-bot checks | | `maxCharactersPerPage` | How much of each page is read | | `includePageContent` | Add the page text to each row | | `concurrency` | Pages worked on at once | | `proxyConfiguration` | Optional proxy settings | ### AI Web Scraper compared with a traditional scraper | | Traditional scraper | AI Web Scraper | |---|---|---| | Setup | A selector per field, per site | Name the fields | | A new site | A new scraper | Same input, new URL | | Site redesign | Breaks, needs fixing | Works out the new structure | | Skills needed | CSS or XPath, usually code | None | | Listing pages | Extra work | One row per item automatically | ### What can you build with it? - **Price and catalogue monitoring** across stores that share no common markup - **Lead lists** from directories: name, role, company, location, contact page - **Job market datasets** normalised across boards that each format things differently - **Competitor tracking** on pricing pages, feature tables and changelogs, diffed week over week - **Research and news datasets** from sources with nothing in common but the topic - **RAG pipelines**, using the page text alongside the extracted fields ### Using AI Web Scraper with the Apify API ```bash curl -X POST "https://api.apify.com/v2/acts/automly~ai-web-scraper/run-sync-get-dataset-items?token=" \ -H "Content-Type: application/json" \ -d '{ "startUrls": [{"url": "https://example.com/products"}], "fields": ["product_name", "price: current price as a number", "in_stock: true or false"], "maxPages": 25, "maxCrawlDepth": 1 }' ``` It also works with the Apify Python and JavaScript clients, scheduled runs, webhooks, and integrations such as Zapier, Make, n8n, LangChain, Google Sheets and Slack. ### What does a run leave behind? Two things in the run's storage, each on its own tab: - **Run report**: pages scraped, how many needed extra work, how many anti-bot checks were cleared, and every page that failed with its reason - **Learned page layouts**, one record per site, showing how each of your columns was located. Read these when a column comes back empty, to tell a miss apart from a page that genuinely lacks the field ### Is it legal to scrape websites with AI? Scraping publicly available pages is broadly lawful in the US and the EU, and this Actor reads only what any visitor can see. It does not log in and does not bypass authentication. You are responsible for what you collect: check the site's terms, and take care with personal data, which is regulated under the GDPR and similar laws. This is not legal advice. ### Frequently asked questions **Do I need to write CSS selectors or XPath?** No. That is the point of this Actor. You name the fields and it finds them. **Do I need my own AI API key or a ChatGPT subscription?** No. Everything is included in the price per page. **Will my columns stay the same between runs?** Yes. They are the field names you typed, in your order, on every row. Tools that let the AI invent its own keys hand you a different spreadsheet each time. **Can it extract a list of products from one page?** Yes. When a page lists many things, it writes one row per item rather than cramming the page into a single row. Nothing to configure. **What happens when a page does not have one of my fields?** That cell is `null`. Values come from the page, never from the model's imagination. **Does it work on sites that block scrapers?** Usually. Pages that resist are retried with more effort, and you can set that effort per run. Sites behind a login, or behind a full image puzzle, are out of scope. **How accurate is it?** Accurate on facts plainly on the page: titles, prices, dates, names, links. Weaker on judgement calls and on values that exist only inside images. Switch on the page text in the output and spot-check your first run. **Why did a page come back empty?** Either the site refused every attempt, or the page genuinely has none of your fields. Failed pages are listed in the log and in the run report with a reason for each. **Can it scrape pages in other languages?** Yes. Pages are read in the language they are written in, and you can ask in the instructions for values to come back in another language. **How do I keep the cost down on a big crawl?** Set a firm page budget, narrow the crawl with URL patterns, keep effort on Auto, and lower the characters read per page when your data sits near the top. **Can I run it on a schedule?** Yes. Schedule the Actor and each run appends to a dataset you can diff against the last one. **How is this different from a website content crawler?** A content crawler hands you the page. This hands you the fields, already named, typed and ready for a spreadsheet. ### Support Something not extracting the way you expect? Open an issue on the Issues tab with the URL and the fields you asked for. # Actor input Schema ## `startUrls` (type: `array`): Pages to scrape. With crawling switched on, these are also where the crawl starts. ## `fields` (type: `array`): Name one column per line. Add a colon and a short hint when the name alone is ambiguous, for example: price: the current price in dollars, without the currency symbol. ## `instructions` (type: `string`): Describe in plain English what to pull out of each page. Use this on its own for free-form extraction, or alongside the field list to explain the context. Example: extract the top stories on this page, one row per story. ## `maxPages` (type: `integer`): Hard budget for the run. Each page costs one fetch and one extraction, so this is the number that controls the bill. ## `maxCrawlDepth` (type: `integer`): 0 scrapes only the URLs you listed. 1 also scrapes the pages they link to, and so on. Links are followed breadth first, so the pages nearest your start URLs are scraped first. ## `stayOnSite` (type: `boolean`): Follow links only within the site you started from. Turn this off with care: an open crawl can spend the whole page budget on somebody else's website. ## `includeUrlPatterns` (type: `array`): Wildcard patterns, for example https://example.com/blog/\*. When set, only matching links are followed. ## `excludeUrlPatterns` (type: `array`): Wildcard patterns to skip, for example */tag/* or */login*. ## `extractionMode` (type: `string`): Reuse works out a site's structure once and applies it to the rest of that site, which is faster and much cheaper on large runs. Read every page treats each page on its own, for a run where no two pages are alike. Reuse falls back to reading a page whenever the structure it worked out does not fit. ## `solveChallenges` (type: `boolean`): Work through the checks some sites show before their content. Turn off to fail fast on protected pages instead. ## `maxCharactersPerPage` (type: `integer`): How much of each page is read before extraction. Longer pages are truncated. Raise it for long articles, lower it to cut cost on pages where the answer is near the top. ## `includePageContent` (type: `boolean`): Adds the readable text of each page to its row, next to the extracted fields. Useful for spot-checking an extraction or for feeding a RAG pipeline. ## `effort` (type: `string`): How hard to work on pages that resist. Auto keeps the run fast and steps up only for pages that need it. Fast never steps up, which is cheapest on sites that scrape easily. Thorough treats every page as difficult and is slower. ## `concurrency` (type: `integer`): How many pages to work on at once. ## `proxyConfiguration` (type: `object`): Optional. Proxies help on sites that limit how much one address may read. Residential addresses work best on the strictest ones. ## Actor input object example ```json { "startUrls": [ { "url": "https://news.ycombinator.com/" } ], "fields": [ "product_name", "price: current price as a number", "in_stock: true or false" ], "instructions": "Extract the top stories listed on this page.", "maxPages": 10, "maxCrawlDepth": 0, "stayOnSite": true, "includeUrlPatterns": [], "excludeUrlPatterns": [], "extractionMode": "selectors", "solveChallenges": true, "maxCharactersPerPage": 12000, "includePageContent": false, "effort": "auto", "concurrency": 5, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `data` (type: `string`): Every scraped page with the fields you named and the page it came from. ## `rawData` (type: `string`): Every field of every row, including the page text when that option is on. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://news.ycombinator.com/" } ], "fields": [ "title", "url: link to the story", "points: score as a number", "author" ], "instructions": "Extract the top stories listed on this page.", "includeUrlPatterns": [], "excludeUrlPatterns": [], "proxyConfiguration": { "useApifyProxy": true } }; // Run the Actor and wait for it to finish const run = await client.actor("automly/ai-web-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://news.ycombinator.com/" }], "fields": [ "title", "url: link to the story", "points: score as a number", "author", ], "instructions": "Extract the top stories listed on this page.", "includeUrlPatterns": [], "excludeUrlPatterns": [], "proxyConfiguration": { "useApifyProxy": True }, } # Run the Actor and wait for it to finish run = client.actor("automly/ai-web-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://news.ycombinator.com/" } ], "fields": [ "title", "url: link to the story", "points: score as a number", "author" ], "instructions": "Extract the top stories listed on this page.", "includeUrlPatterns": [], "excludeUrlPatterns": [], "proxyConfiguration": { "useApifyProxy": true } }' | apify call automly/ai-web-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,automly/ai-web-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/oU9KniruuUhyUgVKr/builds/qA2AWzXsrejBSORth/openapi.json