AI Web Scraper (Any Website) avatar

AI Web Scraper (Any Website)

Pricing

from $1.19 / 1,000 results

Go to Apify Store
AI Web Scraper (Any Website)

AI Web Scraper (Any Website)

AI Web Scraper (Any Website) SD - AI Web Scraper (Any Website) is an AI data extraction tool that extracts structured JSON from any public web page with a plain-English prompt: one row per page or per item, pagination and crawling, JS pages, Markdown and RAG chunks. Free LLM or your own key.

Pricing

from $1.19 / 1,000 results

Rating

0.0

(0)

Developer

Neuro Scraper

Neuro Scraper

Maintained by Community

Actor stats

0

Bookmarked

3

Total users

2

Monthly active users

5 days ago

Last modified

Features

Limited permissions

Share

AI Web Scraper (Any Website)

The AI Web Scraper (Any Website) collects public AI web data and returns it as a clean dataset you can download as JSON, CSV or Excel, or read through the Apify API. No login, no cookies and no code: give the AI Web Scraper (Any Website) your input, start it and get structured AI data for research, monitoring, analytics or lead generation. You pay only for the results you receive.

⭐ Featured by Neuro Scraper: Facebook Reels Scraper - scrape Facebook Reels from any page, profile or reel link with exact views and plays, every reaction type (Like, Love, Care, Haha, Wow, Sad, Angry), comments, shares, captions, music and video links. No login and no cookies. Try the Facebook Reels Scraper →

./README.md

Input

Every option of the AI Web Scraper (Any Website), as it appears in the input form and the API:

FieldTypeDefaultWhat it does
startUrlsarray[{"url": "https://books.toscrape.com/"}]Pages to extract data from, e.g. https://books.toscrape.com/ or a product, article or listing page. Any public http(s) page; no login.
promptstring"Extract every book on this page: title,What to extract, in plain English, e.g. "Extract every product: name, price in USD, rating and product URL. Use null for fields the page does not show." Runs on every page. Empty prompt and no schema = the page is return
instructionsstring""Alternative name for 'Page extraction prompt' (used by GPT scraper tasks). When both are given they are joined.
outputModestring"page"'One row per item' turns every product / post / job / row the prompt describes into its own result row. 'Free-text answer' returns the model's text in 'answer'. Markdown and chunks need no AI model.
outputSchemaobject""A JSON Schema the answer must follow, e.g. {"type": "object", "properties": {"name": {"type": "string"}, "price": {"type": "number"}}}. For 'One row per item' give the schema of ONE item. OpenAI and OpenRouter get it as
outputFieldsarray[]Simpler than a schema: the keys every answer must use, e.g. title, price, availability, url.
schemaDescriptionstring""Extra notes about the fields, e.g. "price without currency symbol; date as YYYY-MM-DD".
extractionModestring"single"Single: each start URL plus its 'next page' links up to 'Max pages'. Crawl: follows links of the start URL's site (link selector, globs, depth). Scout / Agentic: reads the site's sitemap (or the start page's links), lets
maxPagesinteger1Pages read per start URL: the start page and its next pages in single mode (the page's own 'next' link is followed), pages extracted in crawl / scout / agentic mode.
maxItemsinteger100Stops after this many result rows in total (pages, items or chunks).
maxCrawlDepthinteger1Crawl mode: how many links away from the start URL the crawler goes (0 = start page only).
paginationSelectorstring""CSS selector of the 'next page' link, e.g. li.next a. Empty = found automatically (rel=next, or a link labelled Next / › / »).
linkSelectorstring""Crawl mode: CSS selector of the links to follow, e.g. article h3 a. Empty = every link on the page inside the start URL's site.
includeUrlGlobsarray[]Crawl / scout / agentic: only follow URLs matching one of these globs, e.g. https://books.toscrape.com/catalogue/**. ** = anything, * = anything but /.
excludeUrlGlobsarray[]Never follow URLs matching these globs, e.g. /login, /cart.
skipLlmGlobsarray[]Crawl mode: pages matching these globs are only used to find links - no AI call, no result (and no charge), e.g. category pages /category/.
targetSelectorstring""CSS selector of the part of the page sent to the AI, e.g. main, #content or .product-list. Empty = the whole page minus navigation, header, footer and cookie banners.
removeElementsCssSelectorstring""Elements removed before the AI reads the page, e.g. .ads, .related-posts, #comments.
contentScopestring"page"'Main content only' uses an article extractor (trafilatura) and may drop product grids; keep 'Whole page' for listings.
pageFormatInRequeststring"markdown"Markdown is compact; cleaned HTML keeps short class names and attributes some values live in.
removeLinkUrlsbooleanfalseDrops link URLs from the Markdown sent to the AI (fewer tokens, but the AI can no longer return item URLs).
maxContentCharsinteger40000Longer pages are cut at this length (about 4 characters per token). Raise it for long pages and large-context models.
includeMarkdownbooleanfalseAlso saves the page content that was sent to the AI in 'markdown' (always on in Markdown and Scout output).
renderJavaScriptstring"auto"Auto loads each page over HTTP first and opens it in headless Chromium only when the HTML has (almost) no text, or when a browser-only option below is set.
dynamicContentWaitSecsinteger0Browser: extra seconds to wait after the page loads.
waitForSelectorstring""Browser: wait until this CSS selector appears (max 20 s), e.g. .product-card.
hasInfiniteScrollbooleanfalseBrowser: scroll to the bottom until no new content loads (up to 'Max scrolls') before reading the page.
maxScrollsinteger10Browser: upper limit of scrolls per page when 'Infinite scroll' is on.
handleCookieConsentbooleantrueBrowser: clicks a visible 'Accept' / 'Accept all' / 'I agree' button of a cookie banner. In HTTP mode cookie banners are removed from the content.
deviceTypestring"desktop"Browser: desktop or a phone (Playwright's device emulation: screen, user agent, touch). Mobile pages are often lighter.
mobileDeviceModelstring""Browser, mobile: a Playwright device name, e.g. iPhone 13, Pixel 7, Galaxy S9+. Empty = the newest iPhone in Playwright's list.
viewportWidthinteger""Browser: custom window width in pixels (default 1366 desktop / the phone's own).
viewportHeightinteger""Browser: custom window height in pixels (default 900 desktop / the phone's own).
saveScreenshotsbooleanfalseBrowser: saves a full-page PNG of every page in the key-value store; the link is in 'screenshotUrl'. Turns the browser on.
llmProviderstring"free"Free uses Pollinations.ai's anonymous text API: no key, a small open model (GPT-OSS 20B on 2026-10-09), about one AI call every 20-30 seconds (its anonymous limit), answers up to about 1,500 tokens (about 20-30 items per
llmApiKeystring""Your API key for the chosen provider (OpenAI sk-..., Anthropic sk-ant-..., Gemini AIza..., OpenRouter sk-or-...). Not needed for Free.
modelstring""Model id, e.g. gpt-4.1-mini, claude-haiku-4-5, gemini-2.5-flash, openai/gpt-4o-mini. Empty = picked at run time from the provider's own model list (a fast, low-cost model).
llmBaseUrlstring""Only for 'Any OpenAI-compatible API': the base URL ending in /v1, e.g. https://api.groq.com/openai/v1, https://api.deepseek.com/v1, https://api.mistral.ai/v1.
temperaturestring"0"0 = most deterministic (best for extraction). Models that do not take it get the request without it.
topPstring""Nucleus sampling, e.g. 1. Empty = the model's default.
frequencyPenaltystring""OpenAI-format models only, e.g. 0. Empty = the model's default.
presencePenaltystring""OpenAI-format models only, e.g. 0. Empty = the model's default.
maxOutputTokensinteger""Upper limit of the answer length per page. Empty = the model's default (Anthropic: 4096).
reasoningEffortstring""OpenAI-format reasoning models (gpt-5, o-series, GPT-OSS): how long the model thinks before answering. Extraction rarely needs more than low. Models that do not take it get the request without it.
chunkSizeinteger512Chunks output: target chunk size (approx. tokens = characters / 4).
chunkOverlapinteger64Chunks output: tokens repeated from the end of one chunk at the start of the next.
respectRobotsTxtbooleantrueSkips pages the site's robots.txt disallows for all crawlers.
maxConcurrencyinteger2How many start URLs are processed at the same time (pages of one site are read one after another, politely).
proxyConfigurationobject{"useApifyProxy": true, "apifyProxyGroupResidential proxies (the default) get through to most sites; pick a country when a site shows region-specific content. The AI calls do not use the proxy.

Output fields

Each AI Web Scraper (Any Website) row is one JSON object with these fields (CSV and Excel exports use the same columns):

  • url
  • title
  • data
  • answer
  • itemIndex
  • markdown
  • text
  • chunkIndex
  • headingPath
  • tokenCount
  • groundedRatio
  • inputUrl
  • pageNumber
  • depth
  • model
  • usage
  • renderedWith
  • statusCode
  • screenshotUrl
  • scrapedAt

Why use the AI Web Scraper (Any Website)

The AI Web Scraper (Any Website) is a AI scraper and AI data extractor for anyone who needs public AI web data as clean data. The AI Web Scraper (Any Website) handles web scraping, proxy rotation and retries for you, so you get a ready dataset instead of raw pages. Use the AI Web Scraper (Any Website) for market research, competitor analysis, lead generation, analytics or monitoring, and export the results to JSON, CSV, Excel or Google Sheets.

  • No login and no code: the AI Web Scraper (Any Website) runs on Apify with one click or one API call.
  • Pay per result: the AI Web Scraper (Any Website) charges only for rows delivered - no monthly AI API fee.
  • Complete fields: the AI Web Scraper (Any Website) returns every public field, with a stable schema for automation.
  • Integrations: connect the AI Web Scraper (Any Website) to Zapier, Make, n8n, Google Sheets, a webhook or your CRM.

AI Web Scraper (Any Website) use cases

  • Market research: the AI Web Scraper (Any Website) collects AI web data at scale, so you can compare trends across weeks and niches.
  • Competitor analysis: run the AI Web Scraper (Any Website) on a schedule and export the dataset to Google Sheets or Excel to watch what competitors publish.
  • Lead generation and B2B prospecting: the AI Web Scraper (Any Website) gives a sales or agency team a clean list to load into a CRM.
  • Content and social media monitoring: the AI Web Scraper (Any Website) keeps a fresh copy of AI data through an Apify schedule and a webhook.
  • Analytics and dashboards: send the AI Web Scraper (Any Website) output as JSON through the Apify API into Python, Node.js or a BI tool.
  • Academic and data science projects: the AI Web Scraper (Any Website) returns a consistent schema, ready for pandas, R or a notebook.
  • Price, product and brand tracking: the AI Web Scraper (Any Website) turns public AI data into a dataset you can diff run by run.
  • Automation workflows: connect the AI Web Scraper (Any Website) to Zapier, Make or n8n so every new row starts your next step.

How to run it with the API, Python and Node.js

Start the AI Web Scraper (Any Website) from the Apify Console, or call it from code. The AI Web Scraper (Any Website) returns its dataset as JSON, CSV or Excel. With the Apify API you can run the AI Web Scraper (Any Website) from Python or Node.js, wait for it to finish and read the rows, which makes the AI Web Scraper (Any Website) easy to plug into an automation, a data pipeline or a scheduled export.

  1. Open the AI Web Scraper (Any Website) on Apify and fill in the input (or send the same input as JSON to the API).
  2. Start the AI Web Scraper (Any Website). Follow the run log while the AI Web Scraper (Any Website) collects public AI data.
  3. Download the dataset as CSV, JSON, Excel or XLSX, or let a webhook send it on.
  4. Schedule the AI Web Scraper (Any Website) to keep your AI data fresh for monitoring and analytics.

Tips for better results

  • Start small: run the AI Web Scraper (Any Website) with a low maximum first, check the fields, then scale up the same input.
  • Split big jobs: several smaller AI Web Scraper (Any Website) runs (by keyword, URL list or date) finish faster and are easier to resume.
  • Keep a history: name a dataset or use a schedule so each AI Web Scraper (Any Website) run adds to your AI data over time.
  • Filter early: use the input filters of the AI Web Scraper (Any Website) so you only pay for the AI web data you really need.
  • Automate the hand-off: a webhook or the Zapier and Make integrations can move every AI Web Scraper (Any Website) result straight to Google Sheets, a CRM or a database.
  • Watch the run log: the AI Web Scraper (Any Website) writes what it is doing and why an item was skipped, so a surprising result is easy to explain.

AI Web Scraper (Any Website) FAQ

Can I export the results to Excel, CSV or Google Sheets?

Yes. Every AI Web Scraper (Any Website) run stores a dataset you can export as CSV, JSON, Excel (XLSX), XML or HTML, or send to Google Sheets with the Apify integration.

How fast is it?

The AI Web Scraper (Any Website) runs requests in parallel and rotates proxy sessions automatically, so a typical run of a few hundred items finishes in minutes.

Can it replace the official AI API?

For public data, yes. The AI Web Scraper (Any Website) gives you AI data through the Apify API without an official AI API key, quota approval or developer account.

How much does it cost?

The AI Web Scraper (Any Website) is pay per result: you pay only for the rows you receive, with lower prices on higher Apify plans. You can cap the cost of every run with the maximum cost setting.

Can I send the data to my CRM?

Yes. Push AI Web Scraper (Any Website) rows to HubSpot, Salesforce, Pipedrive or any CRM through Zapier, Make or a webhook - a common setup for an agency or B2B sales team.

Do I need a AI account or login?

No. The AI Web Scraper (Any Website) works with no login and no cookies. It reads only public data, the same data any visitor can see.

Can I schedule it?

Yes. Apify schedules run the AI Web Scraper (Any Website) hourly, daily or weekly, and a webhook can notify your system when new AI data is ready.

Can I run it for many inputs at once?

Yes. Give the AI Web Scraper (Any Website) a list of inputs in one run; each row says which input it came from, so one AI Web Scraper (Any Website) run can cover a whole list.

How do I get more results?

Raise the maximum items, add more inputs (keywords, URLs or profiles), or run the AI Web Scraper (Any Website) several times on a schedule. Results are de-duplicated.

How is this different from other AI scrapers?

The AI Web Scraper (Any Website) combines complete public fields, no login, pay-per-result pricing and a stable output schema, so the AI Web Scraper (Any Website) fits both quick one-off exports and long-running automation.

Can I use it through the API?

Yes. Start the AI Web Scraper (Any Website) and read its dataset through the Apify API from Python, Node.js or any HTTP client, or call it from Zapier, Make and n8n.

More scrapers from Neuro Scraper

ActorWhat it extracts
⭐ Facebook Reels ScraperFacebook Reels with exact views, plays, every reaction type, comments and shares
Airbnb Reviews ScraperAirbnb reviews with all public fields, exported to JSON, CSV or Excel
Airbnb ScraperAirbnb data with all public fields, exported to JSON, CSV or Excel
Snapchat Profile ScraperSnapchat profile data with all public fields, exported to JSON, CSV or Excel
Amazon ScraperAmazon data with all public fields, exported to JSON, CSV or Excel
Instagram Reels ScraperInstagram reels with all public fields, exported to JSON, CSV or Excel
Facebook Video Details ScraperFacebook video details with all public fields, exported to JSON, CSV or Excel
Facebook Reels Scraper AdvancedFacebook reels advanced data with all public fields, exported to JSON, CSV or Excel
Facebook Reels DownloaderFacebook reels with all public fields, exported to JSON, CSV or Excel
Instagram Search ScraperInstagram search data with all public fields, exported to JSON, CSV or Excel
TikTok Video Details ScraperTikTok video details with all public fields, exported to JSON, CSV or Excel
Instagram Hashtag ScraperInstagram hashtag data with all public fields, exported to JSON, CSV or Excel
TikTok Profile ScraperTikTok profile data with all public fields, exported to JSON, CSV or Excel