AI Web Scraper (Any Website)
Pricing
from $1.19 / 1,000 results
AI Web Scraper (Any Website)
AI Web Scraper (Any Website) SD - AI Web Scraper (Any Website) is an AI data extraction tool that extracts structured JSON from any public web page with a plain-English prompt: one row per page or per item, pagination and crawling, JS pages, Markdown and RAG chunks. Free LLM or your own key.
Pricing
from $1.19 / 1,000 results
Rating
0.0
(0)
Developer
Neuro Scraper
Maintained by CommunityActor stats
0
Bookmarked
3
Total users
2
Monthly active users
5 days ago
Last modified
Categories
Features
Share
AI Web Scraper (Any Website)
The AI Web Scraper (Any Website) collects public AI web data and returns it as a clean dataset you can download as JSON, CSV or Excel, or read through the Apify API. No login, no cookies and no code: give the AI Web Scraper (Any Website) your input, start it and get structured AI data for research, monitoring, analytics or lead generation. You pay only for the results you receive.
⭐ Featured by Neuro Scraper: Facebook Reels Scraper - scrape Facebook Reels from any page, profile or reel link with exact views and plays, every reaction type (Like, Love, Care, Haha, Wow, Sad, Angry), comments, shares, captions, music and video links. No login and no cookies. Try the Facebook Reels Scraper →
./README.md
Input
Every option of the AI Web Scraper (Any Website), as it appears in the input form and the API:
| Field | Type | Default | What it does |
|---|---|---|---|
startUrls | array | [{"url": "https://books.toscrape.com/"}] | Pages to extract data from, e.g. https://books.toscrape.com/ or a product, article or listing page. Any public http(s) page; no login. |
prompt | string | "Extract every book on this page: title, | What to extract, in plain English, e.g. "Extract every product: name, price in USD, rating and product URL. Use null for fields the page does not show." Runs on every page. Empty prompt and no schema = the page is return |
instructions | string | "" | Alternative name for 'Page extraction prompt' (used by GPT scraper tasks). When both are given they are joined. |
outputMode | string | "page" | 'One row per item' turns every product / post / job / row the prompt describes into its own result row. 'Free-text answer' returns the model's text in 'answer'. Markdown and chunks need no AI model. |
outputSchema | object | "" | A JSON Schema the answer must follow, e.g. {"type": "object", "properties": {"name": {"type": "string"}, "price": {"type": "number"}}}. For 'One row per item' give the schema of ONE item. OpenAI and OpenRouter get it as |
outputFields | array | [] | Simpler than a schema: the keys every answer must use, e.g. title, price, availability, url. |
schemaDescription | string | "" | Extra notes about the fields, e.g. "price without currency symbol; date as YYYY-MM-DD". |
extractionMode | string | "single" | Single: each start URL plus its 'next page' links up to 'Max pages'. Crawl: follows links of the start URL's site (link selector, globs, depth). Scout / Agentic: reads the site's sitemap (or the start page's links), lets |
maxPages | integer | 1 | Pages read per start URL: the start page and its next pages in single mode (the page's own 'next' link is followed), pages extracted in crawl / scout / agentic mode. |
maxItems | integer | 100 | Stops after this many result rows in total (pages, items or chunks). |
maxCrawlDepth | integer | 1 | Crawl mode: how many links away from the start URL the crawler goes (0 = start page only). |
paginationSelector | string | "" | CSS selector of the 'next page' link, e.g. li.next a. Empty = found automatically (rel=next, or a link labelled Next / › / »). |
linkSelector | string | "" | Crawl mode: CSS selector of the links to follow, e.g. article h3 a. Empty = every link on the page inside the start URL's site. |
includeUrlGlobs | array | [] | Crawl / scout / agentic: only follow URLs matching one of these globs, e.g. https://books.toscrape.com/catalogue/**. ** = anything, * = anything but /. |
excludeUrlGlobs | array | [] | Never follow URLs matching these globs, e.g. /login, /cart. |
skipLlmGlobs | array | [] | Crawl mode: pages matching these globs are only used to find links - no AI call, no result (and no charge), e.g. category pages /category/. |
targetSelector | string | "" | CSS selector of the part of the page sent to the AI, e.g. main, #content or .product-list. Empty = the whole page minus navigation, header, footer and cookie banners. |
removeElementsCssSelector | string | "" | Elements removed before the AI reads the page, e.g. .ads, .related-posts, #comments. |
contentScope | string | "page" | 'Main content only' uses an article extractor (trafilatura) and may drop product grids; keep 'Whole page' for listings. |
pageFormatInRequest | string | "markdown" | Markdown is compact; cleaned HTML keeps short class names and attributes some values live in. |
removeLinkUrls | boolean | false | Drops link URLs from the Markdown sent to the AI (fewer tokens, but the AI can no longer return item URLs). |
maxContentChars | integer | 40000 | Longer pages are cut at this length (about 4 characters per token). Raise it for long pages and large-context models. |
includeMarkdown | boolean | false | Also saves the page content that was sent to the AI in 'markdown' (always on in Markdown and Scout output). |
renderJavaScript | string | "auto" | Auto loads each page over HTTP first and opens it in headless Chromium only when the HTML has (almost) no text, or when a browser-only option below is set. |
dynamicContentWaitSecs | integer | 0 | Browser: extra seconds to wait after the page loads. |
waitForSelector | string | "" | Browser: wait until this CSS selector appears (max 20 s), e.g. .product-card. |
hasInfiniteScroll | boolean | false | Browser: scroll to the bottom until no new content loads (up to 'Max scrolls') before reading the page. |
maxScrolls | integer | 10 | Browser: upper limit of scrolls per page when 'Infinite scroll' is on. |
handleCookieConsent | boolean | true | Browser: clicks a visible 'Accept' / 'Accept all' / 'I agree' button of a cookie banner. In HTTP mode cookie banners are removed from the content. |
deviceType | string | "desktop" | Browser: desktop or a phone (Playwright's device emulation: screen, user agent, touch). Mobile pages are often lighter. |
mobileDeviceModel | string | "" | Browser, mobile: a Playwright device name, e.g. iPhone 13, Pixel 7, Galaxy S9+. Empty = the newest iPhone in Playwright's list. |
viewportWidth | integer | "" | Browser: custom window width in pixels (default 1366 desktop / the phone's own). |
viewportHeight | integer | "" | Browser: custom window height in pixels (default 900 desktop / the phone's own). |
saveScreenshots | boolean | false | Browser: saves a full-page PNG of every page in the key-value store; the link is in 'screenshotUrl'. Turns the browser on. |
llmProvider | string | "free" | Free uses Pollinations.ai's anonymous text API: no key, a small open model (GPT-OSS 20B on 2026-10-09), about one AI call every 20-30 seconds (its anonymous limit), answers up to about 1,500 tokens (about 20-30 items per |
llmApiKey | string | "" | Your API key for the chosen provider (OpenAI sk-..., Anthropic sk-ant-..., Gemini AIza..., OpenRouter sk-or-...). Not needed for Free. |
model | string | "" | Model id, e.g. gpt-4.1-mini, claude-haiku-4-5, gemini-2.5-flash, openai/gpt-4o-mini. Empty = picked at run time from the provider's own model list (a fast, low-cost model). |
llmBaseUrl | string | "" | Only for 'Any OpenAI-compatible API': the base URL ending in /v1, e.g. https://api.groq.com/openai/v1, https://api.deepseek.com/v1, https://api.mistral.ai/v1. |
temperature | string | "0" | 0 = most deterministic (best for extraction). Models that do not take it get the request without it. |
topP | string | "" | Nucleus sampling, e.g. 1. Empty = the model's default. |
frequencyPenalty | string | "" | OpenAI-format models only, e.g. 0. Empty = the model's default. |
presencePenalty | string | "" | OpenAI-format models only, e.g. 0. Empty = the model's default. |
maxOutputTokens | integer | "" | Upper limit of the answer length per page. Empty = the model's default (Anthropic: 4096). |
reasoningEffort | string | "" | OpenAI-format reasoning models (gpt-5, o-series, GPT-OSS): how long the model thinks before answering. Extraction rarely needs more than low. Models that do not take it get the request without it. |
chunkSize | integer | 512 | Chunks output: target chunk size (approx. tokens = characters / 4). |
chunkOverlap | integer | 64 | Chunks output: tokens repeated from the end of one chunk at the start of the next. |
respectRobotsTxt | boolean | true | Skips pages the site's robots.txt disallows for all crawlers. |
maxConcurrency | integer | 2 | How many start URLs are processed at the same time (pages of one site are read one after another, politely). |
proxyConfiguration | object | {"useApifyProxy": true, "apifyProxyGroup | Residential proxies (the default) get through to most sites; pick a country when a site shows region-specific content. The AI calls do not use the proxy. |
Output fields
Each AI Web Scraper (Any Website) row is one JSON object with these fields (CSV and Excel exports use the same columns):
urltitledataansweritemIndexmarkdowntextchunkIndexheadingPathtokenCountgroundedRatioinputUrlpageNumberdepthmodelusagerenderedWithstatusCodescreenshotUrlscrapedAt
Why use the AI Web Scraper (Any Website)
The AI Web Scraper (Any Website) is a AI scraper and AI data extractor for anyone who needs public AI web data as clean data. The AI Web Scraper (Any Website) handles web scraping, proxy rotation and retries for you, so you get a ready dataset instead of raw pages. Use the AI Web Scraper (Any Website) for market research, competitor analysis, lead generation, analytics or monitoring, and export the results to JSON, CSV, Excel or Google Sheets.
- No login and no code: the AI Web Scraper (Any Website) runs on Apify with one click or one API call.
- Pay per result: the AI Web Scraper (Any Website) charges only for rows delivered - no monthly AI API fee.
- Complete fields: the AI Web Scraper (Any Website) returns every public field, with a stable schema for automation.
- Integrations: connect the AI Web Scraper (Any Website) to Zapier, Make, n8n, Google Sheets, a webhook or your CRM.
AI Web Scraper (Any Website) use cases
- Market research: the AI Web Scraper (Any Website) collects AI web data at scale, so you can compare trends across weeks and niches.
- Competitor analysis: run the AI Web Scraper (Any Website) on a schedule and export the dataset to Google Sheets or Excel to watch what competitors publish.
- Lead generation and B2B prospecting: the AI Web Scraper (Any Website) gives a sales or agency team a clean list to load into a CRM.
- Content and social media monitoring: the AI Web Scraper (Any Website) keeps a fresh copy of AI data through an Apify schedule and a webhook.
- Analytics and dashboards: send the AI Web Scraper (Any Website) output as JSON through the Apify API into Python, Node.js or a BI tool.
- Academic and data science projects: the AI Web Scraper (Any Website) returns a consistent schema, ready for pandas, R or a notebook.
- Price, product and brand tracking: the AI Web Scraper (Any Website) turns public AI data into a dataset you can diff run by run.
- Automation workflows: connect the AI Web Scraper (Any Website) to Zapier, Make or n8n so every new row starts your next step.
How to run it with the API, Python and Node.js
Start the AI Web Scraper (Any Website) from the Apify Console, or call it from code. The AI Web Scraper (Any Website) returns its dataset as JSON, CSV or Excel. With the Apify API you can run the AI Web Scraper (Any Website) from Python or Node.js, wait for it to finish and read the rows, which makes the AI Web Scraper (Any Website) easy to plug into an automation, a data pipeline or a scheduled export.
- Open the AI Web Scraper (Any Website) on Apify and fill in the input (or send the same input as JSON to the API).
- Start the AI Web Scraper (Any Website). Follow the run log while the AI Web Scraper (Any Website) collects public AI data.
- Download the dataset as CSV, JSON, Excel or XLSX, or let a webhook send it on.
- Schedule the AI Web Scraper (Any Website) to keep your AI data fresh for monitoring and analytics.
Tips for better results
- Start small: run the AI Web Scraper (Any Website) with a low maximum first, check the fields, then scale up the same input.
- Split big jobs: several smaller AI Web Scraper (Any Website) runs (by keyword, URL list or date) finish faster and are easier to resume.
- Keep a history: name a dataset or use a schedule so each AI Web Scraper (Any Website) run adds to your AI data over time.
- Filter early: use the input filters of the AI Web Scraper (Any Website) so you only pay for the AI web data you really need.
- Automate the hand-off: a webhook or the Zapier and Make integrations can move every AI Web Scraper (Any Website) result straight to Google Sheets, a CRM or a database.
- Watch the run log: the AI Web Scraper (Any Website) writes what it is doing and why an item was skipped, so a surprising result is easy to explain.
AI Web Scraper (Any Website) FAQ
Can I export the results to Excel, CSV or Google Sheets?
Yes. Every AI Web Scraper (Any Website) run stores a dataset you can export as CSV, JSON, Excel (XLSX), XML or HTML, or send to Google Sheets with the Apify integration.
How fast is it?
The AI Web Scraper (Any Website) runs requests in parallel and rotates proxy sessions automatically, so a typical run of a few hundred items finishes in minutes.
Can it replace the official AI API?
For public data, yes. The AI Web Scraper (Any Website) gives you AI data through the Apify API without an official AI API key, quota approval or developer account.
How much does it cost?
The AI Web Scraper (Any Website) is pay per result: you pay only for the rows you receive, with lower prices on higher Apify plans. You can cap the cost of every run with the maximum cost setting.
Can I send the data to my CRM?
Yes. Push AI Web Scraper (Any Website) rows to HubSpot, Salesforce, Pipedrive or any CRM through Zapier, Make or a webhook - a common setup for an agency or B2B sales team.
Do I need a AI account or login?
No. The AI Web Scraper (Any Website) works with no login and no cookies. It reads only public data, the same data any visitor can see.
Can I schedule it?
Yes. Apify schedules run the AI Web Scraper (Any Website) hourly, daily or weekly, and a webhook can notify your system when new AI data is ready.
Can I run it for many inputs at once?
Yes. Give the AI Web Scraper (Any Website) a list of inputs in one run; each row says which input it came from, so one AI Web Scraper (Any Website) run can cover a whole list.
How do I get more results?
Raise the maximum items, add more inputs (keywords, URLs or profiles), or run the AI Web Scraper (Any Website) several times on a schedule. Results are de-duplicated.
How is this different from other AI scrapers?
The AI Web Scraper (Any Website) combines complete public fields, no login, pay-per-result pricing and a stable output schema, so the AI Web Scraper (Any Website) fits both quick one-off exports and long-running automation.
Can I use it through the API?
Yes. Start the AI Web Scraper (Any Website) and read its dataset through the Apify API from Python, Node.js or any HTTP client, or call it from Zapier, Make and n8n.
More scrapers from Neuro Scraper
| Actor | What it extracts |
|---|---|
| ⭐ Facebook Reels Scraper | Facebook Reels with exact views, plays, every reaction type, comments and shares |
| Airbnb Reviews Scraper | Airbnb reviews with all public fields, exported to JSON, CSV or Excel |
| Airbnb Scraper | Airbnb data with all public fields, exported to JSON, CSV or Excel |
| Snapchat Profile Scraper | Snapchat profile data with all public fields, exported to JSON, CSV or Excel |
| Amazon Scraper | Amazon data with all public fields, exported to JSON, CSV or Excel |
| Instagram Reels Scraper | Instagram reels with all public fields, exported to JSON, CSV or Excel |
| Facebook Video Details Scraper | Facebook video details with all public fields, exported to JSON, CSV or Excel |
| Facebook Reels Scraper Advanced | Facebook reels advanced data with all public fields, exported to JSON, CSV or Excel |
| Facebook Reels Downloader | Facebook reels with all public fields, exported to JSON, CSV or Excel |
| Instagram Search Scraper | Instagram search data with all public fields, exported to JSON, CSV or Excel |
| TikTok Video Details Scraper | TikTok video details with all public fields, exported to JSON, CSV or Excel |
| Instagram Hashtag Scraper | Instagram hashtag data with all public fields, exported to JSON, CSV or Excel |
| TikTok Profile Scraper | TikTok profile data with all public fields, exported to JSON, CSV or Excel |
