Claude AI Web Automation

Claude AI Web Automation is an Apify Actor that drives a real browser with Anthropic's Claude models to navigate any website and extract structured data — no CSS selectors or page-specific scraping code required. Give it a starting URL and a plain-English instruction (e.g. "Open Sporting Goods > Golf, sort by newly listed, and scrape the first 5 listings") and the agent handles the search, navigation, pagination, and detail-page extraction for you.

Try it directly from the Input tab — fill in a target site and a prompt, hit Start, and the dataset will fill up with clean JSON. Run it on the Apify platform to take advantage of API access, scheduling, integrations (Zapier, Make, Slack), automatic proxy rotation, and run monitoring.

What does Claude AI Web Automation do?

The Actor combines four AI building blocks into a single end-to-end web scraper:

1. Intent parsing — Claude reads your prompt and decides what action to perform, what fields to extract, and how many items you want.
2. Browser automation — A Stagehand agent powered by Claude takes screenshots, clicks, types, and scrolls until it reaches the search results.
3. Selector inference — Claude inspects the DOM of the results page and derives CSS selectors for every listing and the next-page link.
4. Structured extraction — Each detail page is opened and Claude pulls the fields you asked for into a Zod-validated record.

Everything runs serverless on the Apify platform — no local infrastructure required.

Why use Claude AI Web Automation?

• No selectors, no glue code. Describe the task in English and the agent figures out the rest.
• Works across sites. The same Actor scrapes eBay, Etsy, news portals, real-estate listings, or any other public site.
• Resilient defaults. Residential proxy rotation, infinite-scroll handling, cross-page deduplication, and hard timeouts are all built in.
• Cheap models, big results. Uses claude-haiku-4-5 for parsing/selector derivation, keeping per-run costs low.
• Structured output. Each record is a typed JSON object you can pipe into a sheet, database, or downstream Actor.

How to use Claude AI Web Automation

1. Open the Actor page on Apify Console and click Try for free.
2. In the Input tab, paste a starting URL (e.g. https://www.ebay.com).
3. Write your prompt in plain English — what to navigate to and what to scrape.
4. (Optional) Set maxItems to control how many detail records you want, and maxSteps to bound the agent's reasoning loop.
5. (Optional) Provide explicit selectors and a uniqueKeyPattern to skip the LLM selector-inference step on sites you already know.
6. Click Start and watch the run log. When it finishes, switch to the Output tab or call the dataset API.

Input

Field	Type	Required	Description
prompt	string	✅	What to do on the site and what to extract. Include a count if you want (e.g. "the first 10 results").
startUrls	array	✅	URLs the browser opens first. Each item must have a url key.
maxItems	integer	—	Hard cap on detail records. Overrides any count parsed from the prompt. Default: 5.
maxSteps	integer	—	Max reasoning steps for the pre-crawl agent. Default: 10.
selectors	object	—	Optional listingLink and nextPage CSS selectors. If omitted, Claude derives them.
uniqueKeyPattern	string	—	Optional regex with one capture group for deduping listings (e.g. /itm/(\d{9,}) for eBay).
countryCode	string	—	Two-letter ISO code for residential proxy geolocation. Default: US.
loginCookies	array	—	Cookies to inject before navigation, for authenticated scraping. Accepts Cookie-Editor / EditThisCookie exports or raw Playwright cookie objects.

Example input

{
    "startUrls": [{ "url": "https://www.ebay.com" }],
    "prompt": "Open the category page Sporting Goods > Golf, then sort by newly listed and scrape the first 5 listings by going to the detail page to extract title, price, and shipping cost.",
    "maxItems": 5,
    "maxSteps": 10,
    "countryCode": "US",
    "selectors": {
        "listingLink": "a[href*='/itm/']",
        "nextPage": "a[aria-label='Next page'], a[rel='next']"
    },
    "uniqueKeyPattern": "/itm/(\\d{9,})"
}

Output

Each detail page produces one JSON record in the default dataset. The fields are derived from your prompt — if you ask for "title, price, and shipping cost", you get exactly those plus the source URL.

{
    "url": "https://www.ebay.com/itm/186372216016",
    "title": "Callaway Rogue ST Max Driver — 10.5° Stiff Flex",
    "price": "$249.99",
    "shippingCost": "Free shipping"
}

You can download the dataset in JSON, CSV, Excel, HTML, XML, or RSS formats from the Storage tab or via the Apify API.

Data fields

Field	Type	Description
url	string	Canonical URL of the detail page.
(prompt-defined fields)	string / number / boolean	Whatever you asked the agent to extract. Currency-like fields (price, shipping, fee) are always returned as strings to preserve symbols and free-form text such as "Free shipping".

Cost estimation

The Actor charges only the standard Apify platform costs (compute units + residential proxy traffic) plus a small Anthropic API spend. A typical run that scrapes 5 detail records on a clean site:

• Compute: ~0.02–0.05 CU (a few cents on a paid plan).
• Proxy traffic: a few MB of residential bandwidth.
• Anthropic tokens: roughly $0.01–$0.03 per run with claude-haiku-4-5.

Apify's free tier covers many runs per month. Use maxItems and maxSteps to bound costs on slow or anti-bot-protected sites.

Tips and advanced options

• Be specific in the prompt. "Extract the title, price in USD, and number of reviews" beats "Extract product info".
• Pre-supply selectors for sites you scrape often — it skips an LLM call and makes runs faster and more deterministic.
• Bump maxSteps for multi-page flows (search → filter → sort → results often needs 8–12 steps).
• Lower maxSteps to fail fast when debugging a new site.
• Pin a countryCode that matches the target site's primary market — pricing, currency, and availability often depend on geo.
• Anti-bot pages. Some sites (eBay, Amazon, Cloudflare-protected portals) occasionally serve CAPTCHAs to residential proxies. Retry the run or switch country code if you see a challenge page in the logs.
• Authenticated scraping. Sign in to the target site in your normal browser, export the cookies with the Cookie-Editor extension, and paste the JSON array into loginCookies. The Actor runs at concurrency 1 so the session is reused across all requests and is less likely to be flagged.

FAQ and support

Is web scraping legal? Scraping publicly available data is generally legal, but you must respect each site's Terms of Service and robots.txt and avoid extracting personal data without a lawful basis. You are responsible for how you use the output.

What model does it use? claude-haiku-4-5 for prompt parsing, selector derivation, and the Stagehand browser agent. You can change the model by editing src/main.js.

Can I run it on my laptop? Yes — clone the repo, run npm install, set ANTHROPIC_API_KEY in .env, and run apify run. Local runs use a local browser (no Apify proxy unless you pass credentials).

Where do I report issues? Use the Issues tab on the Actor page. For custom scraping projects, contact dtrungtin@gmail.com.

Related Actors

• OpenAI Web Scraper — dtrungtin/openai-web-scraper