Web Scraping API

Imagine scraping websites without having to write a single HTML selector ever again.

Then, add to that dream the freedom of never worrying about proxies, CAPTCHAs, or site updates and broken scrapers.

You wake up, the sun is shining, the birds are singing 🐦, and your scrapers are humming along just nicely.

But here’s the thing... it’s not just a dream.

This is the Web Scraping API that quickly and reliably scrapes any website—no selectors required. Plus, it’s just 0.5¢ per page, with no minimum spend.

⚡ Highlights

• AI-driven Extraction: Automatically parse content and return structured JSON—no manual HTML parsing.
• Multiple Endpoints: /scrape, /links, /next, /search handle everything from detailed data extraction to finding links, pagination, and Google Search.
• No Maintenance Hassles: We handle proxy rotation, CAPTCHA bypass, and site changes behind the scenes.
• Flexible Output: Get your data as JSON, plus optional Markdown or raw HTML.
• Pay Only for What You Use: 0.5¢ per page scraped, no hidden fees or monthly minimums.

📌 What Can It Do?

/scrape – Extract Structured Data

• Use Cases: Product listings, job postings, news articles, user profiles, or any site content.
• How It Works: Specify the fields you want in natural language, and our AI returns them in a neat JSON payload.

Open /scrape Endpoint Documentation

/links – Gather Specific Page Links

• Use Cases: Collect subcategory links on e-commerce sites, internal links for website crawling, external references from a research article, or any links matching your description.
• How It Works: Provide a page URL and a description of the links you’re looking for (e.g., “individual product URLs”). We’ll return matching links in seconds.

Open /links Endpoint Documentation

/next – Fetch “Next Page” Links for Pagination

• Use Cases: Seamlessly navigate through multi-page listings—think job boards, forums, real estate listings, or multi-page e-commerce categories.
• How It Works: Call this endpoint on any paginated page to auto-discover the next page URLs so you can keep scraping.

Open /next Endpoint Documentation

/search – Scrape Google Search Results

• Use Cases: Gather relevant results for lead generation, competitor analysis, SEO research, or building domain lists for further scraping.
• How It Works: Provide a query (and location domain), then let the API handle the rest—bypassing geolocation quirks and potential CAPTCHAs automatically.

Open /search Endpoint Documentation

🛠 Endpoints

/scrape

Extract structured data from any webpage with a single call. There are also advanced usage techniques you can leverage.

1{
2    "endpoint": "scrape",
3    "url": "https://apify.com/pricing",
4    "fields": {
5        "plans": [
6            {
7                "name": "",
8                "monthly_price": "",
9                "prepaid_usage": "",
10                "compute_units": "",
11                "actor_ram": "",
12                "max_concurrent_users": "",
13                "rented_actors": ""
14            }
15        ]
16    }
17}

Sample response:

1{
2    "scrape": {
3        "plans": [
4            {
5                "name": "Free",
6                "monthly_price": "$0",
7                "prepaid_usage": "$5",
8                "compute_units": "$0.4/CU",
9                "actor_ram": "8 GB",
10                "max_concurrent_users": "25",
11                "rented_actors": "Limited"
12            },
13            ...
14        ]
15    },
16    "markdown": "< Markdown of the page ... >",
17    "html": "< HTML of the page ... >"
18}

/links

Find all links on a page that match a specific description.

1{
2    "endpoint": "links",
3    "url": "https://www.ikea.com/au/en/cat/quilt-cover-sets-10680/?page=3",
4    "description": "individual product urls"
5}

Sample response:

1{
2    "links": [
3        "https://www.ikea.com/au/en/p/strandlummer-duvet-cover-and-pillowcase-multicolour-floral-pattern-00579802/",
4        "https://www.ikea.com/au/en/p/gullnattljus-duvet-cover-and-2-pillowcases-white-floral-pattern-60600075/",
5        ...
6    ],
7    "markdown": "< Markdown of the page ... >",
8    "html": "< HTML of the page ... >"
9}

/next

Extract “Next Page” links from a paginated web page—perfect for multi-page listings or search results.

1{
2    "endpoint": "next",
3    "url": "https://www.ikea.com/au/en/cat/quilt-cover-sets-10680/"
4}

Sample response:

1{
2    "next": [
3        "https://www.ikea.com/au/en/cat/quilt-cover-sets-10680/?page=2#products-page-2",
4        ...
5    ],
6    "markdown": "< Markdown of the page ... >",
7    "html": "< HTML of the page ... >"
8}

/search

Scrape and extract relevant Google search result URLs—no messing with headless browsers or geolocation issues.

1{
2    "endpoint": "search",
3    "google_domain": "www.google.com.au",
4    "query": "AVID POWER 20V MAX Lithium Ion Cordless Drill Set",
5    "page": 1
6}

Sample response:

1{
2    "search": [
3        "https://www.amazon.com/Cordless-Variable-Position-Masterworks-MW316/dp/B07CR1GPBQ",
4        "https://www.protoolreviews.com/avid-power-drill-review-is-an-amazon-drill-worth-it/",
5        ...
6    ],
7    "markdown": "< Markdown of the page ... >",
8    "html": "< HTML of the page ... >"
9}

⚙️ Advanced

The /scrape endpoint supports optional advanced techniques that enhance the output of the Web Scraping API.

Field-Level Instructions

You can not only specify which fields to return, but also how they’re generated or validated—embedding constraints and directives directly into the JSON.

1{
2    "endpoint": "scrape",
3    "url": "https://www.amazon.com.au/MSI-PRO-MP341CQW-UltraWide-Compatible/dp/B09Y19TRQ2",
4    "fields": {
5        "monitor_name": "< The product name of the monitor. >",
6        "brand": "< The brand or manufacturer name. >",
7        "display_size_in_inches": "< Numeric only. >",
8        "resolution": "< Example format: 1920x1080. >",
9        "panel_type": "< Type of panel. >",
10        "refresh_rate_hz": "< Numeric only. >",
11        "aspect_ratio": "< Example format: 16:9. >",
12        "ports": "< A comma-delimited list of available ports (e.g., HDMI, DisplayPort, etc.). >",
13        "features": "< Key selling points or capabilities, comma-delimited (e.g., LED, Full HD, etc.). >",
14        "price": "< Numeric price (integer or float). >",
15        "price_currency": "< Price currency (3 character alphabetic ISO 4217). >",
16        "review_count": "< Total number of customer reviews, numeric only. >",
17        "average_rating": "< Float or numeric star rating (e.g., 4.3). >",
18        "review_summary": "< A 50 words or less summary of all the written customer feedback. >"
19    }
20}

Sample response:

1{
2    "scrape": {
3        "monitor_name": "MSI PRO MP341CQW 34\" UltraWide Business Monitor",
4        "brand": "MSI",
5        "display_size_in_inches": 34,
6        "resolution": "3440x1440",
7        "panel_type": "VA",
8        "refresh_rate_hz": 100,
9        "aspect_ratio": "21:9",
10        "ports": "2xHDMI, DisplayPort",
11        "features": "1500R Curved, Frameless, 1ms, Tilt Compatible, HDR Ready, Built-in Speakers, VESA, 3 Year Warranty",
12        "price": 339,
13        "price_currency": "AUD",
14        "review_count": 391,
15        "average_rating": 4.5,
16        "review_summary": "The MSI PRO MP341CQW offers excellent image quality with a high refresh rate and curved design, making it ideal for both work and gaming. Users appreciate its vibrant colors and ergonomic features."
17    },
18    "markdown": "< Markdown of the page ... >",
19    "html": "< HTML of the page ... >"
20}

💡 Error Handling

If any required parameters are missing or an error occurs, the response looks like this:

1{
2    "error": true,
3    "reason": "Missing required parameters. Please check and try again with required parameters."
4}

No mysterious error codes—just clear reasons and suggestions.

🤝 Need Help?

Join our Discord Community for real-time help and feedback. We’re happy to assist with any questions, including:

• Customizing field extraction for your specific website targets.
• Optimizing usage for large-scale scraping projects.
• Integrating with Python, Node.js, or other environments.

From indie hackers to enterprise teams, we’re here to help you supercharge your data extraction. Get started with the Web Scraping API—and let us handle the heavy lifting of scraping for you.