![SuperScraper API avatar](https://images.apifyusercontent.com/-X5yZtC5LyxtLzZaCdPnkaWK45pxFkDfWS7bPA57B0I/rs:fill:250:250/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vamJoMHpIb2wyNGk5VklUWXovbW1TZUZmVjZpR0hTS25zY2ctRnJhbWVfNzM0OC5wbmc.webp)
No credit card required
![SuperScraper API](https://images.apifyusercontent.com/-X5yZtC5LyxtLzZaCdPnkaWK45pxFkDfWS7bPA57B0I/rs:fill:250:250/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vamJoMHpIb2wyNGk5VklUWXovbW1TZUZmVjZpR0hTS25zY2ctRnJhbWVfNzM0OC5wbmc.webp)
SuperScraper API
No credit card required
Generic REST API for scraping websites: send a URL and get back HTML. This Actor is a drop-in replacement for ScrapingBee, ScrapingAnt, and ScraperAPI services. And it is open-source!
Do you want to learn more about this Actor?
Get a demoSuperScraper API is an Actor that provides a REST API for scraping websites. Just pass a URL of a web page and get back the fully-rendered HTML content. SuperScraper API is compatible with ScrapingBee, ScrapingAnt, and ScraperAPI. That means the Actor can be used as a potentially cheaper drop-in replacement for these services.
Main features:
- Extract HTML from arbitrary URLs with a headless browser for dynamic content rendering.
- Circumvent blocking using datacenter or residential proxies, as well as browser fingerprinting.
- Seamlessly scale to a large number of web pages as needed.
- Capture screenshots of the web pages.
Note that SuperScraper API uses the new experimental Actor Standby mode, so it's not started the traditional way from Apify Console. Instead, it's invoked via the HTTP REST API provided directly by the Actor. See the examples below.
Usage examples
To run these examples, you need an Apify API token, which you can find under Settings > Integrations in Apify Console.
You can create an Apify account free of charge.
Node.js
1import axios from 'axios'; 2 3const resp = await axios.get('https://super-scraper-api.apify.actor/', { 4 params: { 5 url: 'https://apify.com/store', 6 wait_for: '.ActorStoreItem-title', 7 json_response: true, 8 screenshot: true, 9 }, 10 headers: { 11 Authorization: 'Bearer <YOUR_APIFY_API_TOKEN>', 12 }, 13}); 14 15console.log(resp.data);
curl
1curl -X GET \ 2 'https://super-scraper-api.apify.actor/?url=https://apify.com/store&wait_for=.ActorStoreItem-title&screenshot=true&json_response=true' \ 3 --header 'Authorization: Bearer <YOUR_APIFY_API_TOKEN>'
Authentication
The best way to authenticate is to pass your Apify API token using the Authorization
HTTP header.
Alternatively, you can pass the API token via the token
query parameter to authenticate the requests, which is more convenient for testing in a web browser.
Node.js
1const resp = await axios.get('https://super-scraper-api.apify.actor/', { 2 params: { 3 url: 'https://apify.com/store', 4 token: '<YOUR_APIFY_API_TOKEN>' 5 }, 6});
curl
curl -X GET 'https://super-scraper-api.apify.actor/?url=https://apify.com/store&wait_for=.ActorStoreItem-title&json_response=true&token=<YOUR_APIFY_API_TOKEN>'
Pricing
When using SuperScraper API, you're charged based on your actual usage of the Apify platform's computing, storage, and networking resources.
Cost depends on the target sites, your settings and API parameters, the load of your requests, and random network and target site conditions.
From our testing, SuperScraper API is cheaper than ScrapingBee, ScrapingAnt, and ScraperAPI in many configurations, while in some others, it's more expensive.
The best way to see your price is to conduct a real-world test.
An example cost on a free account (the pricing is cheaper on higher plans) for 30 one-by-one requests plus 50 batched requests test:
parameters | cost estimate |
---|---|
no render_js + basic proxy | $1/1000 requests |
no render_js + premium (residential) proxy | $2/1000 requests |
render_js + basic proxy | $4/1000 requests |
render_js + premium (residential) proxy | $5/1000 requests |
API parameters
ScrapingBee API parameters
SuperScraper API supports most of the API parameters of ScrapingBee:
parameter | description |
---|---|
url | URL of the webpage to be scraped. This parameter is required. |
json_response | Return a verbose JSON response with additional details about the webpage. Can be either true or false , default is false . |
extract_rules | A stringified JSON containing custom rules how to extract data from the webpage. |
render_js | Indicates that the webpage should be scraped using a headless browser, with dynamic content rendered. Can be true or false , default is true . This is equivalent to ScrapingAnt's browser . |
screenshot | Get screenshot of the browser's current viewport. If json_response is set to true , screenshot will be returned in the Base64 encoding. Can be true or false , default is false . |
screenshot_full_page | Get screenshot of the full page. If json_response is set to true , screenshot will be returned in the Base64 encoding. Can be true or false , default is false . |
screenshot_selector | Get screenshot of the element specified by the selector. If json_response is set to true , screenshot will be returned in Base64. Must be a non-empty string. |
js_scenario | JavaScript instructions that will be executed after loading the webpage. |
wait | Specify a duration that the browser will wait after loading the page, in milliseconds. |
wait_for | Specify a CSS selector of an element for which the browser will wait after loading the page. |
wait_browser | Specify a browser event to wait for. Can be either load , domcontentloaded , or networkidle . |
block_resources | Specify that you want to block images and CSS. Can be true or false , default is true . |
window_width | Specify the width of the browser's viewport, in pixels. |
window_height | Specify the height of the browser's viewport, in pixels. |
cookies | Custom cookies to use to fetch the web pages. This is useful for fetching webpage behing login. The cookies must be specified in a string format: cookie_name_1=cookie_value1;cookie_name_2=cookie_value_2 . |
own_proxy | A custom proxy to be used for scraping, in the format <protocol><username>:<password>@<host>:<port> . |
premium_proxy | Use residential proxies to fetch the web content, in order to reduce the probability of being blocked. Can be either true or false , default is false . |
stealth_proxy | Works same as premium_proxy . |
country_code | Use IP addresses that are geolocated in the specified country by specifying its 2-letter ISO code. When using code other than US , premium_proxy must be set to true . This is equivalent to setting ScrapingAnt's proxy_country . |
custom_google | Use this option if you want to scrape Google-related websites (such as Google Searach or Google Shopping). Can be true or false , default is false . |
return_page_source | Return HTML of the webpage from the response before any dynamic JavaSript rendering. Can be true or false , default is false . |
transparent_status_code | By default, if target webpage responds with HTTP status code other than a 200-299 or a 404, the API will return a HTTP status code 500. Set this paremeter to true to disable this behavior and return the status code of the actual response. |
timeout | Set maximum timeout for the response from this Actor, in milliseconds. The default is 140 000 ms. |
forward_headers | If set to true , HTTP headers starting with prefix Spb- or Ant- will be forwarded to the target webpage alongside headers generated by us (the prefix will be trimmed). |
forward_headers_pure | If set to true , only headers starting with prefix Spb- or Ant- will be forwarded to the target webpage (prefix will be trimmed), without any other HTTP headers from our side. |
device | Can be either desktop (default) or mobile . |
ScrapingBee's API parameters block_ads
and session_id
are currently not supported.
ScrapingAnt API parameters
SuperScraper API supports most of the API parameters of ScrapingAnt:
parameter | description |
---|---|
url | URL of the webpage to be scraped. This parameter is required. |
browser | Indicates that the webpage should be scraped using a headless browser, with dynamic content rendered. Can be true or false , default is true . This is equivalent as ScrapingBee's render_js . |
cookies | Use custom cookies, must be in a string format: cookie_name_1=cookie_value1;cookie_name_2=cookie_value_2 . |
js_snippet | A Base64-encoded JavaScript code to be executed on the webpage. Will be treated as the evaluate instruction. |
proxy_type | Specify the type of proxies, which can be either datacenter (default) or residential . This is equivalent to setting ScrapingBee's premium_proxy or steath_proxy to true . |
wait_for_selector | Specify a CSS selector of an element for which the browser will wait after loading the page. This is equivalent to setting ScrapingBee's wait_for . |
block_resource | Specify one or more resources types you want to block from being downloaded. The parameter can be repeated in the URL (e.g. block_resource=image&block_resource=media ). Available options are: document , stylesheet , image , media , font , script , texttrack , xhr , fetch , eventsource , websocket , manifest , other . |
return_page_source | Return HTML of the webpage from the response before any dynamic JavaSript rendering. Can be true or false , default is false . |
proxy_country | Use IP addresses that are geolocated in the specified country by specifying its 2-letter ISO code. When using code other than US , premium_proxy must be set to true . This is equivalent to setting ScrapingBee's country_code . |
ScrapingAnt's API parameter x-api-key
is not supported.
Note that HTTP headers in a request to this Actor beginning with prefix Ant-
will be forwarded (without the prefix) to the target webpage alongside headers generated by the Actor.
This behavior can be changed using ScrapingBee's forward_headers
or forward_headers_pure
parameters.
ScraperAPI API parameters
SuperScraper API supports most of the API parameters of ScraperAPI:
parameter | description |
---|---|
url | URL of the webpage to be scraped. This parameter is required. |
render | Specify, if you want to scrape the webpage with or without using a headless browser, can be true or false , default true . (Same as render_js .) |
wait_for_selector | Specify a CSS selector of an element for which the browser will wait after loading the page. This is equivalent to setting ScrapingBee's wait_for . |
premium | Use residential proxies to fetch the web content, in order to reduce the probability of being blocked. Can be either true or false , default is false . This is equivalent to setting ScrapingBee's premium_proxy . |
ultra_premium | Same as premium . |
country_code | Use IP addresses that are geolocated in the specified country by specifying its 2-letter ISO code. When using code other than US , premium_proxy must be set to true . This is equivalent to setting ScrapingAnt's proxy_country . |
keep_headers | If true , then all headers sent to this Actor will be forwarded to the target website. The Authorization header will be removed. |
device_type | Can be either desktop (default) or mobile . This is equivalent to setting ScrapingBees's device . |
binary_target | Specify whether the target is a file. Can be true or false , default is false . Currently only supported when JS rendering is set to false via the render_js , browser , or render parameters. |
ScraperAPI's API parameters session_number
and autoparse
are currently not supported, and they are ignored.
Custom extraction rules
Using ScrapingBee's extract_rules
parameter, you can specify a set of rules to extract specific data from the target web pages. You can create an extraction rule in one of two ways: with shortened options, or with full options.
Shortened options
- value for the given key serves as a
selector
- using
@
, we can access attribute of the selected element
Example:
1{ 2 "title": "h1", 3 "link": "a@href" 4}
Full options
selector
is requiredtype
can be eitheritem
(default) orlist
output
indicates how the result for these element(s) will look like. It can be:text
(default option whenoutput
is omitted) - text of the elementhtml
- HTML of the element- attribute name (starts with
@
, for example@href
) - object with other extract rules for the given item (key + shortened or full options)
table_json
ortable_array
to scrape a table in a json or array format
clean
- relevant when havingtext
asoutput
, specifies whether the text of the element should be trimmed of whitespaces (can betrue
orfalse
, defaulttrue
)
Example:
1{ 2 "custom key for links": { 3 "selector": "a", 4 "type": "list", 5 "output": { 6 "linkName" : { 7 "selector": "a", 8 "clean": "false" 9 }, 10 "href": { 11 "selector": "a", 12 "output": "@href" 13 } 14 } 15 16 } 17}
Example
This example extracts all links from Apify Blog along with their titles.
1const extractRules = { 2 title: 'h1', 3 allLinks: { 4 selector: 'a', 5 type: 'list', 6 output: { 7 title: 'a', 8 link: 'a@href', 9 }, 10 }, 11}; 12 13const resp = await axios.get('https://super-scraper-api.apify.actor/', { 14 params: { 15 url: 'https://blog.apify.com/', 16 extract_rules: JSON.stringify(extractRules), 17 // verbose: true, 18 }, 19 headers: { 20 Authorization: 'Bearer <YOUR_APIFY_API_TOKEN>', 21 }, 22}); 23 24console.log(resp.data);
The results look like this:
1{ 2 "title": "Apify Blog", 3 "allLinks": [ 4 { 5 "title": "Data for generative AI & LLM", 6 "link": "https://apify.com/data-for-generative-ai" 7 }, 8 { 9 "title": "Product matching AI", 10 "link": "https://apify.com/product-matching-ai" 11 }, 12 { 13 "title": "Universal web scrapers", 14 "link": "https://apify.com/store/scrapers/universal-web-scrapers" 15 } 16 ] 17}
Custom JavaScript code
Use ScrapingBee's js_scenario
parameter to specify instructions in order to be executed one by one after opening the page.
Set json_response
to true
to get a full report of the executed instructions, the results of evaluate
instructions will be added to the evaluate_results
field.
Example of clicking a button:
1const instructions = { 2 instructions: [ 3 { click: '#button' }, 4 ], 5}; 6 7const resp = await axios.get('https://super-scraper-api.apify.actor/', { 8 params: { 9 url: 'https://www.example.com', 10 js_scenario: JSON.stringify(instructions), 11 }, 12 headers: { 13 Authorization: 'Bearer <YOUR_APIFY_API_TOKEN>', 14 }, 15}); 16 17console.log(resp.data);
Strict mode
If one instruction fails, then the subsequent instructions will not be executed. To disable this behavior, you can optionally set strict
to false
(by default it's true
):
1{ 2 "instructions": [ 3 { "click": "#button1" }, 4 { "click": "#button2" } 5 ], 6 "strict": false 7}
Supported instructions
wait
- wait for some time specified in ms
- example:
{"wait": 10000}
wait_for
- wait for an element specified by the selector
- example
{"wait_for": "#element"}
click
- click on an element specified by the selector
- example
{"click": "#button"}
wait_for_and_click
- combination of previous two
- example
{"wait_for_and_click": "#button"}
scroll_x
and scroll_y
- scroll a specified number of pixels horizontally or vertically
- example
{"scroll_y": 1000}
or{"scroll_x": 1000}
fill
- specify a selector of the input element and the value you want to fill
- example
{"fill": ["input_1", "value_1"]}
evaluate
- evaluate custom javascript on the webpage
- text/number/object results will be saved in the
evaluate_results
field - example
{"evaluate":"document.querySelectorAll('a').length"}
- 61 monthly users
- 24 stars
- 52.5% runs succeeded
- 35 days response time
- Created in May 2024
- Modified about 2 months ago