HTTP Request Scraper — No Charge For Blocks
Pricing
from $0.85 / 1,000 datacenter fetches
HTTP Request Scraper — No Charge For Blocks
Send GET, POST, PUT, PATCH, DELETE, HEAD or OPTIONS to any public URL and get back the status, headers, cookies and full body. Runs through managed datacenter and residential proxies behind a real browser TLS fingerprint, rotating that identity on every retry. Blocked URLs are never charged.
Pricing
from $0.85 / 1,000 datacenter fetches
Rating
0.0
(0)
Developer
The Netaji
Maintained by CommunityActor stats
0
Bookmarked
5
Total users
5
Monthly active users
11 hours ago
Last modified
Categories
Share
HTTP Request Scraper
The Actor sends an HTTP request to each public http or https URL it is given and saves what came
back: the status code, the response headers, any cookies the target set, and the full response body
as text. One URL produces exactly one row, whether the request succeeded or not.
Requests are issued through managed proxies and a browser-grade TLS fingerprint, so targets that
refuse an ordinary scripted request will usually answer this one. A URL that is refused anyway is
still reported, as a row with ok: false and a reason, and it is not charged.
Accepted input
urls is required and accepts one or more public http or https URLs. They can be entered
individually, pasted as a list, or uploaded from a file. Every other field is optional.
method selects the HTTP method sent to every URL and accepts GET, POST, PUT, PATCH,
DELETE, HEAD, and OPTIONS; it defaults to GET. headers, params, and cookies each take
key/value pairs, applied to every URL in the run. params are appended to the query string of each
URL. body carries a request body of up to 10 MiB and applies to POST, PUT, and PATCH; it is
ignored on the methods that do not take one.
proxyTier accepts datacenter or residential and defaults to datacenter. country pins the
request to an uppercase ISO-2 country such as US, and is left unset by default, in which case any
country may serve the request.
escalate defaults to true. When a URL is refused, the Actor tries it again on residential, and
then once more on residential pinned to the requested country, stopping at the first attempt that
returns a response. Setting escalate to false runs exactly the tier that was requested and
nothing further.
browserProfile accepts auto, chrome, firefox, safari, or edge and defaults to auto.
On auto, successive attempts present different browser identities, which is what clears most
challenges; naming one pins every attempt to it instead.
sessionId is an optional label. URLs in a run that share a sessionId also share cookies and a
single exit address, which is what makes a multi-step sequence such as a sign-in followed by a
paginated list behave as one visitor. Left empty, every URL is fetched independently.
retries sets how many further attempts a single URL may take when refused, between 0 and 3,
and defaults to 2. timeoutMs bounds each attempt between 1000 and 60000 milliseconds and
defaults to 30000.
Response fields
url repeats the URL that was fetched, including any params that were appended to it. ok states
whether a response arrived at all, and status carries the HTTP status the target returned, or
null when nothing arrived. blocked states whether the target issued a challenge rather than an
answer.
body holds the response body as text and body_truncated states whether it reached the 10 MiB
ceiling and was cut short. headers holds the response headers, with repeated headers arriving as
an array, and cookies holds the cookies set on that response.
browser_profile names the browser identity that produced the response, which differs from the
first attempt whenever a retry rotated it. proxy_tier names the tier that produced it, which is
higher than the tier requested whenever escalation stepped up. attempts counts how many attempts
the URL took in total, duration_ms measures the successful one, and error gives the reason a
URL failed, or null when it succeeded.
Every field above is present on every row. A value that does not apply to a given row is returned as
null rather than omitted, so the column set does not vary between rows.
Charging
A row is charged when the target returned a response. Rows where ok is false are never charged,
and neither are rows carrying status 429, 444, 503, or 999, which are served by rate
limiters and edge protection rather than by the site itself.
A 404 or a 403 is charged, because both are the site answering the request that was made.
The rate depends on which proxy tier produced the response, since the two do not cost the same to
serve. A row with proxy_tier set to datacenter is charged at the datacenter rate, and a row set
to residential at the residential rate. A URL that started on datacenter and escalated is
charged at the rate of the tier that actually answered, which is the tier recorded on the row.
Every charged fetch includes 2 MB of response body. A body larger than that is charged per additional megabyte. Ordinary web pages do not reach the allowance: six representative pages measured on 2026-08-19, including Walmart search, Nike and Indeed, averaged 852 KB. The allowance matters only when the URLs point at large files rather than pages.
Behaviour on failure
A URL that cannot be fetched does not end the run and does not fail the Actor. It is saved as a row
with ok: false, status set to whatever was observed or null, and error naming the reason.
The run continues to the next URL.
Escalation applies only to refusals. A request rejected for its own contents, such as a malformed URL, is reported immediately without further attempts, since repeating it on another tier would produce the same rejection.
A run that is interrupted and resumed continues at the URL it stopped on. URLs already saved are not fetched again and are not charged again.
Questions
Why did a URL come back with ok: false when the site loads in a browser?
Some protection stacks require a full browser to execute a challenge before serving content. This
Actor issues HTTP requests, so a target that will not answer without executing client-side
JavaScript is reported as refused rather than returned empty. Raising retries or setting
proxyTier to residential clears many such targets; a few will not clear at all.
Why is attempts greater than 1 on a row that succeeded?
The first attempt was refused and a later one was not. proxy_tier and browser_profile on that
row describe the attempt that worked, not the first one, which is why they can differ from the
values requested.
How many URLs can one run take? There is no cap in the Actor. Each URL is one request and one row, so a run costs exactly as many charged rows as there were URLs that returned a response. URLs can be typed in, pasted in bulk, or uploaded as a file through the URL field.
What happens to a body larger than 10 MiB?
It is returned cut to 10 MiB with body_truncated set to true. The row is otherwise complete. It
is charged as a fetch, plus the megabytes above the 2 MB allowance.
Why did escalation stop part-way through a large run? When 25 URLs in a row are refused, the target is not going to yield to the next one, so escalation pauses and the remaining URLs are fetched on the requested tier alone. Every URL is still fetched and still saved; the run simply stops paying for retries that are not working. The first URL that answers restores escalation.
Can a sessionId be reused in a later run?
A session persists for a limited period after its last use and is then discarded. Reusing a label
after that gap starts a new session rather than resuming the old one, which is reported as
attempts beginning again from a fresh identity.
Related Actors
For extracting contact details from the pages rather than handling the responses directly, see Website Email Scraper, which crawls a site and returns the addresses it finds. For collecting the media links on a page, see Website Media Link Scraper.
