# HTTP Request Scraper — No Charge For Blocks (`thenetaji/http-request-scraper`) Actor

Send GET, POST, PUT, PATCH, DELETE, HEAD or OPTIONS to any public URL and get back the status, headers, cookies and full body. Runs through managed datacenter and residential proxies behind a real browser TLS fingerprint, rotating that identity on every retry. Blocked URLs are never charged.

- **URL**: https://apify.com/thenetaji/http-request-scraper.md
- **Developed by:** [The Netaji](https://apify.com/thenetaji) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Stats:** 5 total users, 5 monthly users, 99.6% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.85 / 1,000 datacenter fetches

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## HTTP Request Scraper

The Actor sends an HTTP request to each public `http` or `https` URL it is given and saves what came
back: the status code, the response headers, any cookies the target set, and the full response body
as text. One URL produces exactly one row, whether the request succeeded or not.

Requests are issued through managed proxies and a browser-grade TLS fingerprint, so targets that
refuse an ordinary scripted request will usually answer this one. A URL that is refused anyway is
still reported, as a row with `ok: false` and a reason, and it is not charged.

```json
{
  "urls": [
    { "url": "https://example.com" },
    { "url": "https://www.walmart.com/search?q=laptop" }
  ],
  "method": "GET",
  "proxyTier": "datacenter",
  "escalate": true,
  "retries": 2
}
```

### Accepted input

`urls` is required and accepts one or more public `http` or `https` URLs. They can be entered
individually, pasted as a list, or uploaded from a file. Every other field is optional.

`method` selects the HTTP method sent to every URL and accepts `GET`, `POST`, `PUT`, `PATCH`,
`DELETE`, `HEAD`, and `OPTIONS`; it defaults to `GET`. `headers`, `params`, and `cookies` each take
key/value pairs, applied to every URL in the run. `params` are appended to the query string of each
URL. `body` carries a request body of up to 10 MiB and applies to `POST`, `PUT`, and `PATCH`; it is
ignored on the methods that do not take one.

`proxyTier` accepts `datacenter` or `residential` and defaults to `datacenter`. `country` pins the
request to an uppercase ISO-2 country such as `US`, and is left unset by default, in which case any
country may serve the request.

`escalate` defaults to `true`. When a URL is refused, the Actor tries it again on `residential`, and
then once more on `residential` pinned to the requested country, stopping at the first attempt that
returns a response. Setting `escalate` to `false` runs exactly the tier that was requested and
nothing further.

`browserProfile` accepts `auto`, `chrome`, `firefox`, `safari`, or `edge` and defaults to `auto`.
On `auto`, successive attempts present different browser identities, which is what clears most
challenges; naming one pins every attempt to it instead.

`sessionId` is an optional label. URLs in a run that share a `sessionId` also share cookies and a
single exit address, which is what makes a multi-step sequence such as a sign-in followed by a
paginated list behave as one visitor. Left empty, every URL is fetched independently.

`retries` sets how many further attempts a single URL may take when refused, between `0` and `3`,
and defaults to `2`. `timeoutMs` bounds each attempt between `1000` and `60000` milliseconds and
defaults to `30000`.

### Response fields

`url` repeats the URL that was fetched, including any `params` that were appended to it. `ok` states
whether a response arrived at all, and `status` carries the HTTP status the target returned, or
`null` when nothing arrived. `blocked` states whether the target issued a challenge rather than an
answer.

`body` holds the response body as text and `body_truncated` states whether it reached the 10 MiB
ceiling and was cut short. `headers` holds the response headers, with repeated headers arriving as
an array, and `cookies` holds the cookies set on that response.

`browser_profile` names the browser identity that produced the response, which differs from the
first attempt whenever a retry rotated it. `proxy_tier` names the tier that produced it, which is
higher than the tier requested whenever escalation stepped up. `attempts` counts how many attempts
the URL took in total, `duration_ms` measures the successful one, and `error` gives the reason a
URL failed, or `null` when it succeeded.

```json
{
  "url": "https://example.com",
  "ok": true,
  "status": 200,
  "blocked": false,
  "body": "<!doctype html><html>…</html>",
  "body_truncated": false,
  "headers": { "content-type": "text/html" },
  "cookies": {},
  "browser_profile": "chrome",
  "proxy_tier": "datacenter",
  "attempts": 1,
  "duration_ms": 777.4,
  "error": null
}
```

Every field above is present on every row. A value that does not apply to a given row is returned as
`null` rather than omitted, so the column set does not vary between rows.

### Charging

A row is charged when the target returned a response. Rows where `ok` is `false` are never charged,
and neither are rows carrying status `429`, `444`, `503`, or `999`, which are served by rate
limiters and edge protection rather than by the site itself.

A `404` or a `403` is charged, because both are the site answering the request that was made.

The rate depends on which proxy tier produced the response, since the two do not cost the same to
serve. A row with `proxy_tier` set to `datacenter` is charged at the datacenter rate, and a row set
to `residential` at the residential rate. A URL that started on `datacenter` and escalated is
charged at the rate of the tier that actually answered, which is the tier recorded on the row.

Every charged fetch includes 2 MB of response body. A body larger than that is charged per
additional megabyte. Ordinary web pages do not reach the allowance: six representative pages
measured on 2026-08-19, including Walmart search, Nike and Indeed, averaged 852 KB. The allowance
matters only when the URLs point at large files rather than pages.

### Behaviour on failure

A URL that cannot be fetched does not end the run and does not fail the Actor. It is saved as a row
with `ok: false`, `status` set to whatever was observed or `null`, and `error` naming the reason.
The run continues to the next URL.

Escalation applies only to refusals. A request rejected for its own contents, such as a malformed
URL, is reported immediately without further attempts, since repeating it on another tier would
produce the same rejection.

A run that is interrupted and resumed continues at the URL it stopped on. URLs already saved are not
fetched again and are not charged again.

### Questions

**Why did a URL come back with `ok: false` when the site loads in a browser?**
Some protection stacks require a full browser to execute a challenge before serving content. This
Actor issues HTTP requests, so a target that will not answer without executing client-side
JavaScript is reported as refused rather than returned empty. Raising `retries` or setting
`proxyTier` to `residential` clears many such targets; a few will not clear at all.

**Why is `attempts` greater than `1` on a row that succeeded?**
The first attempt was refused and a later one was not. `proxy_tier` and `browser_profile` on that
row describe the attempt that worked, not the first one, which is why they can differ from the
values requested.

**How many URLs can one run take?**
There is no cap in the Actor. Each URL is one request and one row, so a run costs exactly as many
charged rows as there were URLs that returned a response. URLs can be typed in, pasted in bulk, or
uploaded as a file through the URL field.

**What happens to a body larger than 10 MiB?**
It is returned cut to 10 MiB with `body_truncated` set to `true`. The row is otherwise complete. It
is charged as a fetch, plus the megabytes above the 2 MB allowance.

**Why did escalation stop part-way through a large run?**
When 25 URLs in a row are refused, the target is not going to yield to the next one, so escalation
pauses and the remaining URLs are fetched on the requested tier alone. Every URL is still fetched
and still saved; the run simply stops paying for retries that are not working. The first URL that
answers restores escalation.

**Can a `sessionId` be reused in a later run?**
A session persists for a limited period after its last use and is then discarded. Reusing a label
after that gap starts a new session rather than resuming the old one, which is reported as
`attempts` beginning again from a fresh identity.

### Related Actors

For extracting contact details from the pages rather than handling the responses directly, see
[Website Email Scraper](https://apify.com/thenetaji/website-email-scraper), which crawls a site and
returns the addresses it finds. For collecting the media links on a page, see
[Website Media Link Scraper](https://apify.com/thenetaji/website-media-link-scraper).

# Actor input Schema

## `urls` (type: `array`):

One or more public http(s) URLs to fetch. Each becomes one row.

## `method` (type: `string`):

HTTP method sent to every URL. Bodies are only allowed on POST, PUT and PATCH.

## `headers` (type: `object`):

Request headers sent to the target, as key/value pairs.

## `params` (type: `object`):

Appended to every URL's query string.

## `cookies` (type: `object`):

Cookies sent with the request, as key/value pairs.

## `body` (type: `string`):

Body string for POST, PUT and PATCH. Up to 10 MiB.

## `proxyTier` (type: `string`):

Datacenter is faster and costs less per URL. Residential clears the harder targets and is priced higher, because it costs more to serve.

## `escalate` (type: `boolean`):

On a block, retry the URL once on residential (and again pinned to the chosen country). A URL that succeeds this way is charged at the residential rate. Turn this off to run exactly the tier you picked, at that tier's price, and nothing else.

## `country` (type: `string`):

Uppercase ISO-2 proxy country, e.g. US. Leave empty for any.

## `browserProfile` (type: `string`):

Which browser to impersonate. Leave on auto so retries rotate through the profiles, which is what clears most challenges.

## `sessionId` (type: `string`):

Reuse cookies and a pinned proxy exit across every URL in this run, for multi-step flows such as a login followed by a paginated list. Leave empty for independent stateless requests.

## `retries` (type: `integer`):

Extra attempts per URL when blocked. Each retry changes identity.

## `timeoutMs` (type: `integer`):

Per-attempt timeout in milliseconds.

## Actor input object example

```json
{
  "urls": [
    {
      "url": "https://example.com"
    }
  ],
  "method": "GET",
  "headers": {
    "Accept": "text/html"
  },
  "params": {
    "page": "1"
  },
  "cookies": {
    "consent": "yes"
  },
  "body": "{\"query\":\"toolzer\"}",
  "proxyTier": "datacenter",
  "escalate": true,
  "country": "US",
  "browserProfile": "auto",
  "sessionId": "cart-42",
  "retries": 2,
  "timeoutMs": 30000
}
```

# Actor output Schema

## `dataset` (type: `string`):

All records scraped by this run

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        {
            "url": "https://example.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("thenetaji/http-request-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [{ "url": "https://example.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("thenetaji/http-request-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    {
      "url": "https://example.com"
    }
  ]
}' |
apify call thenetaji/http-request-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,thenetaji/http-request-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/BooZkbIliJ6wxtbe8/builds/h4LPrCfVK9mJq9ilQ/openapi.json