# Flexible HTTP Request Runner (`automation-lab/flexible-http-request-runner`) Actor

Execute batches of public HTTP requests with custom methods, headers, query parameters, bodies, retries, and structured status, response header, body, timing, redirect, and error records.

- **URL**: https://apify.com/automation-lab/flexible-http-request-runner.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** Developer tools
- **Stats:** 8 total users, 6 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Flexible HTTP Request Runner

Run batches of public HTTP requests and export one structured record per request.
Each record includes request identity, final URL, status, response headers, parsed
or encoded body, timing, retry count, redirects, and a normalized error when no
response arrives.

The Actor is designed for recurring public API polling, integration smoke tests,
release checks, and no-code automation. It supports GET, HEAD, POST, PUT, PATCH,
DELETE, and OPTIONS requests without requiring a custom deployment.

### What can Flexible HTTP Request Runner do?

- Execute up to 1,000 public HTTP or HTTPS requests in one run.
- Add custom HTTP request headers and query parameters.
- Send JSON or text request bodies with write methods.
- Run requests concurrently while preserving their original output order.
- Retry network errors and selected transient HTTP status codes.
- Follow redirects and report the final URL and redirect count.
- Parse JSON responses automatically.
- Preserve text responses and base64-encode binary responses.
- Limit the body stored for each response and mark truncation explicitly.
- Return structured DNS, timeout, network, redirect, and security failures.
- Redact common secret headers, sensitive query values, and `Set-Cookie` output.
- Block private, local, reserved, documentation, and link-local destinations.

### Who is this HTTP request Actor for?

**Automation engineers** can schedule a batch of endpoint checks and send the
dataset to Make, Zapier, n8n, a webhook, or a data warehouse.

**API developers** can preserve reproducible status, header, body, and timing
records during integration testing.

**Data teams** can poll public JSON endpoints on a schedule and consume a stable
dataset rather than writing a one-off request script.

**Operations teams** can check multiple public services and distinguish HTTP
errors from DNS, timeout, network, and blocked-destination failures.

This Actor is not a browser, login automation tool, private-network agent, or a
replacement for an endpoint-specific scraper that understands pagination and
business entities.

### Why use it instead of a one-off script?

A command-line request is useful for a single check. Recurring automation also
needs scheduling, storage, consistent records, retries, concurrency, secret-safe
output, failure classification, and integrations. This Actor packages those
behaviors behind one input schema and writes results to the default Apify
dataset.

Unlike a generic browser, the Actor uses lightweight HTTP connections. It does
not download page assets or render JavaScript. Unlike an endpoint-specific
scraper, it does not reinterpret the response: users receive the actual status,
headers, and body representation needed by downstream workflows.

### Getting started

1. Open the Actor input page.
2. Add one or more entries to **HTTP requests**.
3. Give each request a unique `id` so downstream records remain easy to join.
4. Choose an HTTP method and optionally add headers, query parameters, or body.
5. Keep the default retry and concurrency settings for the first run.
6. Start the Actor.
7. Open **Request results** in the run dataset.
8. Schedule the Task or connect the dataset to another tool when the output is
   correct for your endpoint.

The prefilled input makes real GET requests to the public GitHub and npm APIs,
so a first run produces useful JSON records without editing.

### Input parameters

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `requests` | array | required | One to 1,000 public HTTP/HTTPS request objects. |
| `timeoutSecs` | integer | `30` | Timeout for each attempt, from 1 to 120 seconds. |
| `maxConcurrency` | integer | `5` | Simultaneous requests, from 1 to 25. |
| `maxRetries` | integer | `2` | Retries after network errors or configured statuses, from 0 to 5. |
| `retryOnStatusCodes` | integer array | `408, 425, 429, 500, 502, 503, 504` | Responses that trigger backoff and retry. |
| `followRedirects` | boolean | `true` | Whether to follow supported HTTP redirects. |
| `maxRedirects` | integer | `5` | Redirect limit per request, from 0 to 10. |
| `maxResponseBytes` | integer | `1000000` | Maximum body bytes retained per record, from 1,000 to 5,000,000. |

Each request supports:

| Request field | Type | Description |
| --- | --- | --- |
| `id` | string | Optional unique identity; defaults to `request-1`, `request-2`, and so on. |
| `label` | string | Optional human-readable label. |
| `url` | string | Required absolute public HTTP or HTTPS URL. |
| `method` | string | GET, HEAD, POST, PUT, PATCH, DELETE, or OPTIONS. |
| `headers` | object | Custom single-line request headers. |
| `query` | object | Query values; arrays create repeated query parameters. |
| `body` | JSON or string | Optional body for methods other than GET and HEAD. |

`Host`, `Content-Length`, `Connection`, `Transfer-Encoding`, and
`Proxy-Authorization` cannot be overridden. The Actor sets transport-critical
values itself.

### HTTP request example

```json
{
  "requests": [
    {
      "id": "npm-apify-latest",
      "label": "Latest Apify npm package",
      "url": "https://registry.npmjs.org/apify/latest",
      "method": "GET"
    }
  ],
  "maxConcurrency": 1,
  "maxRetries": 2,
  "maxResponseBytes": 500000
}
```

A request with query parameters and headers looks like this:

```json
{
  "id": "github-rate-limit",
  "url": "https://api.github.com/rate_limit",
  "method": "GET",
  "query": {
    "workflow": "scheduled-monitor"
  },
  "headers": {
    "accept": "application/vnd.github+json"
  }
}
```

### Output data

One dataset row is written for every input request, including requests that do
not receive an HTTP response.

| Output field | Meaning |
| --- | --- |
| `requestId`, `label`, `requestIndex` | Identity and original position. |
| `requestedUrl`, `finalUrl`, `method` | Requested and final transport identity. |
| `requestHeaders` | User-supplied headers with secrets redacted. |
| `statusCode`, `statusText`, `ok` | Final HTTP response status. |
| `responseHeaders` | Response headers with `Set-Cookie` redacted. |
| `body`, `bodyEncoding` | Parsed JSON, text, or base64 binary representation. |
| `bodyBytes`, `bodyTruncated` | Body size and storage-limit status. |
| `durationMs`, `attempts`, `redirectCount` | Timing and recovery details. |
| `errorType`, `errorMessage` | Failure details when no response arrives. |
| `fetchedAt` | UTC completion timestamp. |

Example from a real npm registry request, shortened for readability:

```json
{
  "requestId": "npm-apify-latest",
  "label": "Latest Apify npm package",
  "requestIndex": 0,
  "requestedUrl": "https://registry.npmjs.org/apify/latest",
  "finalUrl": "https://registry.npmjs.org/apify/latest",
  "method": "GET",
  "requestHeaders": {},
  "statusCode": 200,
  "statusText": "OK",
  "ok": true,
  "responseHeaders": {
    "content-type": "application/json"
  },
  "body": {
    "name": "apify",
    "version": "3.7.2"
  },
  "bodyEncoding": "json",
  "bodyTruncated": false,
  "attempts": 1,
  "redirectCount": 0,
  "fetchedAt": "2026-08-15T12:00:00.000Z"
}
```

The live package version can change. The example illustrates the output shape,
not a guaranteed current version.

### How retries and redirects work

Network errors and status codes in `retryOnStatusCodes` use bounded exponential
backoff with jitter. A valid server response remains available after the retry
budget is exhausted, including its final status and body.

Redirects are followed manually. Every redirect destination is resolved and
checked before connection. Authentication, cookies, proxy authorization, and
API-key headers are removed when a redirect crosses to a different origin.

For `303`, and for `301` or `302` after POST, the next request uses GET. `307`
and `308` preserve the method and body.

### Safe public-endpoint access

The Actor intentionally accepts only public HTTP and HTTPS destinations. It
rejects localhost, loopback, private networks, link-local services, multicast,
reserved ranges, and documentation-only ranges. DNS is checked before the
socket connects, and the approved public IP is pinned for that connection.
Redirects receive the same checks.

This boundary prevents using the Actor to access Apify container metadata,
cloud metadata services, private corporate endpoints, or other internal
resources. There is no option to disable it.

Common authentication headers can still be sent to a public endpoint. Their
values are replaced with `[REDACTED]` in output. Sensitive query parameter names
such as `token`, `key`, `secret`, `auth`, `password`, and `signature` are also
redacted in URL output.

### How much does it cost to run HTTP requests?

The Actor uses pay-per-event pricing:

- one `start` event for each run;
- one `item` event for each request that receives an HTTP response.

DNS errors, timeouts, blocked destinations, and other no-response failures still
produce diagnostic dataset rows, but they do not incur an `item` event. HTTP
4xx and 5xx responses are charged because the status, headers, and body are a
complete request result.

The one-time start price is **$0.00005**. HTTP response prices decrease through
six monthly usage tiers:

| Tier | Price per received HTTP response |
| --- | ---: |
| FREE | $0.0009246 |
| BRONZE | $0.000804 |
| SILVER | $0.00062712 |
| GOLD | $0.0004824 |
| PLATINUM | $0.0003216 |
| DIAMOND | $0.00022512 |

Apify shows the applicable tier and maximum total charge before execution. At
the BRONZE rate, illustrative run totals are:

- 1 received response: **$0.000854** including the start event;
- 25 received responses: **$0.02015** including the start event;
- 100 received responses: **$0.08045** including the start event.

A no-response diagnostic row does not add an item fee, so actual totals can be
lower than the maximum. No proxy, residential traffic, or browser event is
enabled, so there is no separate proxy charge event.

### Scheduling recurring API polling

Save a working input as an Apify Task, then add an hourly, daily, or weekly
schedule. Keep stable request IDs across runs so downstream systems can compare
records by endpoint.

Useful recurring workflows include:

1. poll release metadata from several public package registries;
2. archive status and selected response headers;
3. compare the newest dataset with the previous run;
4. alert only when status, version, ETag, or body values change.

The Actor captures each run independently. It does not calculate changes or
send alerts itself.

### Integration ideas

- **Make or Zapier:** start a saved Task, wait for completion, then iterate over
  dataset items.
- **n8n:** call the Apify Actor endpoint and branch on `ok`, `statusCode`, or
  `errorType`.
- **Webhooks:** attach an Apify run webhook and fetch the dataset when the run
  succeeds.
- **Google Sheets:** export the overview view for a lightweight endpoint log.
- **Data warehouses:** ingest full dataset JSON for response-history analysis.
- **AI agents:** give an agent structured endpoint results without allowing it
  to connect to private networks.

### Run with the Apify API

Replace `APIFY_TOKEN` with an Apify API token.

#### cURL

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~flexible-http-request-runner/runs?token=APIFY_TOKEN&waitForFinish=120" \
  -H "Content-Type: application/json" \
  -d '{"requests":[{"id":"npm-apify","url":"https://registry.npmjs.org/apify/latest","method":"GET"}]}'
```

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/flexible-http-request-runner').call({
  requests: [
    {
      id: 'npm-apify',
      url: 'https://registry.npmjs.org/apify/latest',
      method: 'GET',
    },
  ],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

#### Python

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/flexible-http-request-runner').call(run_input={
    'requests': [
        {
            'id': 'npm-apify',
            'url': 'https://registry.npmjs.org/apify/latest',
            'method': 'GET',
        }
    ]
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)
```

### Use with Apify MCP

Add the Actor to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/flexible-http-request-runner"
```

#### Claude Desktop, Cursor, and VS Code setup

Claude Desktop, Cursor, and VS Code can use this HTTP MCP configuration:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/flexible-http-request-runner"
    }
  }
}
```

Example prompts:

- “Run the Flexible HTTP Request Runner against the latest Apify and Crawlee npm
  metadata endpoints, then summarize their versions.”
- “Check these public API URLs and return only endpoints with a non-2xx status.”
- “Run this saved endpoint batch and compare ETag response headers with my last
  dataset.”

### Limits and failure behavior

- The Actor accepts at most 1,000 requests per run.
- Each request body is limited to 5,000,000 bytes.
- Stored response bodies are limited by `maxResponseBytes`.
- The Actor does not decompress compressed responses; it requests identity
  encoding, but a server that ignores that header can return binary content.
- JavaScript-rendered pages are not rendered.
- Endpoint pagination is not followed automatically.
- API authentication, quotas, permissions, and terms remain endpoint-specific.
- Some public APIs block cloud datacenter IP addresses.
- No Apify Proxy or residential fallback is provided.
- A failure for one request does not discard successful sibling results.
- Malformed Actor input fails the run before requests are executed.

### Troubleshooting

#### Why did I receive `blocked_url`?

The URL or one of its redirects resolved to a non-public address. Use a public
endpoint. Private networks, localhost, and cloud metadata services are blocked
by design.

#### Why is `bodyEncoding` set to `base64`?

The response content type did not identify text or JSON. Decode the value from
base64 downstream, or change the endpoint to return a textual content type.

#### Why is the body incomplete?

Check `bodyTruncated`. Increase `maxResponseBytes` up to 5,000,000 when the
larger payload is necessary. For very large files, use a purpose-built download
Actor instead of storing the entire payload in dataset records.

#### Why did a request retry?

Inspect `attempts` and the Actor log. Network errors and configured transient
status codes trigger retries. Set `maxRetries` to `0` when each request must be
attempted exactly once.

#### Why did the Actor return HTTP 401 or 403?

The remote server returned a valid response, so the record includes its status,
headers, and body. Verify the endpoint's credentials and permissions. Secret
header values are intentionally redacted in the output record.

### Legality

Send requests only to endpoints you are authorized to access. Follow the
endpoint's terms, robots guidance where applicable, rate limits, copyright
rules, privacy obligations, and data-retention requirements.

Do not use high concurrency to overload services. Start with the default value,
honor `Retry-After`, and reduce concurrency when an API returns rate-limit or
capacity responses.

The Actor blocks private destinations but cannot determine whether every public
URL or dataset is lawful for a particular use. Users remain responsible for the
requests they configure and the data they store.

### Related Automation Lab Actors

- [HTTP Status Checker](https://apify.com/automation-lab/http-status-checker) for
  focused website availability checks.
- [Website Uptime Checker](https://apify.com/automation-lab/website-uptime-checker)
  for uptime-oriented monitoring workflows.
- [JSON Schema Generator](https://apify.com/automation-lab/json-schema-generator)
  for deriving a schema after collecting JSON API samples.

Choose this Actor when you need arbitrary methods, headers, bodies, and full
response records. Choose a focused checker when you only need availability or a
normalized domain-specific result.

### FAQ

#### Does it support POST and custom HTTP request headers?

Yes. POST, PUT, PATCH, DELETE, and OPTIONS can send a JSON or text body. Custom
headers are supported except transport-critical headers controlled by the Actor.

#### Can it call a private API?

No. Every destination must resolve only to public IP addresses. Use an agent
inside your own network for private services.

#### Does it use a proxy?

No. Requests use direct Apify cloud egress. This keeps request identity and cost
predictable but means an endpoint may reject datacenter traffic.

#### Are failed HTTP statuses included?

Yes. A 4xx or 5xx response is valuable structured output and includes available
headers and body. A request that receives no response gets an error record.

#### Are results kept in input order?

Yes. Requests may execute concurrently, but dataset rows are pushed in the same
order as the input array.

#### Can I store binary responses?

Yes, up to `maxResponseBytes`. Binary content is represented as base64 and
identified by `bodyEncoding: "base64"`.

#### Does the Actor publish my request or response data?

No. Results are written to the run's default dataset under your Apify account.
Apply appropriate storage access and retention settings for your data.

# Actor input Schema

## `requests` (type: `array`):

Requests to public endpoints. Supports GET, HEAD, POST, PUT, PATCH, DELETE, and OPTIONS with custom headers, query parameters, and JSON or text bodies.

## `timeoutSecs` (type: `integer`):

Maximum time for one request attempt.

## `maxConcurrency` (type: `integer`):

Maximum requests executed at the same time.

## `maxRetries` (type: `integer`):

Retries per request after network errors or configured transient status codes.

## `retryOnStatusCodes` (type: `array`):

HTTP status codes that trigger bounded exponential backoff.

## `followRedirects` (type: `boolean`):

Follow HTTP redirects while revalidating every destination as public.

## `maxRedirects` (type: `integer`):

Maximum redirects followed for each request.

## `maxResponseBytes` (type: `integer`):

Maximum response-body bytes retained per request. Larger bodies are safely truncated and marked.

## Actor input object example

```json
{
  "requests": [
    {
      "id": "github-repository",
      "label": "Apify SDK repository metadata",
      "url": "https://api.github.com/repos/apify/apify-sdk-js",
      "method": "GET",
      "headers": {
        "accept": "application/vnd.github+json"
      }
    },
    {
      "id": "npm-package",
      "label": "Latest Apify package metadata",
      "url": "https://registry.npmjs.org/apify/latest",
      "method": "GET"
    }
  ],
  "timeoutSecs": 30,
  "maxConcurrency": 5,
  "maxRetries": 2,
  "retryOnStatusCodes": [
    408,
    425,
    429,
    500,
    502,
    503,
    504
  ],
  "followRedirects": true,
  "maxRedirects": 5,
  "maxResponseBytes": 1000000
}
```

# Actor output Schema

## `dataset` (type: `string`):

Structured status, headers, body, timing, redirect, retry, and error data.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "requests": [
        {
            "id": "github-repository",
            "label": "Apify SDK repository metadata",
            "url": "https://api.github.com/repos/apify/apify-sdk-js",
            "method": "GET",
            "headers": {
                "accept": "application/vnd.github+json"
            }
        },
        {
            "id": "npm-package",
            "label": "Latest Apify package metadata",
            "url": "https://registry.npmjs.org/apify/latest",
            "method": "GET"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/flexible-http-request-runner").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "requests": [
        {
            "id": "github-repository",
            "label": "Apify SDK repository metadata",
            "url": "https://api.github.com/repos/apify/apify-sdk-js",
            "method": "GET",
            "headers": { "accept": "application/vnd.github+json" },
        },
        {
            "id": "npm-package",
            "label": "Latest Apify package metadata",
            "url": "https://registry.npmjs.org/apify/latest",
            "method": "GET",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/flexible-http-request-runner").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "requests": [
    {
      "id": "github-repository",
      "label": "Apify SDK repository metadata",
      "url": "https://api.github.com/repos/apify/apify-sdk-js",
      "method": "GET",
      "headers": {
        "accept": "application/vnd.github+json"
      }
    },
    {
      "id": "npm-package",
      "label": "Latest Apify package metadata",
      "url": "https://registry.npmjs.org/apify/latest",
      "method": "GET"
    }
  ]
}' |
apify call automation-lab/flexible-http-request-runner --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/flexible-http-request-runner"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/37tfQ0y4KmVGLTxqn/builds/ghZx5dqJplS9IXd36/openapi.json