Flexible HTTP Request Runner

Run batches of public HTTP requests and export one structured record per request. Each record includes request identity, final URL, status, response headers, parsed or encoded body, timing, retry count, redirects, and a normalized error when no response arrives.

The Actor is designed for recurring public API polling, integration smoke tests, release checks, and no-code automation. It supports GET, HEAD, POST, PUT, PATCH, DELETE, and OPTIONS requests without requiring a custom deployment.

What can Flexible HTTP Request Runner do?

• Execute up to 1,000 public HTTP or HTTPS requests in one run.
• Add custom HTTP request headers and query parameters.
• Send JSON or text request bodies with write methods.
• Run requests concurrently while preserving their original output order.
• Retry network errors and selected transient HTTP status codes.
• Follow redirects and report the final URL and redirect count.
• Parse JSON responses automatically.
• Preserve text responses and base64-encode binary responses.
• Limit the body stored for each response and mark truncation explicitly.
• Return structured DNS, timeout, network, redirect, and security failures.
• Redact common secret headers, sensitive query values, and Set-Cookie output.
• Block private, local, reserved, documentation, and link-local destinations.

Who is this HTTP request Actor for?

Automation engineers can schedule a batch of endpoint checks and send the dataset to Make, Zapier, n8n, a webhook, or a data warehouse.

API developers can preserve reproducible status, header, body, and timing records during integration testing.

Data teams can poll public JSON endpoints on a schedule and consume a stable dataset rather than writing a one-off request script.

Operations teams can check multiple public services and distinguish HTTP errors from DNS, timeout, network, and blocked-destination failures.

This Actor is not a browser, login automation tool, private-network agent, or a replacement for an endpoint-specific scraper that understands pagination and business entities.

Why use it instead of a one-off script?

A command-line request is useful for a single check. Recurring automation also needs scheduling, storage, consistent records, retries, concurrency, secret-safe output, failure classification, and integrations. This Actor packages those behaviors behind one input schema and writes results to the default Apify dataset.

Unlike a generic browser, the Actor uses lightweight HTTP connections. It does not download page assets or render JavaScript. Unlike an endpoint-specific scraper, it does not reinterpret the response: users receive the actual status, headers, and body representation needed by downstream workflows.

Getting started

1. Open the Actor input page.
2. Add one or more entries to HTTP requests.
3. Give each request a unique id so downstream records remain easy to join.
4. Choose an HTTP method and optionally add headers, query parameters, or body.
5. Keep the default retry and concurrency settings for the first run.
6. Start the Actor.
7. Open Request results in the run dataset.
8. Schedule the Task or connect the dataset to another tool when the output is correct for your endpoint.

The prefilled input makes real GET requests to the public GitHub and npm APIs, so a first run produces useful JSON records without editing.

Input parameters

Field	Type	Default	Description
requests	array	required	One to 1,000 public HTTP/HTTPS request objects.
timeoutSecs	integer	30	Timeout for each attempt, from 1 to 120 seconds.
maxConcurrency	integer	5	Simultaneous requests, from 1 to 25.
maxRetries	integer	2	Retries after network errors or configured statuses, from 0 to 5.
retryOnStatusCodes	integer array	408, 425, 429, 500, 502, 503, 504	Responses that trigger backoff and retry.
followRedirects	boolean	true	Whether to follow supported HTTP redirects.
maxRedirects	integer	5	Redirect limit per request, from 0 to 10.
maxResponseBytes	integer	1000000	Maximum body bytes retained per record, from 1,000 to 5,000,000.

Each request supports:

Request field	Type	Description
id	string	Optional unique identity; defaults to request-1, request-2, and so on.
label	string	Optional human-readable label.
url	string	Required absolute public HTTP or HTTPS URL.
method	string	GET, HEAD, POST, PUT, PATCH, DELETE, or OPTIONS.
headers	object	Custom single-line request headers.
query	object	Query values; arrays create repeated query parameters.
body	JSON or string	Optional body for methods other than GET and HEAD.

Host, Content-Length, Connection, Transfer-Encoding, and Proxy-Authorization cannot be overridden. The Actor sets transport-critical values itself.

HTTP request example

{
  "requests": [
    {
      "id": "npm-apify-latest",
      "label": "Latest Apify npm package",
      "url": "https://registry.npmjs.org/apify/latest",
      "method": "GET"
    }
  ],
  "maxConcurrency": 1,
  "maxRetries": 2,
  "maxResponseBytes": 500000
}

A request with query parameters and headers looks like this:

{
  "id": "github-rate-limit",
  "url": "https://api.github.com/rate_limit",
  "method": "GET",
  "query": {
    "workflow": "scheduled-monitor"
  },
  "headers": {
    "accept": "application/vnd.github+json"
  }
}

Output data

One dataset row is written for every input request, including requests that do not receive an HTTP response.

Output field	Meaning
requestId, label, requestIndex	Identity and original position.
requestedUrl, finalUrl, method	Requested and final transport identity.
requestHeaders	User-supplied headers with secrets redacted.
statusCode, statusText, ok	Final HTTP response status.
responseHeaders	Response headers with Set-Cookie redacted.
body, bodyEncoding	Parsed JSON, text, or base64 binary representation.
bodyBytes, bodyTruncated	Body size and storage-limit status.
durationMs, attempts, redirectCount	Timing and recovery details.
errorType, errorMessage	Failure details when no response arrives.
fetchedAt	UTC completion timestamp.

Example from a real npm registry request, shortened for readability:

{
  "requestId": "npm-apify-latest",
  "label": "Latest Apify npm package",
  "requestIndex": 0,
  "requestedUrl": "https://registry.npmjs.org/apify/latest",
  "finalUrl": "https://registry.npmjs.org/apify/latest",
  "method": "GET",
  "requestHeaders": {},
  "statusCode": 200,
  "statusText": "OK",
  "ok": true,
  "responseHeaders": {
    "content-type": "application/json"
  },
  "body": {
    "name": "apify",
    "version": "3.7.2"
  },
  "bodyEncoding": "json",
  "bodyTruncated": false,
  "attempts": 1,
  "redirectCount": 0,
  "fetchedAt": "2026-08-15T12:00:00.000Z"
}

The live package version can change. The example illustrates the output shape, not a guaranteed current version.

How retries and redirects work

Network errors and status codes in retryOnStatusCodes use bounded exponential backoff with jitter. A valid server response remains available after the retry budget is exhausted, including its final status and body.

Redirects are followed manually. Every redirect destination is resolved and checked before connection. Authentication, cookies, proxy authorization, and API-key headers are removed when a redirect crosses to a different origin.

For 303, and for 301 or 302 after POST, the next request uses GET. 307 and 308 preserve the method and body.

Safe public-endpoint access

The Actor intentionally accepts only public HTTP and HTTPS destinations. It rejects localhost, loopback, private networks, link-local services, multicast, reserved ranges, and documentation-only ranges. DNS is checked before the socket connects, and the approved public IP is pinned for that connection. Redirects receive the same checks.

This boundary prevents using the Actor to access Apify container metadata, cloud metadata services, private corporate endpoints, or other internal resources. There is no option to disable it.

Common authentication headers can still be sent to a public endpoint. Their values are replaced with [REDACTED] in output. Sensitive query parameter names such as token, key, secret, auth, password, and signature are also redacted in URL output.

How much does it cost to run HTTP requests?

The Actor uses pay-per-event pricing:

• one start event for each run;
• one item event for each request that receives an HTTP response.

DNS errors, timeouts, blocked destinations, and other no-response failures still produce diagnostic dataset rows, but they do not incur an item event. HTTP 4xx and 5xx responses are charged because the status, headers, and body are a complete request result.

The one-time start price is $0.00005. HTTP response prices decrease through six monthly usage tiers:

Tier	Price per received HTTP response
FREE	$0.0009246
BRONZE	$0.000804
SILVER	$0.00062712
GOLD	$0.0004824
PLATINUM	$0.0003216
DIAMOND	$0.00022512

Apify shows the applicable tier and maximum total charge before execution. At the BRONZE rate, illustrative run totals are:

• 1 received response: $0.000854 including the start event;
• 25 received responses: $0.02015 including the start event;
• 100 received responses: $0.08045 including the start event.

A no-response diagnostic row does not add an item fee, so actual totals can be lower than the maximum. No proxy, residential traffic, or browser event is enabled, so there is no separate proxy charge event.

Scheduling recurring API polling

Save a working input as an Apify Task, then add an hourly, daily, or weekly schedule. Keep stable request IDs across runs so downstream systems can compare records by endpoint.

Useful recurring workflows include:

1. poll release metadata from several public package registries;
2. archive status and selected response headers;
3. compare the newest dataset with the previous run;
4. alert only when status, version, ETag, or body values change.

The Actor captures each run independently. It does not calculate changes or send alerts itself.

Integration ideas

• Make or Zapier: start a saved Task, wait for completion, then iterate over dataset items.
• n8n: call the Apify Actor endpoint and branch on ok, statusCode, or errorType.
• Webhooks: attach an Apify run webhook and fetch the dataset when the run succeeds.
• Google Sheets: export the overview view for a lightweight endpoint log.
• Data warehouses: ingest full dataset JSON for response-history analysis.
• AI agents: give an agent structured endpoint results without allowing it to connect to private networks.

Run with the Apify API

Replace APIFY_TOKEN with an Apify API token.

cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~flexible-http-request-runner/runs?token=APIFY_TOKEN&waitForFinish=120" \
  -H "Content-Type: application/json" \
  -d '{"requests":[{"id":"npm-apify","url":"https://registry.npmjs.org/apify/latest","method":"GET"}]}'

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/flexible-http-request-runner').call({
  requests: [
    {
      id: 'npm-apify',
      url: 'https://registry.npmjs.org/apify/latest',
      method: 'GET',
    },
  ],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/flexible-http-request-runner').call(run_input={
    'requests': [
        {
            'id': 'npm-apify',
            'url': 'https://registry.npmjs.org/apify/latest',
            'method': 'GET',
        }
    ]
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

Use with Apify MCP

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/flexible-http-request-runner"

Claude Desktop, Cursor, and VS Code setup

Claude Desktop, Cursor, and VS Code can use this HTTP MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/flexible-http-request-runner"
    }
  }
}

Example prompts:

• “Run the Flexible HTTP Request Runner against the latest Apify and Crawlee npm metadata endpoints, then summarize their versions.”
• “Check these public API URLs and return only endpoints with a non-2xx status.”
• “Run this saved endpoint batch and compare ETag response headers with my last dataset.”

Limits and failure behavior

• The Actor accepts at most 1,000 requests per run.
• Each request body is limited to 5,000,000 bytes.
• Stored response bodies are limited by maxResponseBytes.
• The Actor does not decompress compressed responses; it requests identity encoding, but a server that ignores that header can return binary content.
• JavaScript-rendered pages are not rendered.
• Endpoint pagination is not followed automatically.
• API authentication, quotas, permissions, and terms remain endpoint-specific.
• Some public APIs block cloud datacenter IP addresses.
• No Apify Proxy or residential fallback is provided.
• A failure for one request does not discard successful sibling results.
• Malformed Actor input fails the run before requests are executed.

Troubleshooting

Why did I receive blocked_url?

The URL or one of its redirects resolved to a non-public address. Use a public endpoint. Private networks, localhost, and cloud metadata services are blocked by design.

Why is bodyEncoding set to base64?

The response content type did not identify text or JSON. Decode the value from base64 downstream, or change the endpoint to return a textual content type.

Why is the body incomplete?

Check bodyTruncated. Increase maxResponseBytes up to 5,000,000 when the larger payload is necessary. For very large files, use a purpose-built download Actor instead of storing the entire payload in dataset records.

Why did a request retry?

Inspect attempts and the Actor log. Network errors and configured transient status codes trigger retries. Set maxRetries to 0 when each request must be attempted exactly once.

Why did the Actor return HTTP 401 or 403?

The remote server returned a valid response, so the record includes its status, headers, and body. Verify the endpoint's credentials and permissions. Secret header values are intentionally redacted in the output record.

Legality

Send requests only to endpoints you are authorized to access. Follow the endpoint's terms, robots guidance where applicable, rate limits, copyright rules, privacy obligations, and data-retention requirements.

Do not use high concurrency to overload services. Start with the default value, honor Retry-After, and reduce concurrency when an API returns rate-limit or capacity responses.

The Actor blocks private destinations but cannot determine whether every public URL or dataset is lawful for a particular use. Users remain responsible for the requests they configure and the data they store.

Related Automation Lab Actors

• HTTP Status Checker for focused website availability checks.
• Website Uptime Checker for uptime-oriented monitoring workflows.
• JSON Schema Generator for deriving a schema after collecting JSON API samples.

Choose this Actor when you need arbitrary methods, headers, bodies, and full response records. Choose a focused checker when you only need availability or a normalized domain-specific result.

FAQ

Does it support POST and custom HTTP request headers?

Yes. POST, PUT, PATCH, DELETE, and OPTIONS can send a JSON or text body. Custom headers are supported except transport-critical headers controlled by the Actor.

Can it call a private API?

No. Every destination must resolve only to public IP addresses. Use an agent inside your own network for private services.

Does it use a proxy?

No. Requests use direct Apify cloud egress. This keeps request identity and cost predictable but means an endpoint may reject datacenter traffic.

Are failed HTTP statuses included?

Yes. A 4xx or 5xx response is valuable structured output and includes available headers and body. A request that receives no response gets an error record.

Are results kept in input order?

Yes. Requests may execute concurrently, but dataset rows are pushed in the same order as the input array.

Can I store binary responses?

Yes, up to maxResponseBytes. Binary content is represented as base64 and identified by bodyEncoding: "base64".

Does the Actor publish my request or response data?

No. Results are written to the run's default dataset under your Apify account. Apply appropriate storage access and retention settings for your data.