HTTP Request Runner: bulk API calls and webhooks

Send one HTTP request or a few thousand from a single run and get every response back as a row: status, headers, body, parsed JSON and timing. It handles GET, POST, PUT, PATCH, DELETE, HEAD and OPTIONS, with your own headers and a JSON, text or form body on each request.

People use it to call an API on a schedule, fire webhooks from an Apify task, test endpoints after a deploy, or pull data from a JSON API into a dataset without writing code.

Input

Each request is a JSON object in the Requests list:

[
  {"label": "list users", "url": "https://api.example.com/users?page=1",
   "headers": {"Accept": "application/json"}},
  {"label": "create order", "method": "POST", "url": "https://api.example.com/orders",
   "headers": {"X-Api-Key": "{{secrets.SHOP_KEY}}"}, "json": {"sku": "A-100", "qty": 2}}
]

Only url is required. A request can also carry method, headers, one body (json, body for raw text, or form for a URL encoded form), a label of your own and its own timeoutSecs. An address without http:// or https:// gets https://.

For plain GET calls you can paste URLs, one per line, in the URLs field instead. They use the default method and default headers.

Other settings:

• Default method (GET) and default headers sent with every request. A request's own headers win where the names match.
• Follow redirects (on) and the maximum number of redirects (10). A 301, 302 or 303 after a POST is repeated as a GET without the body, as a browser does. A 307 or 308 keeps the method and body.
• Maximum body characters (100,000). Longer bodies are cut and marked bodyTruncated.
• Include response headers (on)
• Timeout per request in seconds (30). It covers the whole request, body included.
• Retries (2)
• Parallel requests in total (10) and per host (4), and a delay between requests to one host (0 ms)

Tokens and keys

Two fields keep credentials out of the request list. Both are stored encrypted by Apify.

Secrets takes NAME=value lines. Write {{secrets.NAME}} in a URL, header, body, JSON or form value and the run puts the real value in just before sending. That works for keys that have to go in a query string too.

Secret headers takes Name: value lines, such as Authorization: Bearer ..., and adds them to every request. Fill in the hosts list if they should go to some hosts only.

Secret values are never written to the log. Wherever one comes back in a result (an API that echoes your headers, a query string in an error), it shows as (secret). Request rows keep the URL as you wrote it, placeholders and all. When a redirect leads to another host, the secret headers, Authorization and Cookie are dropped before the next hop. Each request runs on its own, and cookies from one response are never sent with another request.

Output

One dataset row per request, in the order the requests finish, with index giving its place in your input. A row with an answer has:

• status, statusText and ok (true for 2xx)
• body as text, and json when the answer is JSON and fits whole within the character limit
• bodyBase64 in place of body for images, PDFs and other binary answers
• contentType, bodyBytes, bodyTruncated, httpVersion and responseHeaders
• timeMs, attempts, sentAt, plus finalUrl, redirectCount and redirectChain when it was redirected

A 404 or 500 is still an answer: you get the row with its status and body.

A request with no answer gets a row with responded: false, an error and an errorType, and the run moves on to the next request. The types are dnsError, timeout, tlsError, connectionError, tooManyRedirects, blockedAddress and invalidInput (a bad URL or method, two bodies on one request, or a placeholder with no matching secret).

The OUTPUT record in the key value store holds the totals for the run.

Rate limits and retries

A request that gets 429 is sent again after the wait the server asks for in Retry-After, up to 60 seconds, or after a short backoff when there is no header. A 502, 503 or 504 is retried the same way for GET, HEAD, OPTIONS, PUT and DELETE only. POST and PATCH are never repeated after the server answered, so a webhook fires once. A connection that times out before it opens is retried for every method, since nothing was sent.

For an API that allows one call per second, set the delay to 1000 ms. In one test, three calls to one host with a 1500 ms delay went out 1.5 seconds apart.

Blocked addresses

Hosts that resolve to private, loopback, link local or reserved addresses are refused, and so are localhost names. Each redirect hop is checked the same way, so a public URL that redirects to 127.0.0.1 stops with a blockedAddress row.

Pricing

Pay per event: one event for each request that gets an answer from the server, whatever the status code. Requests that fail to connect, time out or are refused cost nothing.

Before each request goes out, the run checks that the event can still be paid for within the maximum charge you set. Once it cannot, no more requests are sent, the run finishes cleanly, and OUTPUT shows stoppedAtChargeLimit: true with the number of requests not sent.

Scope and conduct

It sends only the requests you list, through no proxy. Call only APIs and endpoints you are allowed to use, within their terms and rate limits.

Support

Questions and problems: the Issues tab on this page.