# Bulk URL Status & Redirect Chain Checker (`madpai/bulk-url-status-checker`) Actor

Check a list of URLs in bulk: final HTTP status, the full redirect chain, redirect loops, likely soft-404s, response time and content type. No crawling, no login, no API key. Polite: robots.txt-aware, about 1 request per second per host. Clean JSON for SEO migrations and AI agents.

- **URL**: https://apify.com/madpai/bulk-url-status-checker.md
- **Developed by:** [William Schultz](https://apify.com/madpai) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.80 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Bulk URL Status & Redirect Chain Checker

### What does it do?

Paste a list of URLs and get one clean result per URL: the final HTTP status, the full redirect chain (every hop, each marked permanent or temporary), redirect loops, the final URL, a likely soft-404 flag, the response time and the content type. It checks only the URLs you give it and never crawls. No API key, no login, no proxies.

### Who is it for?

SEO teams checking redirects after a site migration, people cleaning up link lists, backlink or sitemap exports, and AI agents that need to know whether a set of URLs still works and where they end up.

### Input

| Field | Default | What it does |
|---|---|---|
| `urls` (required) | prefill: 10 sample URLs | One URL per line. Bare domains (`example.com`) are checked as `https://example.com/`. Duplicates are removed after normalisation (`#fragments` dropped). |
| `maxRedirects` | 10 | Redirect hops followed per URL before it counts as a loop. |
| `method` | `HEAD-then-GET` | `HEAD-then-GET` sends a light HEAD request per hop and falls back to GET when a server refuses HEAD (405, 403 or 501). `GET` uses GET on every hop and reads the first 16 KB of 2xx HTML pages, in memory only, so the text-based soft-404 check runs on every page. |
| `timeoutSecs` | 15 | Seconds to wait for each request's response headers. |
| `maxUrls` | 1000 | Cap on unique URLs checked. You pay per URL, so this also caps cost. |
| `respectRobotsTxt` | true | Can be turned off only for your own sites, together with `ownSiteConfirmed`. |
| `ownSiteConfirmed` | false | Tick only for sites you own or are authorised to check. |

Example input:

```json
{
  "urls": [
    "crawlee.dev",
    "https://crawlee.dev/docs/quick-start",
    "https://crawlee.dev/this-page-does-not-exist",
    "https://python.org/",
    "https://developer.mozilla.org/docs/Web/HTTP/Status"
  ],
  "method": "HEAD-then-GET",
  "maxRedirects": 10
}
```

### Output

One item per unique URL. `category` is one of:

- `ok`: a 2xx answer with no redirect.
- `redirect`: one redirect, then a 2xx page.
- `redirect-chain`: two or more redirects, then a 2xx page.
- `broken`: a 4xx or 5xx answer, or the connection failed.
- `loop`: a redirect loop, or more than `maxRedirects` hops.
- `soft-404`: likely a "not found" page served with a 2xx status. This is a heuristic (see Limits).
- `dns-error`: the host name doesn't exist. The name is looked up before anything is requested, robots.txt included, so a dead domain is reported as `dns-error` (charged) and never as robots-blocked. A temporary DNS failure (the resolver didn't answer) is reported as `timeout` (free).
- `blocked-by-site`: HTTP 401, 403 or 999. The site refused the automated check (bot protection or a login wall), which says nothing about whether the page exists.
- `rate-limited`: HTTP 429 after retries, or not requested because the host answered 429 three times in a row.
- `timeout`: no answer within `timeoutSecs`.
- `blocked-by-robots`: the host's robots.txt disallows the URL, so it wasn't requested.
- `blocked-private`: the URL points at, or redirects to, localhost or a private network address, so it wasn't requested.
- `invalid-url`: the entry isn't a valid http(s) URL or domain.

Every hop in `redirectChain` has `url`, `status`, `type` (`permanent` for 301/308, `temporary` for 302/303/307, `null` for the final answer) and `ms`. Items that weren't requested or got no normal answer carry an `errorCode`.

Real item from a local test run on the default prefill (2 Nov 2026), with the timestamp trimmed:

```json
{
  "input": "https://developer.mozilla.org/docs/Web/HTTP/Status",
  "url": "https://developer.mozilla.org/docs/Web/HTTP/Status",
  "category": "redirect-chain",
  "ok": true,
  "status": 200,
  "finalUrl": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status",
  "hops": 2,
  "redirectType": "mixed",
  "redirectChain": [
    { "url": "https://developer.mozilla.org/docs/Web/HTTP/Status", "status": 302, "type": "temporary", "ms": 13 },
    { "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Status", "status": 301, "type": "permanent", "ms": 69 },
    { "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status", "status": 200, "type": null, "ms": 41 }
  ],
  "loop": false,
  "likelySoft404": false,
  "soft404Reason": null,
  "soft404Check": "redirect-only",
  "responseTimeMs": 123,
  "contentType": "text/html",
  "method": "HEAD",
  "reason": "2 redirects (mixed)",
  "error": null,
  "errorCode": null,
  "chargeable": true
}
```

The run also saves an `OUTPUT_SUMMARY` record in the default key-value store: input count, unique URLs, duplicates removed, invalid entries, URLs checked and not checked, counts per category, events charged and when the run finished.

### How it stays polite

- It requests only the URLs you list, plus their redirect hops and each host's robots.txt. It never crawls.
- About 1 request per second per host, or longer if robots.txt sets a Crawl-delay (honoured up to 30 seconds). Up to 10 different hosts are checked at the same time, and each host still gets about 1 request per second.
- It backs off on HTTP 429 and 503 and honours Retry-After (up to 30 seconds). A host that answers 429 three times in a row gets no more requests in that run.
- robots.txt is respected (RFC 9309). If a host's robots.txt returns a server error or can't be reached, the whole host is treated as off-limits. A missing robots.txt (4xx) means everything is allowed. For your own sites only, `ownSiteConfirmed` lets you turn robots.txt off. Redirect hops to other hosts always keep their robots.txt.
- Localhost and private network addresses are never requested, including as a redirect target.
- It identifies itself as `BulkUrlStatusChecker/0.1 (+Apify Actor; polite, robots-aware)`. No proxies, no logins, and no attempt to get past bot protection or CAPTCHAs.
- It stores no page content. In `GET` mode, the first 16 KB of a 2xx HTML page is read in memory for the soft-404 check and then discarded.

### How long does it take?

The default prefill (10 URLs on 8 hosts) finished in about 7 seconds in a local test run, with a peak of about 116 MB of memory. Many hosts are checked in parallel, but each host gets only about 1 request per second, so a list dominated by one host is slow: 1,000 URLs on one host take roughly 17 minutes. The default run timeout is 300 seconds, which fits about 250 URLs on a single host. Raise the run timeout or split the list for more. The run stops cleanly shortly before its timeout, keeps everything already checked, and reports the rest as not checked in `OUTPUT_SUMMARY`.

### How much does it cost?

Pay per event: one `url-checked` event per unique URL with a definitive result (a final status, a redirect chain, a loop, a likely soft-404 or a DNS failure). These are free: timeouts, rate-limited URLs, `blocked-by-site`, robots-blocked URLs, private addresses and invalid entries. Duplicates are removed before checking, so you never pay twice for the same URL in a run. Price: $0.80 per 1,000 URLs checked ($0.0008 per `url-checked` event). `maxUrls` and your maximum charge per run both cap the total.

### Use with AI agents

Designed for limited permissions, so it can be called through the Apify MCP server. Example: "Check these 200 old URLs and tell me which ones don't redirect to a working page."

### Limits

- **Soft-404 detection is a heuristic.** It flags 2xx pages whose title, heading or short text says "not found", and deep URLs that redirect to the homepage. In the default `HEAD-then-GET` mode only the redirect rule runs, unless the server refused HEAD; `soft404Check` on each item says which checks ran. Use `method: "GET"` for the full check. It can be wrong in both directions, so open flagged URLs in a browser.
- **A 401, 403 or 999 doesn't mean the URL is dead.** Many sites block automated requests. These are `blocked-by-site`, free, and worth checking in a browser.
- **Rate-limited and timed-out URLs aren't checked.** Re-run them later or open them in a browser.
- Some servers answer HEAD differently from GET. If a result looks wrong, re-check it with `method: "GET"`.
- `responseTimeMs` is measured from Apify's servers to the response headers, summed over the redirect hops. It isn't a page-speed or Core Web Vitals measurement.

### FAQ

**Does it follow links on the pages?** No. It checks only the URLs you give it. To find broken links across a whole site, use a link checker that crawls.
**What's the difference between `redirect` and `redirect-chain`?** One redirect hop versus two or more. `redirectType` shows whether the hops were permanent, temporary or mixed.
**Does `http://` get upgraded to `https://`?** No. A URL with `http://` is checked as given, so you can see its redirect to HTTPS. Only entries without a scheme get `https://`.

### Legal

Checks public URLs only. Stores status codes, redirect information, timings and the Content-Type header, not page content or personal data.

### Changelog

See CHANGELOG.md.

# Changelog

This Actor's version history is a separate document: https://apify.com/madpai/bulk-url-status-checker/changelog.md

# Actor input Schema

## `urls` (type: `array`):

One URL per line. Bare domains (example.com) are checked as https://example.com/. Duplicates are removed after normalisation (#fragments dropped). Only these URLs are requested: nothing is crawled. Invalid entries get a free item with errorCode "invalid-url".

## `maxRedirects` (type: `integer`):

Redirect hops followed per URL before it is reported as a loop (too many redirects).

## `method` (type: `string`):

HEAD-then-GET (default): a light HEAD request per hop, falling back to GET when a server refuses HEAD (405, 403 or 501). The soft-404 check then covers deep URLs redirected to the homepage, plus the page text only where GET was used. GET: every hop uses GET and 2xx HTML pages are read (first 16 KB, in memory only) so the text-based soft-404 check runs on every page.

## `timeoutSecs` (type: `integer`):

How long to wait for each request's response headers. A URL that times out is reported as timeout (free) and isn't retried.

## `maxUrls` (type: `integer`):

Cap on unique URLs checked in this run. You pay per URL, so this also caps cost. Your maximum charge per run caps it too.

## `respectRobotsTxt` (type: `boolean`):

Don't request URLs disallowed by robots.txt (reported as blocked-by-robots, free). Can only be turned off together with "I own these sites", and then only for the hosts in your list; redirect targets on other hosts always keep their robots.txt.

## `ownSiteConfirmed` (type: `boolean`):

Only check this for your own sites. It allows turning off robots.txt for the hosts in "urls". The 1 request per second per host limit always applies.

## Actor input object example

```json
{
  "urls": [
    "crawlee.dev",
    "https://crawlee.dev/docs/quick-start",
    "https://crawlee.dev/this-page-does-not-exist",
    "https://github.com/apify",
    "https://docs.apify.com/",
    "https://apify.com/store",
    "https://python.org/",
    "https://wikipedia.org/",
    "https://developer.mozilla.org/docs/Web/HTTP/Status",
    "https://www.rfc-editor.org/rfc/rfc9309"
  ],
  "maxRedirects": 10,
  "method": "HEAD-then-GET",
  "timeoutSecs": 15,
  "maxUrls": 1000,
  "respectRobotsTxt": true,
  "ownSiteConfirmed": false
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset, one item per unique URL: input, url, category (ok, broken, redirect, redirect-chain, loop, soft-404, blocked-by-site, rate-limited, blocked-by-robots, blocked-private, timeout, dns-error, invalid-url), ok, status, finalUrl, hops, redirectType, redirectChain (url, status, permanent or temporary, ms), loop, likelySoft404 (heuristic), soft404Reason, responseTimeMs, contentType, method, error, errorCode, chargeable. Every item that wasn't requested or got no normal response has an errorCode.

## `summary` (type: `string`):

OUTPUT_SUMMARY JSON: urlsInput, uniqueUrls, duplicatesRemoved, invalidUrls, checked, notChecked (over maxUrls, over budget, time limit), byCategory counts, charged events, budgetHit, timeHit, the politeness policy used and finishedAt.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "crawlee.dev",
        "https://crawlee.dev/docs/quick-start",
        "https://crawlee.dev/this-page-does-not-exist",
        "https://github.com/apify",
        "https://docs.apify.com/",
        "https://apify.com/store",
        "https://python.org/",
        "https://wikipedia.org/",
        "https://developer.mozilla.org/docs/Web/HTTP/Status",
        "https://www.rfc-editor.org/rfc/rfc9309"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("madpai/bulk-url-status-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "crawlee.dev",
        "https://crawlee.dev/docs/quick-start",
        "https://crawlee.dev/this-page-does-not-exist",
        "https://github.com/apify",
        "https://docs.apify.com/",
        "https://apify.com/store",
        "https://python.org/",
        "https://wikipedia.org/",
        "https://developer.mozilla.org/docs/Web/HTTP/Status",
        "https://www.rfc-editor.org/rfc/rfc9309",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("madpai/bulk-url-status-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "crawlee.dev",
    "https://crawlee.dev/docs/quick-start",
    "https://crawlee.dev/this-page-does-not-exist",
    "https://github.com/apify",
    "https://docs.apify.com/",
    "https://apify.com/store",
    "https://python.org/",
    "https://wikipedia.org/",
    "https://developer.mozilla.org/docs/Web/HTTP/Status",
    "https://www.rfc-editor.org/rfc/rfc9309"
  ]
}' |
apify call madpai/bulk-url-status-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,madpai/bulk-url-status-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/e8rRj5Q6lCkUEqTAK/builds/qg1Bs7hlr8VSVpJ9P/openapi.json