# HTTP Status Checker: Bulk URL & Redirect Audit (`aiqlabs/http-status-checker`) Actor

Bulk URL status check that also grades the redirect: 301s that silently drop the path to the homepage, loops and long chains, HTTPS downgrades, meta-refresh and JavaScript redirects, soft 404s answering 200, and noindex on live pages. One row per URL with a plain-English issue list.

- **URL**: https://apify.com/aiqlabs/http-status-checker.md
- **Developed by:** [Ai-Q Labs](https://apify.com/aiqlabs) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## HTTP Status Checker: Bulk URLs & Redirect Audit

Paste a list of URLs. Get **one row per URL** with the status code, the full
redirect chain, and — the part other status checkers skip — a verdict on whether
that redirect actually did its job.

A status code on its own lies to you. `301 → 200 OK` looks healthy in every bulk
checker on the market. It is also exactly what you see when 4,000 old article
URLs quietly redirect to your homepage after a migration, which Google treats as
a soft 404 and drops from the index.

This Actor reads the chain, not just the last number.

### What it finds

| Issue | Severity | Why it costs you |
|---|---|---|
| `redirect_drops_path` | error | The URL redirects to the site root and the path is gone. Google calls a redirect-to-homepage a soft 404: the old URL's ranking is discarded instead of passed on. |
| `soft_404` | error | Answers `200` but the page reads like a not-found page. Indexable, rankable, and useless. |
| `broken` | error | `404` or `410`. |
| `server_error` | error | `5xx`. |
| `unreachable` | error | DNS failure, TLS failure, connection refused or timeout, reported with the underlying message. |
| `redirect_loop` | error | A URL in the chain redirects back to one already visited. |
| `too_many_redirects` | error | The chain never terminates inside the hop limit. |
| `https_downgrade` | error | A hop moves from `https://` to `http://`. |
| `meta_refresh_redirect` | warning | HTTP said `200`, then the HTML redirects with `<meta http-equiv="refresh">`. Status-only checkers call this URL fine. |
| `noindex` | warning | Answers `2xx` but carries `noindex` in a meta tag **or** the `X-Robots-Tag` header. |
| `canonical_to_homepage` | warning | A sub-page canonicalises to `/`, asking search engines to drop it. |
| `redirect_drops_path_offsite` | warning | Same as the headline error, but the destination is another site — normal for a link shortener, a problem after an acquisition. |
| `redirect_to_parent` | warning | `/a/b/c` ends at `/a/b`. Sometimes right, often a lazy catch-all rule. |
| `redirect_chain_long` | warning | Three or more hops. Every hop is a round trip. |
| `temporary_redirect_for_permanent_move` | warning | `302`/`307` used for an HTTP→HTTPS or www change. Those moves are permanent; use `301`. |
| `query_dropped` | warning | The redirect lost query parameters that are **not** tracking codes. `utm_*`, `gclid`, `fbclid` and friends are ignored on purpose. |
| `content_type_mismatch` | warning | The URL ends in `.pdf` but the server answered with HTML — a missing file served as a page still returns `200`. |
| `blocked_by_robots` | warning | The site's own `robots.txt` blocks the URL. Reported with the exact rule. |
| `slow_response` | warning | The whole chain took longer than your threshold. |
| `access_denied` / `rate_limited` | warning | `401`, `403`, `429`, `999`. Reachable but refused — not the same as broken. |
| `cross_site_redirect` / `path_changed` / `js_redirect_hint` / `canonical_elsewhere` / `http_to_https` / `redirected` | info | Facts worth knowing that are not necessarily faults. |

Every row carries `issueDetails`: one plain sentence per issue saying what to do
about it.

#### The one that pays for itself

After a site move, the tempting shortcut is a catch-all rule that sends anything
unmatched to `/`. Visitors land somewhere, nothing 404s, the dashboard is green.

What actually happens: Google follows each redirect, sees a homepage that has
nothing to do with the requested URL, classifies it as a soft 404, and drops the
old URL — along with every link pointing at it. Because the response is
`200 OK`, no status checker complains.

This Actor compares the path you asked for with the path you ended up on:

```
/blog/2019/migration-notes  ->  /        error   redirect_drops_path
/products/old-sku-1182      ->  /        error   redirect_drops_path
/index.html                 ->  /        ok      (index tidying, not a fault)
/a/b                        ->  /a/b/    info    (trailing slash, not a fault)
```

`index.html`, `default.aspx` and friends are recognised as directory tidying and
never flagged. Trailing-slash and `http→https` hops are not faults either.

### What makes a status checker miss things

Three cases where the HTTP layer says `200` and the page still fails:

1. **`<meta http-equiv="refresh">`** — the browser and Google both move on. The
   status check sees a healthy page.
2. **A not-found page served as `200`** — the title says "Page not found", the
   status says success. Detected from `<title>` and the first `<h1>` in nine
   languages, and the matched text is quoted back to you in
   `softNotFoundEvidence` so you can judge it yourself.
3. **`noindex`** — the URL works perfectly and is excluded from search. Read from
   both the meta tag and the `X-Robots-Tag` header, so `noindex` on a PDF is
   caught too.

A bare `404` in a page title is not enough to trigger a soft-404 report: an
article called "Our 404 page redesign story" is left alone.

### Input

```json
{
  "urls": [
    "https://example.com/blog/old-post",
    "https://example.com/products/sku-1182",
    "http://example.com/"
  ],
  "analyzeHtml": true,
  "respectRobotsTxt": true,
  "onlyIssues": false,
  "maxConcurrency": 8
}
```

**The path is never thrown away.** `https://example.com/blog/old-post` is
checked as that page, not reduced to `example.com`. Bare hosts work too. One
line may hold several URLs separated by spaces or newlines.

| Option | Default | What it does |
|---|---|---|
| `analyzeHtml` | `true` | Read the final HTML for soft 404s, `noindex`, meta refresh, canonical and JS redirects. Off = status and chain only. |
| `respectRobotsTxt` | `true` | Skip URLs the site's `robots.txt` blocks and report them as `blocked_by_robots`. Turn off for sites you own that block all crawlers. |
| `robotsAgent` | `Googlebot` | Whose `robots.txt` rules to apply. |
| `onlyIssues` | `false` | Keep only rows with a warning or an error. |
| `maxConcurrency` | `8` | Requests in flight. Lower it for a small server. |
| `requestTimeoutSecs` | `20` | Per-request timeout. |
| `maxRedirects` | `8` | Hop limit before the chain is called too long. |
| `slowResponseMs` | `3000` | Slow-response threshold. |
| `maxUrls` | `10000` | Safety cap per run. |

### Output

One row per URL. The key fields:

| Field | Meaning |
|---|---|
| `status` / `ok` / `severity` | Final status code, a clean-run boolean, and `ok`/`info`/`warning`/`error`. |
| `issues` / `issueDetails` | The codes above, and one sentence each on what to do. |
| `finalUrl` / `redirectPath` / `redirectChain` | Where it ended up, the chain as `301 -> 301 -> 200`, and every hop with its own status. |
| `pathDropped` | `true` when the redirect discarded the path. |
| `softNotFound` / `softNotFoundEvidence` | The soft-404 verdict and the text it was based on. |
| `noindex` / `metaRobots` | Indexing verdict and the raw directives, header and meta together. |
| `canonicalUrl` / `metaRefreshTo` / `jsRedirectHints` | What the HTML says about where this page really lives. |
| `title` / `h1` / `contentType` / `contentLength` / `responseTimeMs` / `charset` | The plain facts. |
| `blockedByRobots` / `robotsRule` | Whether `robots.txt` stopped the check, and the exact rule. |

A `SUMMARY` record in the key-value store holds the totals: how many were
broken, how many redirects dropped the path, how many soft 404s, plus a count
per issue code.

Dataset views: **Overview**, **Problems only**, **Redirects**.

### How it behaves on your servers

- **Two requests per URL at most** — `robots.txt` once per host (cached), then
  the URL itself. No crawling, no link following, no JavaScript execution.
- **Redirects are followed by hand**, so the chain survives into the output
  instead of being collapsed into a single `200`.
- **The body is read up to 200 KB and then the connection is dropped.** A status
  check has no business downloading a video.
- **Non-HTML responses are not parsed as HTML** — a `.zip` is checked and
  released.
- **Legacy charsets are decoded properly**, so a Shift\_JIS "page not found"
  title is still readable.
- An honest user agent that identifies the Actor and links back to this page.

### Limits, stated plainly

- The HTML checks are regex over the first 200 KB, not a browser. Pages that
  build their `<head>` in JavaScript will report less. `truncated_body` tells you
  when `</head>` was past the cut.
- `js_redirect_hint` is a **hint**: the assignment it found may sit behind a
  condition that never fires.
- `soft_404` is a heuristic over nine languages. The evidence is always quoted so
  you can overrule it.
- Sites behind bot protection may answer `403` to any non-browser client. That is
  reported as `access_denied`, not as broken, because it usually is not.
- `respectRobotsTxt` is on by default. If you are checking a staging site with a
  blanket `Disallow: /`, turn it off.

***

Built by **Ai-Q Labs**. Verified by 61 unit tests and 72 live checks against
real servers before publishing.

# Actor input Schema

## `urls` (type: `array`):

URLs to check, one per line. The full path is kept, so /blog/old-post is checked as that page and not as the domain. Bare hosts (example.com) are accepted too. One line may hold several URLs separated by spaces or newlines.

## `analyzeHtml` (type: `boolean`):

Look inside the final HTML for soft 404s, noindex, meta refresh redirects, canonical tags and JavaScript redirects. Turn off for a status-only run.

## `respectRobotsTxt` (type: `boolean`):

Read the robots.txt of every host and skip URLs it blocks, reporting them as blocked\_by\_robots. Turn off for sites you own that block all crawlers.

## `robotsAgent` (type: `string`):

Which user-agent's rules to apply when reading robots.txt.

## `onlyIssues` (type: `boolean`):

Keep rows whose severity is warning or error and drop the clean ones.

## `maxConcurrency` (type: `integer`):

How many URLs to request at the same time. Lower this when checking a single small server.

## `requestTimeoutSecs` (type: `integer`):

Give up on a single request after this many seconds.

## `maxRedirects` (type: `integer`):

Stop following after this many hops and report the chain as too long.

## `slowResponseMs` (type: `integer`):

Flag a URL as slow when the whole chain takes longer than this.

## `maxUrls` (type: `integer`):

Safety cap on how many URLs one run will check.

## Actor input object example

```json
{
  "urls": [
    "https://apify.com/store",
    "https://apify.com/this-page-does-not-exist",
    "http://apify.com/"
  ],
  "analyzeHtml": true,
  "respectRobotsTxt": true,
  "robotsAgent": "Googlebot",
  "onlyIssues": false,
  "maxConcurrency": 8,
  "requestTimeoutSecs": 20,
  "maxRedirects": 8,
  "slowResponseMs": 3000,
  "maxUrls": 10000
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://apify.com/store",
        "https://apify.com/this-page-does-not-exist",
        "http://apify.com/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("aiqlabs/http-status-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://apify.com/store",
        "https://apify.com/this-page-does-not-exist",
        "http://apify.com/",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("aiqlabs/http-status-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://apify.com/store",
    "https://apify.com/this-page-does-not-exist",
    "http://apify.com/"
  ]
}' |
apify call aiqlabs/http-status-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=aiqlabs/http-status-checker",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/pbvmbdzyIB8UdhFeQ/builds/cyC1Myko4wca3jPiW/openapi.json