# Soft 404 Detector – Dead Link & Redirect-to-Homepage Checker (`accountable_eel/soft-404-detector`) Actor

Find soft 404s and dead links that return HTTP 200 while actually broken — homepage redirects, 'page not found' content, or noindex flags. Compares each domain against a live baseline probe. Built for SEO audits and link-rot monitoring. Charged only when a signal fires.

- **URL**: https://apify.com/accountable\_eel/soft-404-detector.md
- **Developed by:** [Adrian Voss](https://apify.com/accountable_eel) (community)
- **Categories:** Lead generation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.80 / 1,000 successful lookups

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Soft 404 Detector

**Find pages that return 200 OK while actually dead.** Bulk soft-404 and dead-link detector for
SEO audits, content-migration QA, and link-rot monitoring — catches the URLs a plain HTTP-status
checker waves through because the server says "200 OK" and moves on: pages silently redirected to
the homepage, serving a "page not found" message, or flagged `noindex`, all while returning a
healthy status code.

### Features

- 🎯 **True soft-404 detection, not just status codes** — a real 4xx/5xx is reported as a distinct hard error; only HTTP 200 pages get scored for "secretly dead" signals.
- 🏠 **Homepage-redirect detection** — catches the single most common soft-404 pattern: a deep URL silently collapsing to `/`.
- 📝 **Content-based detection** — title/body matched against a curated "not found / moved / expired" phrase list covering English, Spanish, French, German, and Portuguese.
- 🔗 **Canonical & meta signals** — `<link rel="canonical">` pointing at the root, `<meta http-equiv="refresh">` soft-redirects, and `noindex` via meta-robots or `X-Robots-Tag`.
- 🧪 **Per-domain baseline probe** — fetches one deliberately-nonexistent path per domain (cached, proxied) and compares real pages against it, catching soft 404s the phrase list alone would miss.
- ⚖️ **Weighted, transparent scoring** — every signal that fired is reported alongside a numeric score, not a single opaque true/false.
- 💸 **Pay only for real findings** — clean, live pages are never charged.

### Use cases

- **SEO audits** — find pages search engines still see as "live" (200 OK) that are actually dead, quietly hurting crawl budget and rankings without ever showing up in a status-code-only report.
- **Content migration QA** — confirm moved or retired articles, listings, or products didn't silently collapse to the homepage instead of redirecting to their real new location.
- **Link-rot monitoring** — schedule recurring runs over your outbound/internal links to catch rot a plain broken-link checker can't see, since these pages never return an error code.
- **Affiliate & backlink audits** — verify a "live" backlink target isn't secretly a soft-404 that no longer serves real content.

Pair this with a plain HTTP-status/broken-link checker (e.g. this account's own [http-status-lookup](../http-status-lookup)) for full coverage: one catches hard 4xx/5xx and redirect chains, this one catches everything that dodges that check by answering 200.

### How to use Soft 404 Detector – Dead Link & Redirect-to-Homepage Checker

1. **In the Apify Console.** Open the actor page and click **Start** — the `items` field is already pre-filled with a working example. Results land in the run's dataset as soon as each item is found.
2. **Via the API.** Call it directly with a POST request — no Console needed once you have an API token:
   ```bash
   curl "https://api.apify.com/v2/acts/accountable_eel~soft-404-detector/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
     -X POST \
     -H "Content-Type: application/json" \
     -d '{"items":["https://answers.yahoo.com/question/index?qid=20080101000000AAxyz12","https://answers.yahoo.com/question/index?qid=20090909090909AAabc99"]}'
   ```
3. **On a schedule.** Save this actor as an Apify **Task** with the input you want, then add a **Schedule** (hourly, daily, weekly) so it runs on its own — no server of your own required.

### Input

```json
{
  "items": ["<identifier-1>", "<identifier-2>"],
  "maxConcurrency": 5
}
```

`items` is a list of URLs to check (e.g. https://example.com/old-article). One dataset row is returned
per item; rows with `"found": false` are never charged.

### Output

One row per URL. A page that redirects to the homepage and matches a "not found" phrase comes back
likely-soft-404, with every signal that fired listed alongside its weight:

```json
{
  "query": "https://example.com/old-product-page",
  "found": true,
  "outcome": "likely_soft_404",
  "httpStatus": 200,
  "score": 5,
  "signals": ["redirected_to_homepage", "not_found_phrase_in_content", "canonical_points_to_homepage"],
  "finalUrl": "https://example.com/",
  "scrapedAt": "2026-08-20T12:00:00.000Z"
}
```

A real HTTP 4xx/5xx comes back as a distinct, still-charged hard error — never mislabeled as a soft
404:

```json
{
  "query": "https://example.com/deleted-page",
  "found": true,
  "outcome": "hard_error",
  "httpStatus": 404,
  "scrapedAt": "2026-08-20T12:00:00.000Z"
}
```

A genuinely live 2xx page with zero soft-404 signals comes back as `"found": false` and is never
charged.

### How detection works

A real HTTP 4xx/5xx is a **hard** error, never a "soft" one — a soft 404 is specifically a page that
answers 200 OK while actually being dead. So the first thing `extract()` checks is `httpStatus`:

- **4xx/5xx** → `{ "outcome": "hard_error", "httpStatus": ... }`, charged (a hard-broken link is still a
  useful, chargeable answer to "is this URL dead?").
- **2xx** → scored for soft-404 signals:
  - `redirected_to_homepage` / `redirected_to_different_path` — final path vs. requested path.
  - `not_found_phrase_in_content` — title/body matched against a curated English + Spanish/French/German/Portuguese
    "not found" phrase list (not full i18n).
  - `canonical_points_to_homepage` — `<link rel="canonical">` collapses to `/` for a deep request.
  - `meta_refresh_redirect` — `<meta http-equiv="refresh">` present.
  - `noindex_or_x_robots_tag` — `noindex` via `<meta name="robots">` or the `X-Robots-Tag` header.
  - `matches_baseline_final_url` / `title_similarity_to_baseline` / `content_length_within_baseline_range` —
    comparison against a one-per-domain baseline probe (see below).

Each signal has a weight; signals sum to a `score`, and `likelySoft404`/`outcome: "likely_soft_404"` only
trips at `score >= 3`. A single strong signal (a homepage redirect, or a matched not-found phrase) is
decisive alone; weaker signals need corroboration. A 2xx page with zero signals returns `null`
(`"found": false`, unbilled) — it looks like a genuine live page.

**Baseline probe (Tier B):** once per domain per run, the actor fetches a deliberately-nonexistent sibling
path on that domain (proxied, cached) and compares each item's page against it — the standard differential
technique for soft-404 detection, and the only way to catch sites whose "nothing here" page doesn't match
any phrase in the list. Probe failures degrade gracefully to "no baseline signals" rather than failing the
item.

### Pricing

$5 per 1,000 results, plus a $0.005 start fee. Misses (`found:false`) are never charged.

### Use it from Clay, n8n, Make, or an AI agent

This actor runs synchronously over plain HTTP — call it directly from a script, a workflow tool, or an AI agent, no Apify Console needed once you have an API token.

```bash
curl "https://api.apify.com/v2/acts/accountable_eel~soft-404-detector/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"items":["https://answers.yahoo.com/question/index?qid=20080101000000AAxyz12","https://answers.yahoo.com/question/index?qid=20090909090909AAabc99"]}'
```

**n8n.** Add an HTTP Request node: Method `POST`, URL `https://api.apify.com/v2/acts/accountable_eel~soft-404-detector/run-sync-get-dataset-items?token=<YOUR_TOKEN>`, Body Content Type `JSON`, JSON Body `{"items":["https://answers.yahoo.com/question/index?qid=20080101000000AAxyz12","https://answers.yahoo.com/question/index?qid=20090909090909AAabc99"]}` (swap in an expression from an earlier node for a real value).

**Clay.** Add an "HTTP API" column: Method `POST`, URL `https://api.apify.com/v2/acts/accountable_eel~soft-404-detector/run-sync-get-dataset-items?token=<YOUR_TOKEN>`, Body `{"items":["{{value}}"]}`, mapping the row's value into the `items` array.

**MCP.** In Claude, Cursor, or any MCP client with the Apify MCP server, ask for "Soft 404 Detector – Dead Link & Redirect-to-Homepage Chec..." — the agent will find and run this actor.

### FAQ

**Does this replace a broken-link/HTTP-status checker?** No — the two are complementary. A status-code
checker catches hard 4xx/5xx and redirect chains; this actor catches pages that dodge that check entirely
by returning 200 while actually dead.

**Does it render JavaScript?** No — it's a static-HTML (Cheerio) crawler. A soft 404 that only appears
after client-side JS runs (e.g. a React "not found" route rendered post-load) will not be caught.

**Why does a real hard 404 still get charged?** Because "is this URL dead?" is a useful answer either
way — a hard 404/5xx is reported distinctly as `outcome: "hard_error"`, never mislabeled as a soft 404,
but it's still a real finding rather than a false negative hidden behind an empty, unbilled row.

# Actor input Schema

## `items` (type: `array`):

One item per line — see the item shape and examples below. Only the items we actually find are charged — never per run, and never for a miss.

## `maxConcurrency` (type: `integer`):

Parallel requests. Keep conservative — this target has no browser fallback, so getting blocked costs more than slow-and-steady.

## `proxyConfiguration` (type: `object`):

Apify Proxy config. Residential recommended for anti-bot-sensitive targets.

## Actor input object example

```json
{
  "items": [
    "https://answers.yahoo.com/question/index?qid=20080101000000AAxyz12",
    "https://answers.yahoo.com/question/index?qid=20090909090909AAabc99"
  ],
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        "https://answers.yahoo.com/question/index?qid=20080101000000AAxyz12",
        "https://answers.yahoo.com/question/index?qid=20090909090909AAabc99"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("accountable_eel/soft-404-detector").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "items": [
        "https://answers.yahoo.com/question/index?qid=20080101000000AAxyz12",
        "https://answers.yahoo.com/question/index?qid=20090909090909AAabc99",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("accountable_eel/soft-404-detector").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    "https://answers.yahoo.com/question/index?qid=20080101000000AAxyz12",
    "https://answers.yahoo.com/question/index?qid=20090909090909AAabc99"
  ]
}' |
apify call accountable_eel/soft-404-detector --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,accountable_eel/soft-404-detector"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/PIQBdy2mAYqGJB9MD/builds/cJHElBKsYbYlLlnLk/openapi.json