Soft 404 Detector

Find pages that return 200 OK while actually dead. Bulk soft-404 and dead-link detector for SEO audits, content-migration QA, and link-rot monitoring — catches the URLs a plain HTTP-status checker waves through because the server says "200 OK" and moves on: pages silently redirected to the homepage, serving a "page not found" message, or flagged noindex, all while returning a healthy status code.

Features

• 🎯 True soft-404 detection, not just status codes — a real 4xx/5xx is reported as a distinct hard error; only HTTP 200 pages get scored for "secretly dead" signals.
• 🏠 Homepage-redirect detection — catches the single most common soft-404 pattern: a deep URL silently collapsing to /.
• 📝 Content-based detection — title/body matched against a curated "not found / moved / expired" phrase list covering English, Spanish, French, German, and Portuguese.
• 🔗 Canonical & meta signals — <link rel="canonical"> pointing at the root, <meta http-equiv="refresh"> soft-redirects, and noindex via meta-robots or X-Robots-Tag.
• 🧪 Per-domain baseline probe — fetches one deliberately-nonexistent path per domain (cached, proxied) and compares real pages against it, catching soft 404s the phrase list alone would miss.
• ⚖️ Weighted, transparent scoring — every signal that fired is reported alongside a numeric score, not a single opaque true/false.
• 💸 Pay only for real findings — clean, live pages are never charged.

Use cases

• SEO audits — find pages search engines still see as "live" (200 OK) that are actually dead, quietly hurting crawl budget and rankings without ever showing up in a status-code-only report.
• Content migration QA — confirm moved or retired articles, listings, or products didn't silently collapse to the homepage instead of redirecting to their real new location.
• Link-rot monitoring — schedule recurring runs over your outbound/internal links to catch rot a plain broken-link checker can't see, since these pages never return an error code.
• Affiliate & backlink audits — verify a "live" backlink target isn't secretly a soft-404 that no longer serves real content.

Pair this with a plain HTTP-status/broken-link checker (e.g. this account's own ../http-status-lookup) for full coverage: one catches hard 4xx/5xx and redirect chains, this one catches everything that dodges that check by answering 200.

How to use Soft 404 Detector – Dead Link & Redirect-to-Homepage Checker

1. In the Apify Console. Open the actor page and click Start — the items field is already pre-filled with a working example. Results land in the run's dataset as soon as each item is found.
2. Via the API. Call it directly with a POST request — no Console needed once you have an API token:

   curl "https://api.apify.com/v2/acts/accountable_eel~soft-404-detector/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
     -X POST \
     -H "Content-Type: application/json" \
     -d '{"items":["https://answers.yahoo.com/question/index?qid=20080101000000AAxyz12","https://answers.yahoo.com/question/index?qid=20090909090909AAabc99"]}'
   

3. On a schedule. Save this actor as an Apify Task with the input you want, then add a Schedule (hourly, daily, weekly) so it runs on its own — no server of your own required.

Input

{
  "items": ["<identifier-1>", "<identifier-2>"],
  "maxConcurrency": 5
}

items is a list of URLs to check (e.g. https://example.com/old-article). One dataset row is returned per item; rows with "found": false are never charged.

Output

One row per URL. A page that redirects to the homepage and matches a "not found" phrase comes back likely-soft-404, with every signal that fired listed alongside its weight:

{
  "query": "https://example.com/old-product-page",
  "found": true,
  "outcome": "likely_soft_404",
  "httpStatus": 200,
  "score": 5,
  "signals": ["redirected_to_homepage", "not_found_phrase_in_content", "canonical_points_to_homepage"],
  "finalUrl": "https://example.com/",
  "scrapedAt": "2026-08-20T12:00:00.000Z"
}

A real HTTP 4xx/5xx comes back as a distinct, still-charged hard error — never mislabeled as a soft 404:

{
  "query": "https://example.com/deleted-page",
  "found": true,
  "outcome": "hard_error",
  "httpStatus": 404,
  "scrapedAt": "2026-08-20T12:00:00.000Z"
}

A genuinely live 2xx page with zero soft-404 signals comes back as "found": false and is never charged.

How detection works

A real HTTP 4xx/5xx is a hard error, never a "soft" one — a soft 404 is specifically a page that answers 200 OK while actually being dead. So the first thing extract() checks is httpStatus:

• 4xx/5xx → { "outcome": "hard_error", "httpStatus": ... }, charged (a hard-broken link is still a useful, chargeable answer to "is this URL dead?").
• 2xx → scored for soft-404 signals:
  • redirected_to_homepage / redirected_to_different_path — final path vs. requested path.
  • not_found_phrase_in_content — title/body matched against a curated English + Spanish/French/German/Portuguese "not found" phrase list (not full i18n).
  • canonical_points_to_homepage — <link rel="canonical"> collapses to / for a deep request.
  • meta_refresh_redirect — <meta http-equiv="refresh"> present.
  • noindex_or_x_robots_tag — noindex via <meta name="robots"> or the X-Robots-Tag header.
  • matches_baseline_final_url / title_similarity_to_baseline / content_length_within_baseline_range — comparison against a one-per-domain baseline probe (see below).

Each signal has a weight; signals sum to a score, and likelySoft404/outcome: "likely_soft_404" only trips at score >= 3. A single strong signal (a homepage redirect, or a matched not-found phrase) is decisive alone; weaker signals need corroboration. A 2xx page with zero signals returns null ("found": false, unbilled) — it looks like a genuine live page.

Baseline probe (Tier B): once per domain per run, the actor fetches a deliberately-nonexistent sibling path on that domain (proxied, cached) and compares each item's page against it — the standard differential technique for soft-404 detection, and the only way to catch sites whose "nothing here" page doesn't match any phrase in the list. Probe failures degrade gracefully to "no baseline signals" rather than failing the item.

Pricing

$5 per 1,000 results, plus a $0.005 start fee. Misses (found:false) are never charged.

Use it from Clay, n8n, Make, or an AI agent

This actor runs synchronously over plain HTTP — call it directly from a script, a workflow tool, or an AI agent, no Apify Console needed once you have an API token.

curl "https://api.apify.com/v2/acts/accountable_eel~soft-404-detector/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"items":["https://answers.yahoo.com/question/index?qid=20080101000000AAxyz12","https://answers.yahoo.com/question/index?qid=20090909090909AAabc99"]}'

n8n. Add an HTTP Request node: Method POST, URL https://api.apify.com/v2/acts/accountable_eel~soft-404-detector/run-sync-get-dataset-items?token=<YOUR_TOKEN>, Body Content Type JSON, JSON Body {"items":["https://answers.yahoo.com/question/index?qid=20080101000000AAxyz12","https://answers.yahoo.com/question/index?qid=20090909090909AAabc99"]} (swap in an expression from an earlier node for a real value).

Clay. Add an "HTTP API" column: Method POST, URL https://api.apify.com/v2/acts/accountable_eel~soft-404-detector/run-sync-get-dataset-items?token=<YOUR_TOKEN>, Body {"items":["{{value}}"]}, mapping the row's value into the items array.

MCP. In Claude, Cursor, or any MCP client with the Apify MCP server, ask for "Soft 404 Detector – Dead Link & Redirect-to-Homepage Chec..." — the agent will find and run this actor.

FAQ

Does this replace a broken-link/HTTP-status checker? No — the two are complementary. A status-code checker catches hard 4xx/5xx and redirect chains; this actor catches pages that dodge that check entirely by returning 200 while actually dead.

Does it render JavaScript? No — it's a static-HTML (Cheerio) crawler. A soft 404 that only appears after client-side JS runs (e.g. a React "not found" route rendered post-load) will not be caught.

Why does a real hard 404 still get charged? Because "is this URL dead?" is a useful answer either way — a hard 404/5xx is reported distinctly as outcome: "hard_error", never mislabeled as a soft 404, but it's still a real finding rather than a false negative hidden behind an empty, unbilled row.