Pricing

Pay per usage

Try for free

Go to Apify Store

Soft 404 & Indexability Auditor

Try for free

Finds pages that answer HTTP 200 while showing 'not found', probes every route family with impossible URLs, and audits sitemap URLs for errors, redirects, noindex and canonical mismatches — with cache-header evidence for Next.js, Vercel, Netlify and Cloudflare.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Ricardo Faya

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Why this exists

On a production Next.js site we found that every dynamic route answered 200 on its not-found page. The ISR cache had stored the notFound() render and served it as a valid page (x-nextjs-prerender: 1). Uptime monitors were green, nothing logged an error, and search engines could index an unlimited number of empty pages. Adding a root not-found.tsx did not fix it.

Classic crawlers only check the URLs that exist. The defect lives in the URLs that don't.

What it does

Discovers your sitemaps from robots.txt (or takes a sitemap/page URL), follows sitemap indexes.
Audits every URL and flags:
- soft-404 — HTTP 200 whose title or H1 says "not found" (EN, PT, ES, FR, DE built in; add your own phrases);
- possible-soft-404 — thin 200 page containing a not-found phrase;
- error-status — 4xx/5xx listed in the sitemap;
- redirect — sitemap URL that redirects (with the full chain);
- noindex — sitemap URL that is noindex (meta or X-Robots-Tag);
- canonical-elsewhere — canonical pointing to another URL;
- missing-title.
Probes each route family (/blog/, /pt-BR/course/, /products/…) with a URL that cannot exist. A SOFT-404 ROUTE verdict comes with the cache headers as evidence — x-nextjs-prerender, x-nextjs-cache, x-vercel-cache, cf-cache-status, netlify-cdn-cache-control, age — and a plain-English explanation.

Output

Dataset rows of two types:

{
  "type": "route-probe",
  "url": "https://example.com/pt-BR/curso/soft404-probe-x7k2m9qa",
  "routePrefix": "/pt-BR/curso/",
  "statusCode": 200,
  "verdict": "SOFT-404 ROUTE",
  "title": "Página não encontrada | Example",
  "evidenceHeaders": { "x-nextjs-prerender": "1", "cache-control": "public, max-age=0, must-revalidate" },
  "explanation": "A URL that cannot exist answered HTTP 200. Any mistyped, deleted or invented URL under /pt-BR/curso/ can be crawled and indexed as a real page. Next.js cache headers are present: ..."
}

{
  "type": "page",
  "url": "https://example.com/old-offer",
  "statusCode": 200,
  "worstSeverity": "critical",
  "issues": [{ "code": "soft-404", "severity": "critical", "detail": "HTTP 200 but the page says \"page not found\"." }]
}

The key-value store OUTPUT has totals per issue code and per site.

Good to know

Audit your own sites or sites you are authorized to test. Keep concurrency low on small servers.
Compressed .xml.gz sitemaps are skipped with a warning — pass the uncompressed URL.
Pages rendered only in the browser (client-side "not found") are judged by the HTML the server returns, which is what crawlers see first.

SEO Audit: On-Page, Redirects & AI Crawler Access

aiqlabs/seo-audit-tool

Give it a list of URLs and it audits each page for on-page and technical SEO, plus the four checks general auditors skip: AI crawler rules, /llms.txt, redirects that throw the path away, and noindex via X-Robots-Tag. One row per URL - to crawl a whole site instead, use SEO & AI Visibility Auditor.

Ai-Q Labs

Meta Robots Auditor - Is This Page Actually Indexable?

eliai/webpage-meta-robots-auditor

Indexability verdict per URL from ALL the signals: meta robots, meta googlebot, the X-Robots-Tag header (the one checkers miss), canonical + self-canonical check — with the exact reasons. Catches shipped staging noindex and CDN-injected headers. $0.002 per page vs $0.007-$0.011 measured incumbents.

Broke to Built

SEO Migration Redirect Auditor & Client Report

exceptional_nugget/migration-redirect-auditor

Check old-to-new URL mappings, redirect chains, broken destinations, canonical tags and noindex directives. Export actionable JSON/CSV results and a printable HTML client report. HTTP-only, no proxy or AI API key required.

LibriHouse

SEO Audit + AI Visibility · Are You Citable by ChatGPT?

siccscha/seo-health-auditor

Technical SEO audit plus the question no other audit answers: can AI systems cite you? Checks which AI crawlers your robots.txt allows (only retrieval agents can cite you), flags pages that are empty without JavaScript, and diffs against your previous run. Plus fixes you can paste.

Chris S

Sitemap Indexability Auditor

nexgenwatch/sitemap-indexability-auditor

Audit whether the URLs in your sitemaps (or a plain URL list) are actually indexable — sitemaps are expanded (bounded nested depth + a hard URL cap) and every page URL gets one evidenced indexability verdict.

NexGen Watch

Robots.txt & Security.txt Compliance Auditor

devilscrapes/robots-txt-security-txt-auditor

Keyless, plain-HTTP compliance auditor. Given a list of sites, it fetches and parses robots.txt, security.txt (RFC 9116), ads.txt, and humans.txt into one structured audit row per site — AI-crawler posture, security contacts/expiry, and ad-tech supply-chain record counts. No API key needed.

DevilScrapes

Agent Accessibility Auditor llms txt Robots AI Schema Check

mambalabs/agent-accessibility-auditor

Can an AI agent read this site. Checks llms.txt, robots AI crawler rules and the Content Signal policy, structured data, render mode, and machine readable endpoints like sitemap, OpenAPI and well known files. Every field is a fact read off a fetch, never a model opinion.

Mamba Labs

GEO & AEO Site Audit — llms.txt, AI Crawlers & Schema Fix List

wheaten_steelpan/geo-site-auditor

Audit your website for Generative Engine Optimization (GEO) and Answer Engine Optimization (AEO). Crawls pages, tests robots.txt/llms.txt/schema.org, evaluates AI answerability, and generates ranked fixes + visual report.

Vanja V

Internal Link Architecture Verifier

nexgenwatch/internal-link-architecture-verifier

Runs a bounded same-origin crawl unioned with your sitemap, builds the internal link graph, and returns one evidenced verdict per page: orphan-from-sitemap status, click depth, nofollow conflicts, canonical inlinks, inbound/outbound counts and anchor-text concentration.

NexGen Watch

Broken Image Checker - Find 404 Images on Any Page

eliai/webpage-broken-image-checker

Scan a page for broken images: every img src/srcset HEAD-checked for 404s/errors/non-image types, plus missing-alt count. Bulk. $0.002 per page, cheaper than paid broken-link checkers; failed page fetches free.

Broke to Built

Indexability and Canonical Audit Crawler

kayhermes/indexability-and-canonical-audit-crawler

Deterministic local indexability + canonical audit of explicit public HTTP(S) URLs from static HTML: robots.txt enforcement, meta/X-Robots-Tag noindex, HTTP status, redirect chains, and declared canonical analysis.