SEO Indexability Checker – robots.txt, noindex & Canonical

For each URL, a clear indexable yes/no with reasons: robots.txt rules for Googlebot, Bingbot or GPTBot, meta robots, X-Robots-Tag, canonical target, status code and redirects.

Why this actor

• Single verdict + reasons per URL, and evaluates AI crawlers too.
• Pay only for results: $1.00 per 1,000 urls, no start fee, failed lookups are free.
• Robust: automatic retries with exponential backoff, respects Retry-After, per-item error rows instead of crashed runs.
• Clean, flat JSON/CSV/Excel output ready for Google Sheets, Make, Zapier, n8n, Clay, or the Apify API.

Use cases

• Debug 'Discovered – not indexed' pages
• Audit after deploys
• Check AI-bot access
• Bulk-validate sitemap URLs

How to use SEO Indexability Checker

1. Configure urls, userAgent in the Input tab (or leave the prefilled sample values to test immediately).
2. Optionally set maxItems to cap the number of results and control spend.
3. Click Start and export structured results as JSON, CSV, Excel, or connect directly via API / MCP / n8n / Make.

Input

Field	Type	Description
urls	array	Pages to check.
userAgent	string	Bot whose rules to evaluate.
maxItems	integer	0 = no limit. Caps cost.

Example input:

{
  "urls": [
    "https://www.python.org/",
    "https://www.python.org/search/"
  ]
}

Output

Each dataset item looks like this (real sample from a test run, long fields trimmed):

{
  "url": "https://www.python.org/",
  "indexable": true,
  "reasons": [],
  "finalUrl": "https://www.python.org/",
  "statusCode": 200,
  "redirectChain": null,
  "robotsTxtAllowed": true,
  "metaRobots": null,
  "xRobotsTag": null,
  "canonical": null,
  "canonicalIsSelf": true,
  "checkedFor": "Googlebot",
  "checkedAt": "2026-10-14T23:42:38+00:00"
}

Rows that could not be processed contain an error field and are not charged.

Extracted output fields

Every dataset row includes structured fields ready for spreadsheets, CRMs, and AI pipelines:

• Fields: url, indexable, reasons, finalUrl, statusCode, redirectChain, robotsTxtAllowed, metaRobots, xRobotsTag, canonical, canonicalIsSelf, checkedFor, checkedAt

Comparison: SEO Indexability Checker vs alternatives

Feature	This Actor (shawndunn/seo-indexability-checker)	Typical alternatives
Pricing model	$1 / 1,000 results, $0.00 start fee	$20–$49/mo subscription or per-run start fees
Failed / error lookups	$0.00 (never charged)	Often billed by compute time even on failures
API keys / proxies required	None (works out of the box)	Often requires paid third-party keys or residential proxies
AI Agent & MCP ready	Yes (mcp.apify.com compatible, optional defaults)	Strict required schemas that reject agent calls

Pricing

Pay-per-event: $0.001 per url (= $1.00 / 1,000). Use Max results to cap spend. Apify's free plan credit covers thousands of results per month.

Integrations & API

Run on a schedule in Apify Console, call via the Apify API / Python & JS clients, or connect to Make, Zapier, n8n, Google Sheets, Slack and webhooks. Works as a tool for AI agents via Apify MCP.

Use with AI agents (MCP)

SEO Indexability Checker is configured for zero-config AI agent tool use via the Apify MCP Server (no required-field blocking on defaulted inputs).

Streamable HTTP endpoint:

https://mcp.apify.com/?tools=shawndunn/seo-indexability-checker

Claude Desktop / Cursor / Windsurf mcpServers config:

{
  "mcpServers": {
    "apify-seo-indexability-checker": {
      "url": "https://mcp.apify.com/?tools=shawndunn/seo-indexability-checker",
      "headers": {
        "Authorization": "Bearer <YOUR_APIFY_TOKEN>"
      }
    }
  }
}

Direct synchronous API call:

POST https://api.apify.com/v2/acts/shawndunn~seo-indexability-checker/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>

FAQ

Is this an official tool? No. It is an independent actor and is not affiliated with, sponsored or endorsed by the data source or any trademark owner named here.

Is it legal? The actor only reads publicly available data from public endpoints, without logins, and does not bypass paywalls or access controls. You are responsible for using the data in line with applicable laws (e.g. GDPR) and the source sites' terms.

Something broke? Open an issue in the Issues tab; we typically respond within 24 hours and ship fixes fast.

More tools by shawndunn

• Bulk PageSpeed Insights & Core Web Vitals Checker (Lighthouse) — Audit hundreds of URLs with Google PageSpeed Insights: performance, SEO, accessibility and best-practices scores, LCP, CLS, TBT ($3/1k, no start fee).
• Sitemap URL Extractor — Extract every URL from any website's XML sitemaps: auto-discovery via robots.
• Broken Link Checker & Bulk URL Status Checker — Crawl a website and test every internal and external link, or bulk-check a list of URLs (url, statusCode, ok, finalUrl) ($1/1k, no start fee).
• On-Page SEO Checker (Bulk): Titles, Meta, Headings, Canonicals, Issues — Audit any list of URLs for on-page SEO: title and meta description length, canonical, robots/noindex, H1–H3, word count ($2/1k, no start fee).
• Link Preview & Open Graph Extractor (Unfurl URLs) — Unfurl any URL like Slack or iMessage: title, description, preview image, site name, favicon/icons, theme color, Open Graph and ($1/1k, no start fee).