Meta Robots Auditor — Is This Page Actually Indexable?

One request per URL answers the question that decides whether a page can rank at all: is it indexable? This actor reads every on-page indexability signal — <meta name="robots">, <meta name="googlebot">, the X-Robots-Tag HTTP header (the one most checkers forget), and the canonical link — and returns a verdict with the exact reasons. Up to 50 URLs per run, online, by API, or as an agent tool via Apify MCP.

The classic disasters this catches: a staging noindex shipped to production, a CDN adding X-Robots-Tag: noindex nobody sees in the HTML, a canonical pointing at another domain after a migration. Each is invisible in the browser and fatal in search.

What you get

• indexable — the bottom-line verdict
• reasons — exactly why: "noindex directive present (X-Robots-Tag)", "canonical points elsewhere (...)"
• directives — resolved flags (noindex, nofollow, noarchive, nosnippet, noimageindex, none) + the full raw list, merged across all three sources
• metaRobots / metaGooglebot / xRobotsTag — each source's raw value, so you know where a directive came from
• canonical / selfCanonical — resolved absolute canonical + whether it points at itself
• Fail-soft: an unreachable URL never fails the run — {ok: false, error}, never charged. (An HTTP 404 IS an honest verdict — "not indexable, because 404" — and is a normal audited record.)

Input

{ "urls": ["https://yoursite.com/", "https://yoursite.com/pricing"], "maxUrls": 50 }

Or a single URL via url.

Output (real run)

{
  "url": "https://en.wikipedia.org/wiki/Web_crawler",
  "ok": true,
  "status": 200,
  "indexable": true,
  "metaRobots": "max-image-preview:standard",
  "xRobotsTag": null,
  "directives": { "noindex": false, "nofollow": false, "all": ["max-image-preview"] },
  "canonical": "https://en.wikipedia.org/wiki/Web_crawler",
  "selfCanonical": true,
  "reasons": ["canonical is self-referential."]
}

Wikipedia's Special:Search page in the same run comes back indexable: false with the meta-robots noindex caught and named — asserted by our release test.

Pricing

$0.002 per page audited — all three signal sources + canonical in one price. No start fee. Unreachable URLs are never charged.

Measured against store incumbents (2026-08-07): burly_bat/site-launch-migration-audit charges $0.007 per URL, 213x/sitemap-seo-auditor $0.01 start + $0.001 per item. A 50-page release check here costs $0.10.

Honest limits

• On-page + header signals only: robots.txt crawl rules are a separate mechanism — pair with our Robots Rule Tester for the crawl half (a page can be indexable by meta but blocked by robots.txt, and vice versa).
• Static fetch: a noindex injected by client-side JavaScript after load is not visible (rare, and itself a bad practice — Google may miss it too).
• "Indexable" means no blocking directive was found — it does not guarantee Google will index or rank the page.

FAQ

Why check X-Robots-Tag when I can view source? Because it is an HTTP header, not HTML — CDNs, reverse proxies and app middleware add it invisibly. It carries exactly the same force as meta robots and is the most-missed cause of "why did we vanish from Google".

What does a wrong canonical look like here? selfCanonical: false with the reason naming the target URL — the page is telling search engines "index that other URL instead of me".

Can I gate deploys on this? Yes — run your critical URLs post-deploy and fail the pipeline when any record has indexable: false (or a non-self canonical you didn't expect). That single check would have prevented most public staging-noindex incidents.

Does it follow redirects? Yes — the audit applies to the final URL after redirects, and finalUrl tells you where it landed.

Why did some rows come back ok: false? DNS failure or timeout — nothing was audited. Recorded, never charged.

Use from code or AI agents

curl -s "https://api.apify.com/v2/acts/EliAI~webpage-meta-robots-auditor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST -H 'Content-Type: application/json' \
  -d '{"urls": ["https://yoursite.com/", "https://yoursite.com/key-landing-page"]}'

Agents: connect Apify MCP and call the EliAI/webpage-meta-robots-auditor tool.

• Capability: audit indexability of one or many URLs — meta robots, meta googlebot, X-Robots-Tag, canonical, verdict + reasons
• Required input: url (string) or urls (array)
• Returns: one record per URL; indexable + reasons are the verdict
• Bounded: 50 URLs per run; failures isolate per URL
• Side effects: none (one GET per page)

Who uses it

• SEO specialists catching a staging noindex that shipped to production
• Developers finding X-Robots-Tag: noindex headers a CDN adds outside the HTML
• Migration leads checking canonicals do not point at the old domain
• Agencies diagnosing why client pages are not indexed
• Content teams verifying new landing pages are indexable before a launch