Meta Robots Checker: Is a Page Indexable? noindex, Canonical
Pricing
from $1.60 / 1,000 audited pages
Meta Robots Checker: Is a Page Indexable? noindex, Canonical
Checks whether a page URL can be indexed by reading meta robots, meta googlebot, the X-Robots-Tag header and the canonical link. Returns a yes/no indexable verdict with the reasons and the raw directives, up to 50 URLs per run; unreachable URLs are free. For SEO specialists, developers and agencies.
Pricing
from $1.60 / 1,000 audited pages
Rating
0.0
(0)
Developer
Broke to Built
Maintained by CommunityActor stats
0
Bookmarked
1
Total users
0
Monthly active users
15 days ago
Last modified
Categories
Features
Share
Meta Robots Auditor — Is This Page Actually Indexable?
One request per URL answers the question that decides whether a page can rank at all: is it indexable? This actor reads every on-page indexability signal — <meta name="robots">, <meta name="googlebot">, the X-Robots-Tag HTTP header (the one most checkers forget), and the canonical link — and returns a verdict with the exact reasons. Up to 50 URLs per run, online, by API, or as an agent tool via Apify MCP.
The classic disasters this catches: a staging noindex shipped to production, a CDN adding X-Robots-Tag: noindex nobody sees in the HTML, a canonical pointing at another domain after a migration. Each is invisible in the browser and fatal in search.
What you get
- indexable — the bottom-line verdict
- reasons — exactly why: "noindex directive present (X-Robots-Tag)", "canonical points elsewhere (...)"
- directives — resolved flags (
noindex,nofollow,noarchive,nosnippet,noimageindex,none) + the full raw list, merged across all three sources - metaRobots / metaGooglebot / xRobotsTag — each source's raw value, so you know where a directive came from
- canonical / selfCanonical — resolved absolute canonical + whether it points at itself
- Fail-soft: an unreachable URL never fails the run —
{ok: false, error}, never charged. (An HTTP 404 IS an honest verdict — "not indexable, because 404" — and is a normal audited record.)
Input
{ "urls": ["https://yoursite.com/", "https://yoursite.com/pricing"], "maxUrls": 50 }
Or a single URL via url.
Output (real run)
{"url": "https://en.wikipedia.org/wiki/Web_crawler","ok": true,"status": 200,"indexable": true,"metaRobots": "max-image-preview:standard","xRobotsTag": null,"directives": { "noindex": false, "nofollow": false, "all": ["max-image-preview"] },"canonical": "https://en.wikipedia.org/wiki/Web_crawler","selfCanonical": true,"reasons": ["canonical is self-referential."]}
Wikipedia's Special:Search page in the same run comes back indexable: false with the meta-robots noindex caught and named — asserted by our release test.
Example
A real run on 2026-10-31 with the input this page's form is prefilled with:
{"url": "https://github.com","urls": ["https://github.com","https://www.cloudflare.com"]}
It finished in 2 seconds and returned 2 results. It was charged for 2 page-scanned events ($0.004 on the Free plan). The first result (long text and lists cut for display):
{"url": "https://github.com","ok": true,"finalUrl": "https://github.com/","status": 200,"indexable": true,"metaRobots": null,"metaGooglebot": null,"xRobotsTag": null,"directives": {"noindex": false,"nofollow": false,"noarchive": false,"nosnippet": false,"noimageindex": false,"none": false,"all": []},"canonical": "https://github.com/","selfCanonical": true,"reasons": ["No robots restrictions found — defaults to indexable & followable.","canonical is self-referential."]}
Pricing
$0.002 per page audited — all three signal sources + canonical in one price. No start fee. Unreachable URLs are never charged.
Measured against store incumbents (2026-08-07): burly_bat/site-launch-migration-audit charges $0.007 per URL, 213x/sitemap-seo-auditor $0.01 start + $0.001 per item. A 50-page release check here costs $0.10.
Honest limits
- On-page + header signals only: robots.txt crawl rules are a separate mechanism — pair with our Robots Rule Tester for the crawl half (a page can be indexable by meta but blocked by robots.txt, and vice versa).
- Static fetch: a
noindexinjected by client-side JavaScript after load is not visible (rare, and itself a bad practice — Google may miss it too). - "Indexable" means no blocking directive was found — it does not guarantee Google will index or rank the page.
FAQ
Why check X-Robots-Tag when I can view source? Because it is an HTTP header, not HTML — CDNs, reverse proxies and app middleware add it invisibly. It carries exactly the same force as meta robots and is the most-missed cause of "why did we vanish from Google".
What does a wrong canonical look like here?
selfCanonical: false with the reason naming the target URL — the page is telling search engines "index that other URL instead of me".
Can I gate deploys on this?
Yes — run your critical URLs post-deploy and fail the pipeline when any record has indexable: false (or a non-self canonical you didn't expect). That single check would have prevented most public staging-noindex incidents.
Does it follow redirects?
Yes — the audit applies to the final URL after redirects, and finalUrl tells you where it landed.
Why did some rows come back ok: false?
DNS failure or timeout — nothing was audited. Recorded, never charged.
Use from code or AI agents
curl -s "https://api.apify.com/v2/acts/EliAI~webpage-meta-robots-auditor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \-X POST -H 'Content-Type: application/json' \-d '{"urls": ["https://yoursite.com/", "https://yoursite.com/key-landing-page"]}'
Agents: connect Apify MCP and call the EliAI/webpage-meta-robots-auditor tool.
- Capability: audit indexability of one or many URLs — meta robots, meta googlebot, X-Robots-Tag, canonical, verdict + reasons
- Required input:
url(string) orurls(array) - Returns: one record per URL;
indexable+reasonsare the verdict - Bounded: 50 URLs per run; failures isolate per URL
- Side effects: none (one GET per page)
Who uses it
- SEO specialists catching a staging
noindexthat shipped to production - Developers finding
X-Robots-Tag: noindexheaders a CDN adds outside the HTML - Migration leads checking canonicals do not point at the old domain
- Agencies diagnosing why client pages are not indexed
- Content teams verifying new landing pages are indexable before a launch
Changelog
- 2026-11-01: Page text only: an Example section from a real run. Code, output and price are unchanged.