Pricing

from $0.99 / 1,000 results

Try for free

Go to Apify Store

Noindex Directive Validator

Try for free

Noindex checker that scans URLs for meta robots and X-Robots-Tag headers, so SEO teams can find pages accidentally blocked from indexing before they drop out of search results.

Pricing

from $0.99 / 1,000 results

Rating

0.0

(0)

Developer

ZeroBreak

Actor stats

Bookmarked

Total users

Monthly active users

19 days ago

Last modified

Categories

SEO tools

URL

url

Optional

Single URL to check for noindex directives.

Type:string

Example:

https://apify.com/about

URLs

urls

Optional

List of URLs to check. Provide one URL per line. Combined with the single URL field above.

Type:array

Example:

[
  "https://apify.com",
  "https://apify.com/about"
]

Max URLs

maxUrls

Optional

Maximum number of URLs to process in a single run. Capped at 1000.

Type:integer

Minimum:1

Maximum:1000

Default:100

Request timeout (seconds)

requestTimeoutSecs

Optional

Timeout in seconds for each individual HTTP request.

Type:integer

Minimum:5

Maximum:120

Default:30

Proxy configuration

proxyConfiguration

Optional

Select proxies to use for requests. Helps avoid IP blocking and rate limits. Datacenter proxies are fastest; Residential proxies are harder to detect.

Type:object

Robots-Noindex Conflict Checker

junipr/robots-noindex-conflict-checker

Detect conflicts between robots.txt blocking and page-level meta/X-Robots noindex signals.

junipr

SEO Indexability Checker

q_services/seo-indexability-checker

Audit pages for noindex, X-Robots-Tag, canonicals, titles, meta descriptions and H1 signals.

Q Services

Robots Meta X-Robots Auditor

junipr/robots-meta-xrobots-auditor

Audit page-level robots directives from meta tags and X-Robots-Tag headers and detect conflicting indexability signals.

junipr

Indexability Audit

zerobreak/indexability-audit

Indexability audit tool that checks robots.txt, meta robots tags, X-Robots-Tag headers, and canonical URLs for any list of pages, so SEO teams know which ones Google can actually crawl and index.

ZeroBreak

Indexability Audit

conserving_mastodon/indexability-audit

Can search engines index this page, and if not, why? Checks the meta robots tag, X-Robots-Tag header, canonical link, and robots.txt in one call and returns a plain verdict with the exact directive responsible. Charged only on completed audits.

Chris Arsenault

Meta Robots Auditor - Is This Page Actually Indexable?

eliai/webpage-meta-robots-auditor

Indexability verdict per URL from ALL the signals: meta robots, meta googlebot, the X-Robots-Tag header (the one checkers miss), canonical + self-canonical check — with the exact reasons. Catches shipped staging noindex and CDN-injected headers. $0.002 per page vs $0.007-$0.011 measured incumbents.

Broke to Built

Indexability and Canonical Audit Crawler

kayhermes/indexability-and-canonical-audit-crawler

Deterministic local indexability + canonical audit of explicit public HTTP(S) URLs from static HTML: robots.txt enforcement, meta/X-Robots-Tag noindex, HTTP status, redirect chains, and declared canonical analysis.

Khoa Nguyen

Website SEO Audit Crawler — On-Page Analyzer

logiover/website-seo-audit-crawler

Crawl a whole website and export an on-page SEO audit to CSV/JSON: meta tag checker, canonical & noindex checker, schema.org type extractor. No browser.

Logiover

103

Robots Txt Analyzer

zerobreak/robots-txt-analyzer

Robots txt analyzer that fetches and parses crawl rules from any website in bulk, so SEO teams and developers can audit blocked paths, user agents, and sitemap locations across hundreds of domains without manual work.

ZeroBreak

Check Page Sizes

zerobreak/check-page-sizes

Page size checker that crawls any website and flags HTML pages over 2MB or PDFs over 64MB, the exact thresholds where Google stops indexing — so SEO teams can fix oversized files before they drop from search.

ZeroBreak