Pricing

from $4.90 / 1,000 robots meta xrobots auditor page auditeds

Go to Apify Store

Robots Meta X-Robots Auditor

Try for free

Audit page-level robots directives from meta tags and X-Robots-Tag headers and detect conflicting indexability signals.

Pricing

from $4.90 / 1,000 robots meta xrobots auditor page auditeds

Rating

0.0

(0)

Developer

junipr

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Categories

SEO tools

Start Urls

startUrls

Optional

Values for startUrls.

Type:string[]

Default:

[]

Sitemap Urls

sitemapUrls

Optional

Values for sitemapUrls.

Type:string[]

Default:

[]

Html Inputs

htmlInputs

Optional

Supplied htmlInputs records for offline processing.

Type:object[]

Default:

[
  {
    "sourceUrl": "https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag",
    "html": "<!doctype html><html><head><title>Google robots directive reference capture</title><meta name=\"robots\" content=\"noindex, follow\"></head><body><main><p>Robots meta directive example captured from Google Search documentation.</p></main></body></html>"
  }
]

Header Inputs

headerInputs

Optional

Supplied headerInputs records for offline processing.

Type:object[]

Default:

[]

Allowed Domains

allowedDomains

Optional

Values for allowedDomains.

Type:string[]

Default:

[]

Max Pages

maxPages

Optional

Numeric limit for maxPages.

Type:number

Minimum:1

Maximum:100

Default:1

Max Depth

maxDepth

Optional

Numeric limit for maxDepth.

Type:number

Minimum:0

Maximum:5

Default:0

Include Bot Specific Directives

includeBotSpecificDirectives

Optional

Enable includeBotSpecificDirectives.

Type:boolean

Default:true

Include Header Directives

includeHeaderDirectives

Optional

Enable includeHeaderDirectives.

Type:boolean

Default:true

Detect Conflicts

detectConflicts

Optional

Enable detectConflicts.

Type:boolean

Default:true

Request Delay Ms

requestDelayMs

Optional

Numeric limit for requestDelayMs.

Type:number

Minimum:0

Maximum:5000

Default:0

Timeout Ms

timeoutMs

Optional

Numeric limit for timeoutMs.

Type:number

Minimum:500

Maximum:30000

Default:10000

Maximum charge (USD)

maxChargeUsd

Optional

Stop before paid output would exceed this pay-per-event PPE limit.

Type:number

Minimum:0

Maximum:1000

Default:1

Indexability Audit

zerobreak/indexability-audit

Indexability audit tool that checks robots.txt, meta robots tags, X-Robots-Tag headers, and canonical URLs for any list of pages, so SEO teams know which ones Google can actually crawl and index.

ZeroBreak

Robots Indexability Audit

toronto_777/robots-indexability-audit

Audit public robots.txt, sitemap declarations, homepage robots directives, and crawler allow/block signals for SEO and AI visibility checks.

Steven Feng

Robots-Noindex Conflict Checker

junipr/robots-noindex-conflict-checker

Detect conflicts between robots.txt blocking and page-level meta/X-Robots noindex signals.

junipr

SEO Indexability Checker

q_services/seo-indexability-checker

Audit pages for noindex, X-Robots-Tag, canonicals, titles, meta descriptions and H1 signals.

Q Services

Sitemap & Robots SEO Index Auditor

glowing_glove/sitemap-robots-index-auditor

Audit robots.txt, sitemap discovery, indexability signals, canonical tags, and SEO crawl readiness for business websites.

Ushba Khan

Site Launch & Migration Technical SEO Audit

burly_bat/site-launch-migration-audit

Bulk technical launch/indexability audit for a list of URLs, a sitemap or a chained dataset. Checks HTTP status, redirects, robots.txt, meta robots, X-Robots-Tag, canonical, on-page tags, JSON-LD and hreflang, then returns a PASS/WARN/FAIL verdict per URL plus a go/no-go rollup.