Pricing

from $4.90 / 1,000 page auditeds

Go to Apify Store

Robots-Noindex Conflict Checker

Try for free

Detect conflicts between robots.txt blocking and page-level meta/X-Robots noindex signals.

Pricing

from $4.90 / 1,000 page auditeds

Rating

0.0

(0)

Developer

junipr

Actor stats

Bookmarked

Total users

Monthly active users

22 days ago

Last modified

Categories

SEO tools

Robots and noindex evidence

inputItems

Optional

Pages with robots.txt, headers, and HTML observations.

Type:object[]

Default:

[
  {
    "url": "https://en.wikipedia.org/wiki/Special:UserLogin",
    "robotsUrl": "https://en.wikipedia.org/robots.txt",
    "statusCode": 200,
    "robotsTxt": "User-agent: *\nDisallow: /w/\nDisallow: /wiki/Special:",
    "pageHeaders": {
      "content-type": "text/html; charset=UTF-8"
    },
    "html": "<meta name=\"robots\" content=\"noindex,nofollow,max-image-preview:standard\">"
  }
]

Maximum pages

maxPages

Optional

Hard cap for paid page rows.

Type:integer

Minimum:1

Maximum:5000

Default:1

Include report

includeReport

Optional

Create charged Markdown and JSON report artifacts.

Type:boolean

Default:true

Maximum charge USD

maxChargeUsd

Optional

Stop before an event would exceed this run budget.

Type:number

Minimum:0

Maximum:100

Default:1

Noindex Directive Validator

zerobreak/noindex-directive-validator

Noindex checker that scans URLs for meta robots and X-Robots-Tag headers, so SEO teams can find pages accidentally blocked from indexing before they drop out of search results.

ZeroBreak

SEO Indexability Checker

q_services/seo-indexability-checker

Audit pages for noindex, X-Robots-Tag, canonicals, titles, meta descriptions and H1 signals.

Q Services

robots.txt Analyzer

eliai/robots-txt-analyzer

Anthony Snider

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Robots Indexability Audit

toronto_777/robots-indexability-audit

Audit public robots.txt, sitemap declarations, homepage robots directives, and crawler allow/block signals for SEO and AI visibility checks.

Steven Feng

Robots.txt Validator

predictable_function/my-actor-3

List of website base URLs whose robots.txt files will be validated

riya rawat

5.0

SEO Audit: On-Page, Redirects & AI Crawler Access

aiqlabs/seo-audit-tool

Audits pages for on-page and technical SEO, and adds four checks general auditors skip: AI crawler access in robots.txt, llms.txt, redirects that drop the path, and noindex sent via X-Robots-Tag.

Ai-Q Labs

Store/Crawler Surface Preflight

checksmithcats/store-crawler-surface-preflight

Cross-check public robots.txt, sitemap, noindex, llms.txt, policy links, and contact signals from a storefront URL. Reads only public URLs; no account access.

Checksmith Cats

AI Crawler Policy Drift Monitor

checksmithcats/ai-crawler-policy-drift-monitor

Tracks changes in public robots.txt, noindex, sitemap, and llms.txt signals against a stored baseline. Public crawler-facing signals only.

Checksmith Cats

Indexability Audit

zerobreak/indexability-audit

Indexability audit tool that checks robots.txt, meta robots tags, X-Robots-Tag headers, and canonical URLs for any list of pages, so SEO teams know which ones Google can actually crawl and index.

ZeroBreak