Pricing

from $4.90 / 1,000 robots file checkeds

Go to Apify Store

Robots.txt Auditor

Try for free

Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

Pricing

from $4.90 / 1,000 robots file checkeds

Rating

0.0

(0)

Developer

junipr

Actor stats

Bookmarked

Total users

Monthly active users

14 days ago

Last modified

Inputs

domains
maxDirectives
includeReport
maxChargeUsd

{
  "domains": [
    {
      "domain": "www.iana.org",
      "statusCode": 200,
      "robotsTxt": "User-agent: *\nDisallow:"
    }
  ],
  "maxDirectives": 2,
  "includeReport": true,
  "maxChargeUsd": 1
}

The bundled exact input is a public observation of https://www.iana.org/robots.txt captured at 2026-07-12T05:03:20Z; the actor analyzes the supplied snapshot and does not refetch it.

Dataset

The actor emits structured, actor-specific evidence rows. Empty source input, or a comparison with no selected changes, produces no dataset rows.

Billing

This actor uses pay-per-event billing with platform usage included in the configured event prices. Configured events are:

actor-start
robots-file-checked
rule-parsed
finding-emitted
audit-report-generated

Every dataset row and report artifact is charge-gated. A zero-dollar cap stops before work with no paid output.

Use Cases

Find a site-wide robots.txt block
Review Googlebot-specific directives
Export a robots.txt sitemap directive
Audit multiple robots user agents
Verify an explicit robots.txt allow rule

Limitations

The actor analyzes supplied robots.txt observations and does not fetch URLs. It parses User-agent, Allow, Disallow, and Sitemap directives; unsupported extensions remain outside the dataset.

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Robots Txt Analyzer

zerobreak/robots-txt-analyzer

Robots txt analyzer that fetches and parses crawl rules from any website in bulk, so SEO teams and developers can audit blocked paths, user agents, and sitemap locations across hundreds of domains without manual work.

ZeroBreak

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

Robots.txt & Sitemap Discovery API

213x/robots-sitemap-discovery

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

robots.txt Analyzer

eliai/robots-txt-analyzer

Anthony Snider

Robots.txt & Sitemap Auditor

phoenix2810/robots-sitemap-auditor

Audit a website's robots.txt directives and sitemap discovery in one fast API call. Detect blocked paths, missing sitemaps, and crawlability issues. Built for SEO agencies, technical SEO consultants, and site migration QA teams.

Sanskar Jaiswal

Robots.txt Validator - Check Rules, Sitemaps & Crawl Directives

scrappy_garden/robots-txt-validator

Validate robots.txt for one or more websites: fetches /robots.txt per host, parses directive groups (User-agent/Allow/Disallow/Crawl-delay/Sitemap), reports common errors and warnings, and can test URLs against the chosen User-Agent.