robots.txt Analyzer

Parse and audit any site's robots.txt in one call — user-agent groups, allow/disallow rules, declared sitemaps, and common issues. No API key, pay per site.

▶ Live on the Apify Store: https://apify.com/eliai/robots-txt-analyzer — run it instantly, or call it as an agent tool via Apify MCP.

What it does

• Fetches and parses /robots.txt
• Lists user-agent groups and their allow/disallow rules
• Extracts all declared Sitemap URLs
• Flags issues: no sitemap declared, Disallow: / for * (blocks the whole site), missing/HTML robots.txt

Input

{ "url": "https://example.com" }

or bulk:

{ "urls": ["https://a.com", "https://b.com"], "maxUrls": 25 }

Output (per site)

{
  "url": "https://example.com",
  "robotsUrl": "https://example.com/robots.txt",
  "hasRobots": true,
  "userAgentGroups": 3,
  "totalRules": 12,
  "sitemaps": ["https://example.com/sitemap.xml"],
  "blocksAllCrawlers": false,
  "issues": [],
  "summary": "3 groups, 1 sitemap(s), 0 issue(s)"
}

Use cases

SEO audits, crawl-control reviews, and catching accidental site-wide blocks — by hand or wired into an agent. Pairs with our Sitemap Analyzer and SEO & Tech Auditor.

Pricing

$0.02 per site analyzed — flat per robots.txt, billed as the robots-analyzed event. Bulk runs are capped by maxUrls, which is also your budget cap.

FAQ

How do I check if robots.txt is blocking my site from Google? Run your domain — blocksAllCrawlers tells you instantly if a Disallow: / under User-agent: * is shutting out every crawler, the single most damaging robots.txt mistake.

Does it find the sitemaps a site declares? Yes — every Sitemap: line is extracted into the sitemaps array, which is also how you discover sitemaps that live at non-standard paths.

What if the site has no robots.txt at all? You get hasRobots: false with the issues list noting it. No robots.txt means crawlers assume everything is allowed — worth knowing either way.

What does it flag as an issue? No sitemap declared, Disallow: / for all user agents (blocks the whole site), and a missing robots.txt or one that serves HTML instead of plain text.

Can I audit robots.txt for many sites at once? Yes — pass an array in urls (capped by maxUrls) and get one record per site.

Related actors

Blocking a URL that redirects hides the destination from crawlers entirely — Redirect Chain Checker traces every hop so you know what is on the other side of the rule. See also Sitemap Analyzer and Website SEO Tech Auditor.

For AI agents

This Actor is built to be called by software, not just by people.

• Mount it directly as an MCP tool — no Store search, no ranking, just this one tool: https://mcp.apify.com/?actors=eliai/robots-txt-analyzer
• Or call it over HTTP and get the results in the same request: POST https://api.apify.com/v2/acts/eliai~robots-txt-analyzer/run-sync-get-dataset-items
• Pay with x402, without an Apify account. This Actor is whitelisted for agentic payments, so an agent holding USDC on Base can buy a prepaid token and spend it here. The minimum purchase is $1, the token balance is an absolute spending cap, and it expires 14 days after purchase.
• Costs are predictable before you call. Pricing is pay-per-event (see Pricing above), so an agent can budget a run in advance instead of discovering the bill afterwards.
• Send only the field you mean. If you pass the bulk field, it is used on its own; the single-value field is a fallback, never merged into your request. You are charged for the items you sent and nothing else.