Robots Rule Tester — Would Google Crawl This URL?

Test any URL against its site's robots.txt with real Google semantics — longest-match rule, Allow wins ties, most-specific user-agent group, * and $ wildcards — and get the verdict plus the exact rule that decided it. Up to 100 URLs per run (robots.txt fetched once per origin), online, by API, or as an agent tool via Apify MCP.

"Is this page blocked for Googlebot?" is answered wrong constantly, because robots.txt evaluation has real rules people eyeball incorrectly: Allow: /search/about beats Disallow: /search for /search/about (longer match), ties go to Allow, and a GPTBot group overrides * for GPTBot. This actor implements the spec and shows its work.

What you get

• allowed — the verdict for your chosen userAgent (default *; try Googlebot, GPTBot, ClaudeBot...)
• matchedRule — the exact allow/disallow line that decided it (null = no rule matched)
• matchedUserAgentGroup — which UA group applied
• sitemaps — every Sitemap: URL declared in robots.txt
• crawlDelay, robotsTxtUrl, robotsStatus — plus honest notes when robots.txt is missing (HTTP 4xx = everything allowed, stated explicitly)
• Fail-soft: an invalid URL never fails the run — {ok: false, error}, never charged.

Input

{ "urls": ["https://yoursite.com/private/page", "https://yoursite.com/blog/post"], "userAgent": "Googlebot" }

Or a single URL via url.

Output (real run)

{
  "url": "https://www.google.com/search?q=test",
  "ok": true,
  "userAgent": "*",
  "allowed": false,
  "matchedRule": { "type": "disallow", "path": "/search" },
  "matchedUserAgentGroup": ["*"],
  "sitemaps": ["https://www.google.com/sitemap.xml"],
  "robotsStatus": 200,
  "testedPath": "/search?q=test"
}

In the same run, https://www.google.com/search/about comes back allowed: true via the longer Allow: /search/about rule — the longest-match subtlety, demonstrated live and asserted by our release test.

Pricing

$0.0008 per URL tested. No start fee. Invalid URLs are never charged; bulk URLs on one site share a single robots.txt fetch.

Measured against store incumbents (2026-08-07): scrapeworks/robots-txt charges $0.001 per item, apage/robots-txt-audit $0.005 start + $0.005 per item, alizarin robots_txt_analysis $0.1 start + $0.05 per analysis.

Honest limits

• This evaluates the published robots.txt exactly as a spec-following crawler would. It cannot tell you whether a given bot actually honors robots.txt (some don't), nor does it check meta-robots/X-Robots-Tag — pair with our webpage-meta-robots-auditor for the on-page half of indexability.
• UA matching is substring-based on the group token (per de-facto crawler behavior); pass the product token (e.g. Googlebot, not a full UA string) for clearest results.
• crawl-delay is reported when declared; Google ignores it, others honor it — reported, not judged.

FAQ

Why do I need a tester instead of reading robots.txt? Because of precedence: wildcards, longest-match, Allow-vs-Disallow ties, and UA group specificity interact. The matchedRule field shows exactly which line won, so you can fix the file with confidence.

Can I check AI-crawler access? Yes — run the same URLs with userAgent: "GPTBot", "ClaudeBot", "CCBot", etc., and compare verdicts against *.

What happens when there's no robots.txt? HTTP 4xx on /robots.txt means everything is allowed — the record says so explicitly in note rather than silently passing.

Does it validate my whole robots.txt? It answers per-URL verdicts, which is what audits actually need: feed it your critical URLs and assert allowed in CI before every deploy.

Why did some rows come back ok: false? The input could not be parsed as a URL. Recorded, never charged.

Use from code or AI agents

curl -s "https://api.apify.com/v2/acts/EliAI~robots-rule-tester/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST -H 'Content-Type: application/json' \
  -d '{"urls": ["https://yoursite.com/checkout", "https://yoursite.com/blog"], "userAgent": "Googlebot"}'

Agents: connect Apify MCP and call the EliAI/robots-rule-tester tool.

• Capability: test one or many URLs against robots.txt with Google longest-match semantics for any user agent
• Required input: url (string) or urls (array); optional userAgent (default *)
• Returns: one record per URL; allowed + matchedRule are the verdict
• Bounded: 100 URLs per run, one robots.txt fetch per origin; failures isolate per URL
• Side effects: none (GET robots.txt only — the tested URLs are never fetched)

For AI agents

This Actor is built to be called by software, not just by people.

• Mount it directly as an MCP tool — no Store search, no ranking, just this one tool: https://mcp.apify.com/?actors=eliai/robots-rule-tester
• Or call it over HTTP and get the results in the same request: POST https://api.apify.com/v2/acts/eliai~robots-rule-tester/run-sync-get-dataset-items
• Pay with x402, without an Apify account. This Actor is whitelisted for agentic payments, so an agent holding USDC on Base can buy a prepaid token and spend it here. The minimum purchase is $1, the token balance is an absolute spending cap, and it expires 14 days after purchase.
• Costs are predictable before you call. Pricing is pay-per-event (see Pricing above), so an agent can budget a run in advance instead of discovering the bill afterwards.
• Send only the field you mean. If you pass the bulk field, it is used on its own; the single-value field is a fallback, never merged into your request. You are charged for the items you sent and nothing else.