AI Crawler Access Auditor

See which major AI crawlers a public site allows or blocks on the paths you care about, and whether the site publishes usable llms.txt / llms-full.txt policy files. The Actor returns rule-level evidence rather than crawling page content or guessing from an AI-readiness score.

Minimal input

{
  "sites": ["https://example.com"],
  "paths": ["/", "/blog/"]
}

Bare domains are treated as HTTPS. Each supplied URL is reduced to its origin; use paths for the robots.txt decisions you want to test.

What one result contains

One Dataset row represents one selected site:

• robots.txt status/outcome and declared Sitemap URLs
• 14 fixed AI crawler tokens, each with allow/block evidence for every requested path
• matched user-agent group, Allow/Disallow rule, pattern, and source line
• llms.txt / llms-full.txt presence and basic structure (H1, summary, sections, links)
• explicit findings and an error when the bounded audit is incomplete

The fixed crawler set includes GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent, and Amazonbot.

Scope and limits

• 1-100 sites, 1-10 paths, at most 10 sites audited concurrently
• public HTTP(S) only; URL credentials and nonstandard ports are rejected
• every redirect and resolved IP passes the shared public-target safety checks
• no browser, login, cookies, proxy configuration, arbitrary request headers, or paid API
• no TDM/meta/homepage signals, anti-bot impersonation, site content crawling, or history/monitor mode in v0.1
• robots.txt and llms files are bounded to 1 MB each; run-level request/byte ceilings apply
• a missing robots.txt is reported as allow-all for this audit; an unreachable/invalid robots file yields unknown decisions
• 404/410 llms files are reported as missing; HTML masquerading as llms.txt is not treated as a valid file

The Actor evaluates published policy files. It does not prove that a firewall or CDN will actually admit the named crawler.

Billing

The billable unit is one successfully completed unique site audit (domain-audited) plus Apify's normal run-start event. Invalid or incomplete sites produce diagnostic rows without the domain-audited charge. Billable capacity is selected before network work so the user's maximum charge limit can reduce the number of sites fetched.

Current launch price:

• $0.00005 per run start
• $0.00025 per successfully audited site ($0.25 per 1,000 successful sites)

A one-site successful run therefore costs $0.00030 before any pricing tier discount. In the accepted 10-site private cloud sample, the observed owner-side platform cost was about $0.01955 per 1,000 successful sites. Apify's current standard developer payout is 80% of PPE revenue minus platform costs.

Private PPE readback confirmed these exact event names/prices. Budget run JlPDo6kPLSu2msFCb requested three sites with maxTotalChargeUsd=0.00030 and selected/fetched exactly one site before charging one domain-audited event plus the run-start event.

Verified private example

Private build YjUsyGG2u6LWNVcSH, run yxeI9aEM6l6REs4r2, input:

{
  "sites": ["https://apify.com"],
  "paths": ["/", "/store/"]
}

Returned one successful row with:

• robotsOutcome: "parsed"
• 14 crawler policies evaluated; all 14 allowed on both requested paths
• llmsTxtOutcome: "found"
• llmsFullTxtOutcome: "missing"
• finding LLMS_FULL_TXT_MISSING

This is a source-bound example from the private acceptance run, not a guarantee that Apify's policy files will remain unchanged.

Cost / reliability guard

Pause expansion and re-check pricing or scope if observed successful-site platform cost rises above $0.08 per 1,000, or if stable-public-site acceptance falls below 95% once at least 20 representative observations exist. Support burden is tracked separately; persistent manual support beyond the intended low-maintenance profile also triggers a scope/price review.

Support

Open an Issue on this Actor's Apify page with the run ID and a non-sensitive reproduction. Never post access tokens, cookies, internal hosts, or private URLs.