AI Crawler Access Auditor

Audit how a website treats AI crawlers. Given public website domains or root URLs, the Actor fetches robots.txt, llms.txt, and the sitemaps declared in robots.txt — safely, with every redirect target validated as a public address — and deterministically evaluates whether each named AI crawler user agent may access each test path. Every decision cites the exact matched Allow/Disallow rule and the robots group it came from. There is no opaque score and no AI-generated recommendation: you get the rules, the match, and the fetch statuses.

The package reuses the SSRF-safe HTTP and URL validation core of the sitemap_health_monitor Actor in this repository; that package and its tests are unchanged.

What it checks

• robots.txt fetch status and content, evaluated per crawler per test path with RFC 9309 semantics: case-insensitive substring user-agent matching, most-specific group selection, merging of equally specific groups, * wildcards, $ end anchors, longest-rule match, and allow-on-tie precedence
• The exact matched rule (matchedRule, matchedRuleType) and the user-agent token of the selected group (matchedUserAgent)
• llms.txt presence and fetch status
• Sitemap declarations from robots.txt and their reachability (bounded probe of the first 10 declarations)

Default crawler user agents: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot. You can replace them with a custom list.

Input

{
  "startUrls": ["example.com", "https://docs.example.org"],
  "crawlerUserAgents": ["GPTBot", "ClaudeBot"],
  "testPaths": ["/", "/pricing"],
  "maxSites": 100,
  "concurrency": 10,
  "timeoutSecs": 20,
  "maxRedirects": 5
}

Field	Description
startUrls	1-100 public website domains or root URLs. Each entry is normalized to its origin (https://example.com/blog becomes https://example.com); a missing scheme defaults to https. Duplicates are removed. Required.
crawlerUserAgents	Optional 1-30 crawler user-agent tokens. Defaults to the eight AI crawlers listed above. Tokens are 1-64 characters of letters, numbers, dot, underscore, hyphen; duplicates (case-insensitive) are removed.
testPaths	Optional 1-10 site-relative paths (starting with /, up to 200 characters) evaluated for every crawler. Default ["/"].
maxSites	Hard cap on distinct sites audited. Default 100; range 1-100. Inputs with more distinct sites than maxSites are rejected before any network work.
concurrency	Concurrent site audits. Default 10; range 1-30.
timeoutSecs	Per-request timeout. Default 20 seconds; range 2-60.
maxRedirects	Maximum redirect hops followed per fetch. Default 5; range 0-5. Every redirect target is validated as public before it is followed.

URLs with credentials, non-HTTP(S) schemes, empty hosts, or targets resolving to non-public addresses are rejected before or during the run.

Output

Every run writes one summary record, one site-result record per audited site, and one crawler-result record per site × crawler × test path to the default dataset.

Summary:

{
  "recordType": "summary",
  "checkedAt": "2026-08-07T09:15:00+00:00",
  "siteCount": 2,
  "crawlerCount": 8,
  "pathCount": 1,
  "checkCount": 16,
  "allowedCount": 10,
  "disallowedCount": 5,
  "unknownCount": 1,
  "sitesWithRobots": 2,
  "sitesWithLlmsTxt": 1,
  "sitesWithReachableSitemap": 1,
  "errors": 1
}

Site result:

{
  "recordType": "site-result",
  "siteUrl": "https://example.com",
  "robotsStatus": "ok",
  "robotsPresent": true,
  "llmsTxtStatus": "ok",
  "llmsTxtPresent": true,
  "sitemapUrls": ["https://example.com/sitemap.xml"],
  "reachableSitemapCount": 1,
  "issueCodes": [],
  "error": null,
  "checkedAt": "2026-08-07T09:15:00+00:00"
}

Crawler result:

{
  "recordType": "crawler-result",
  "siteUrl": "https://example.com",
  "crawler": "GPTBot",
  "path": "/",
  "allowed": false,
  "matchedRule": "Disallow: /",
  "matchedRuleType": "disallow",
  "matchedUserAgent": "gptbot",
  "robotsStatus": "ok",
  "robotsPresent": true,
  "llmsTxtStatus": "ok",
  "llmsTxtPresent": true,
  "sitemapUrls": ["https://example.com/sitemap.xml"],
  "reachableSitemapCount": 1,
  "issueCodes": [],
  "error": null,
  "checkedAt": "2026-08-07T09:15:00+00:00"
}

robotsStatus and llmsTxtStatus are one of ok, not-found, blocked (robots.txt only, HTTP 401/403), or error (HTTP 5xx, 429, redirect cap, or fetch failure). robotsPresent is true only when a robots.txt document was successfully retrieved.

Access decisions follow the robots.txt fetch state:

• ok — allowed is the deterministic rule evaluation, with matchedRule, matchedRuleType, and matchedUserAgent set when a rule matched.
• not-found (404 and other 4xx) — allowed is true with no matched rule: there are no robots restrictions, stated explicitly via robotsStatus.
• blocked (401/403) — allowed is conservatively false with issue robots-access-denied.
• error (5xx, 429, or fetch failure) — allowed is null (unknown). The Actor never fabricates an allowed or blocked verdict when the rules could not be read.

Issue codes: robots-not-found, robots-access-denied, robots-unreachable, llms-txt-error, sitemap-unreachable.

Pricing

The Actor uses Apify pay-per-event pricing with the site-audited charge event. When monetization is enabled, users are charged $0.003 per site audited ($3 per 1,000 sites). One site-audited event covers one site: its robots.txt, llms.txt, bounded sitemap reachability probes, and the deterministic evaluation of every crawler × test path.

Apify platform usage (compute units and other resources consumed by the run) may still be shown to users according to their plan and Apify's pricing rules, as described in the Actor's listing.

The Actor respects the run's maximum total charge: before any network work it trims the site list to the chargeable prefix; if no site can be charged, it stops with a clear error.

Final pricing is configured in the Apify Store listing and may change subject to Apify's pricing-change notice rules.

Security and privacy

• Only public HTTP(S) targets are accepted.
• URL credentials, localhost, and non-public, loopback, link-local, multicast, unspecified, or reserved addresses are rejected.
• Every redirect target is resolved and validated before it is followed; a redirect to a private address fails that fetch with an error state instead of being fetched.
• Site counts, crawler counts, path counts, concurrency, redirects, response bytes, sitemap probes, and timeouts are all capped before or during network work.
• The Actor does not use a browser, proxy, LLM, external database, or third-party analytics service.
• Each run is stateless; results live only in the run's default dataset, subject to the retention and access settings of the Apify account running the Actor.

Do not place secrets, private URLs, or personal data in any input field.

Limitations

• Access evaluation reflects robots.txt rules as served at audit time; it does not execute JavaScript, render pages, or verify crawler behavior in practice.
• Only sitemaps declared in robots.txt are probed; a missing or unreadable robots.txt yields no sitemap declarations.
• At most 20 sitemap declarations are recorded and the first 10 are probed for reachability.
• Network failures and rate limits are reported as per-site error states; they are not automatically retried indefinitely.
• The Actor does not send notifications itself. Use Apify schedules, webhooks, or an automation platform.

Support

For reproducible issues, open an issue from the Actor page and include the Apify run ID, sanitized input, expected result, and affected public URL. Do not include API tokens or private data.

This Actor reports robots.txt access rules; it does not provide legal advice, crawler-compliance guarantees, or uptime guarantees.