AI Crawler Policy Auditor

Store Positioning

Store title: AI Crawler Policy Auditor

Short description: Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

SEO title: AI Crawler Policy Auditor — technical SEO, web, and domain audit

SEO description: Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness. Use it to find crawlability, indexability, security, metadata, and page-quality issues with evidence-backed rows and audit reports.

Categories: SEO_TOOLS

Keywords: ai, crawler, policy, auditor, web/domain audit

Fixed-Inclusive PPE Pricing

This actor uses pay-per-event pricing. Event prices include Apify platform usage; users are not expected to pay a separate platform-usage pass-through charge for the configured pricing model.

• Tier: W1 — Web/domain audit
• Primary event: page-audited at $0.00490 base
• Default max charge: $10.00
• Store discounts: FREE/BRONZE base, SILVER discounted, GOLD deepest approved discount

Event set:

• actor-start: base $0.00500, GOLD $0.00400. Ai Crawler Policy Auditor: charged when actor start is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• page-audited: base $0.00490, GOLD $0.00392. Ai Crawler Policy Auditor: charged when page audited is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• rule-parsed: base $0.00372, GOLD $0.00298. Ai Crawler Policy Auditor: charged when rule parsed is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• finding-emitted: base $0.00372, GOLD $0.00298. Ai Crawler Policy Auditor: charged when finding emitted is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• audit-report-generated: base $0.05000, GOLD $0.04000. Ai Crawler Policy Auditor: charged when audit report generated is completed. The price includes Apify platform usage; no separate usage pass-through is intended.

Public Task Concepts

• Audit AI Crawler Policy controls with capped supplied targets
• Find high-priority AI Crawler Policy issues before release
• Validate AI Crawler Policy evidence from supplied pages
• Prioritize AI Crawler Policy fixes with severity and proof
• Export AI Crawler Policy QA rows for client review

Audit robots.txt, llms.txt, X-Robots-Tag, and meta robots signals for AI crawler access, blocking, and site-readiness.

What This Actor Does

AI Crawler Policy Auditor checks whether major AI and search crawler user agents are allowed, blocked, partially blocked, or unmanaged for a target path. It parses supplied robots.txt snapshots by default, can compare previous and current policies, detects page-level robots restrictions such as noindex and noai, and writes one structured dataset row per audited target.

The default run is local-first: it uses an inline fixture, makes no network calls, and writes summary diagnostics to the key-value store. Optional live robots fetching is available only when fetchUrls is explicitly set to true, with strict target, timeout, and byte caps.

What This Actor Does Not Do

• It does not guarantee search ranking, AI citation, traffic, revenue, or compliance outcomes.
• It does not provide legal, medical, or financial advice.
• It does not collect sensitive personal data.
• It does not crawl full websites or render pages in a browser.
• It does not fetch live URLs unless fetchUrls is explicitly enabled.

Operational Status

• Package status: local-green.
• Cloud status: cloud-deferred until the main actor publication workflow pushes and verifies the actor remotely.
• Default input: inline robots fixture with fetchUrls: false.
• Store icon workflow: actor-local icon.png is included; shared landing icon manifests and live Console upload are intentionally left to the main agent.

Best Use Cases

• Confirm whether GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, and similar agents can access key pages.
• Audit an intentional AI crawler opt-out policy before a launch or client delivery.
• Catch robots.txt changes that newly block or unblock AI crawler agents.
• Detect mixed policy signals where robots.txt allows crawlers but page-level tags say noindex, noai, or noimageai.
• Produce a concise KVS report for SEO, content, platform, or AI search readiness QA.

Why Use This Actor

Option	Strength	Weakness	Best fit
Manual robots.txt review	Fast for one site	Easy to miss wildcard behavior, path specificity, and changed blocked agents	One-off spot checks
Generic SEO crawler	Broad page crawl	Usually not focused on AI crawler user-agent policy or supplied examples testing	Full-site SEO diagnostics
AI Crawler Policy Auditor	Focused AI crawler access decisions, deterministic defaults, KVS report, PPE billing controls	Does not render pages or crawl deep paths	AI crawler policy QA and repeatable robots snapshots

How To Use

Run the actor with the default input for a zero-config local policy audit. To audit your own policy without network calls, paste robots.txt content into targets[].robotsTxt and keep fetchUrls set to false.

{
  "targets": [
    {
      "sourceId": "homepage-policy",
      "sourceUrl": "https://example.com",
      "targetPath": "/",
      "robotsTxt": "User-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nDisallow: /\nSitemap: https://example.com/sitemap.xml",
      "llmsTxt": "# AI crawler guidance\nAllowed for documentation indexing.",
      "headers": {
        "x-robots-tag": "index, follow"
      },
      "html": "<meta name=\"robots\" content=\"index, follow\">"
    }
  ],
  "expectedAccess": "allow",
  "fetchUrls": false,
  "includeReport": true,
  "maxTargets": 10
}

AI Readiness Check

Use expectedAccess: "allow" when the goal is discoverability or AI search readiness. Blocked AI crawler agents become warnings or failures depending on scope.

AI Opt-Out Check

Use expectedAccess: "block" when the goal is to verify an opt-out policy. The actor flags agents that remain allowed, partially blocked, or unmanaged.

Inventory-Only Review

Use expectedAccess: "neutral" when you only want a policy inventory. Mixed policies are still surfaced, but the actor avoids assuming whether blocking is good or bad.

Input Configuration

Field	Type	Default	Description
targets	array	Inline fixture	Robots, llms.txt, header, and HTML snapshots to audit.
urls	string array	[]	Optional site URLs used only when fetchUrls is true.
fetchUrls	boolean	false	Fetch /robots.txt and /llms.txt with timeout and byte caps.
expectedAccess	string	allow	allow, block, or neutral.
auditedAgents	string array	Major AI and search crawlers	User-agent tokens to evaluate against robots.txt.
comparePrevious	boolean	true	Compare previousRobotsTxt when supplied.
requireLlmsTxt	boolean	false	Flag targets without llms.txt content.
includeReport	boolean	true	Write JSON and Markdown report artifacts to KVS.
maxTargets	integer	25	Maximum targets to audit; hard cap is 250.
fetchTimeoutMs	integer	10000	Timeout for optional live fetches.
maxRobotsBytes	integer	200000	Maximum bytes to read from fetched robots.txt.
maxChargeUsd	number	10	Hard PPE cap; stops before work or withholds uncharged reports when the next event would exceed it.
debug	boolean	false	Enable debug logs.

Output Format

Each dataset row represents one audited target. Important fields include:

• status, severity, and aiCrawlerReadinessScore: Overall result.
• expectedAccess and policySummary: Goal and plain-English outcome.
• allowedAgents, blockedAgents, partiallyBlockedAgents, and unknownAgents: Per-agent access buckets.
• agentPolicies: Detailed matched robots group and rule for each audited user agent.
• hasRobotsTxt, robotsGroupCount, hasSitemap, hasLlmsTxt, and byte counts: Policy inventory.
• xRobotsTag, metaRobotsDirectives, hasNoindexSignal, and hasNoaiSignal: Page-level policy signals.
• changedRobotsPolicy, addedBlockedAgents, and removedBlockedAgents: Previous/current diff results.
• issues and recommendation: Structured findings and next action.

The key-value store also contains:

• AI_CRAWLER_POLICY_RESULTS.json: Full result array.
• AI_CRAWLER_POLICY_SUMMARY.json: Run-level counts, scores, blocked agents, and top issues.
• AI_CRAWLER_POLICY_REPORT.md: Human-readable report.

Example Output

{
  "sourceId": "sample-ai-ready-site",
  "sourceUrl": "https://example.com",
  "targetPath": "/",
  "status": "pass",
  "severity": "none",
  "aiCrawlerReadinessScore": 100,
  "expectedAccess": "allow",
  "policySummary": "Audited crawlers are allowed for the target path.",
  "allowedAgents": ["Bingbot", "Bytespider", "CCBot", "ChatGPT-User", "Claude-User", "ClaudeBot", "GPTBot", "Google-Extended", "GoogleOther", "Googlebot", "Meta-ExternalAgent", "OAI-SearchBot", "PerplexityBot", "anthropic-ai"],
  "blockedAgents": [],
  "unknownAgents": [],
  "hasRobotsTxt": true,
  "hasSitemap": true,
  "hasLlmsTxt": true,
  "hasNoindexSignal": false,
  "hasNoaiSignal": false,
  "issueCount": 0,
  "recommendation": "AI crawler policy signals are aligned for the supplied target."
}

Integration Examples

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('junipr/ai-crawler-policy-auditor').call({
  targets: [{ sourceId: 'site', robotsTxt: 'User-agent: *\nAllow: /' }],
  expectedAccess: 'allow',
  fetchUrls: false
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].blockedAgents);

Python

from apify_client import ApifyClient

client = ApifyClient(token="YOUR_APIFY_TOKEN")
run = client.actor("junipr/ai-crawler-policy-auditor").call(run_input={
    "targets": [{"sourceId": "site", "robotsTxt": "User-agent: *\nAllow: /"}],
    "expectedAccess": "allow",
    "fetchUrls": False,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items[0]["recommendation"])

Cost-Control Tips

• Keep fetchUrls false for deterministic CI, QA, and client snapshot reviews.
• Use maxTargets between 1 and 10 for initial live checks.
• Prefer inline robotsTxt snapshots for repeatable regression tests.
• Disable includeReport if you only need dataset rows.
• Use expectedAccess: "neutral" for inventory-only reports where blocking may be intentional.

Public Task Examples

This package includes five prepared task concepts:

• AI crawler allow-list check.
• AI crawler block audit.
• Mixed AI crawler policy review.
• Page-level noai signal scan.
• Optional live robots fetch.

FAQ

Does this actor make network calls by default?

No. The default input is fully inline and fetchUrls defaults to false.

Can it fetch live robots.txt files?

Yes. Set fetchUrls to true and provide urls. The actor fetches /robots.txt and optionally /llms.txt with strict caps.

Does blocking an AI crawler always mean the site is wrong?

No. That depends on your policy goal. Use expectedAccess: "block" for opt-out audits or neutral for inventory-only checks.

Does this actor replace legal review?

No. It reports technical policy signals and does not provide legal advice.

Are diagnostics billed as dataset rows?

Troubleshooting

• robots-txt-missing: Supply inline robots content or enable live fetching for authorized URLs.
• ai-crawlers-blocked: Review blocked agents against your AI search or opt-out goal.
• mixed-policy-signals: Check for conflicts between robots.txt, meta robots, X-Robots-Tag, and selected expected access.
• robots-policy-changed: Review the previous/current diff before treating the current policy as intentional.

Limitations

• Robots matching is designed for practical policy QA, not a full crawler implementation.
• Live fetching can fail due auth, redirects, firewalls, DNS, or network conditions.
• The actor audits a target path, not every URL on a site.
• Page-level signals are parsed from supplied HTML and headers; the actor does not render JavaScript.

Source And Safety Notes

Audit only URLs and policy snapshots you are authorized to inspect. Avoid uploading sensitive internal policy content unless you are allowed to process it in Apify.

Changelog

• 1.0.0: Initial package build with local-first robots auditing, optional capped fetches, previous/current diffing, KVS reports, PPE billing, examples, and fixture tests.