AI Crawler & llms.txt Policy Auditor

Audit public website signals that communicate how AI-oriented crawlers may access content, and track adoption of the emerging llms.txt convention.

Key features

• Fetches and parses /robots.txt.
• Summarizes explicit rules for major AI crawler user agents.
• Distinguishes explicit allow, explicit disallow, and no dedicated policy.
• Checks /llms.txt and /.well-known/llms.txt.
• Reports one structured policy record per website with score and issues.

Input

{"urls":["https://www.python.org","https://example.com"]}

Output

Each dataset item contains the requested site, normalized origin, robots.txt status, AI crawler policy summary, llms.txt discovery status, content metadata, score, and issues.

Example

A site may return robotsFound: true, llmsTxtFound: false, and a crawlerPolicies object showing whether user agents such as GPTBot, ClaudeBot, Google-Extended, CCBot, and PerplexityBot have explicit allow or disallow directives.

Why use this Actor

1. AI-specific policy summary avoids hand-reading robots files across many domains.
2. llms.txt discovery tracks an emerging machine-readable publishing convention.
3. Absent policy is explicit, helping governance teams identify sites that need review.
4. No paid monitoring API or AI dependency, keeping recurring portfolio checks inexpensive.

Pricing

Pay per dataset result. One website policy audit creates one primary chargeable result. Pricing is designed for recurring multi-site monitoring.

Good use cases

• Publisher AI-access governance
• SEO and content policy audits
• Portfolio-wide crawler rule inventories
• Monitoring changes to robots.txt and llms.txt adoption

Limitations

• Published directives do not guarantee crawler behavior or legal compliance.
• AI crawler user-agent conventions can change and should be reviewed periodically.
• llms.txt remains an emerging convention; absence is informational, not automatically a defect.

Privacy and safety

The Actor reads only public HTTP(S) policy files, rejects private network targets, follows a small number of validated redirects, and requires no credentials.

Automation and recurring checks

This Actor is designed for repeatable Apify workflows. Save a task when you need to run the same check regularly, schedule it from Apify Console, or call the Actor through the Apify API. Each run writes structured results to the default dataset so the output can be exported to JSON, CSV, Excel, or consumed by another automation.

For reliable monitoring, start with a small input and confirm the expected output before increasing the batch size. Public websites can change behavior over time, so recurring users should review issue flags and HTTP failures rather than assuming an unavailable value means the same thing as a passing result. The Actor uses limited Apify permissions and does not require access to unrelated account data.