AI Crawler, robots.txt & llms.txt Checker

A direct website preflight for AI crawlability and LLM discovery: URL in, crawler-policy and llms.txt evidence out, with regression monitoring available when needed. Returns structured, machine-readable results for repeatable research, monitoring, automation, Apify Tasks, API.

Why use this Actor

Publishers need a fast way to see how public robots.txt and llms.txt signals govern AI crawlers without manually inspecting policy files.

Key features

• Buyer-native inputs aligned to llms.txt.
• Predictable structured output in the default Apify Dataset.
• Pay Per Event pricing tied to measurable result rows or reports.
• Optional MCP connector delivery for downstream automation and AI agents.

Quick start

Start with this working example and replace the target values with your own:

{
  "urls": [
    "https://www.python.org",
    "https://example.com"
  ]
}

The Actor writes structured results to the default Apify Dataset and can be used from the Store, API, schedules, Tasks, automations and MCP-compatible AI workflows.

Input

• urls — Sites or URLs: Public HTTP(S) sites to inspect
• previous_results — Previous policy snapshots: Optional map keyed by input URL. Enables AI crawler policy drift and regression detection.

All integration and advanced analysis fields are optional. The default example is intentionally runnable without configuring MCP or a previous-run baseline.

Output

The Dataset exposes predictable machine-readable output. Representative fields include results.

Pricing

This Actor uses Pay Per Event. The current factory base price is $0.001200 per result event. The Apify Store remains the source of truth for the price and plan/tier details shown to the buyer.

Use cases

• Run the buyer-ready workflows exposed as Apify Tasks without preparing a custom integration first.
• Use the Actor from API or schedules for recurring collection, comparison or monitoring.
• Feed the structured Dataset output into spreadsheets, databases, automations or AI agents.
• Compare repeat runs when the product supports snapshots or previous-run inputs.

Automation and AI

Use the same Actor through Apify API, schedules, public Tasks and the Apify MCP server. Outputs are structured for downstream workflows and AI agents rather than requiring manual copy/paste.

Limitations

• Public websites and upstream APIs can change markup, access rules, rate limits or field availability without notice.
• Fields that are not publicly available are returned as unavailable or omitted rather than fabricated.
• Analytical outputs depend on the quality and coverage of the supplied or collected source data.
• Treat marketplace, reputation, workforce, safety or commercial signals as decision support and validate material decisions against the underlying source evidence.

Detailed documentation

Detailed documentation

Detailed documentation

Detailed documentation

Detailed documentation

Detailed documentation

Detailed documentation

A direct website preflight for AI crawlability and LLM discovery: URL in, crawler-policy and llms.txt evidence out, with regression monitoring available when needed.

A direct website preflight for AI crawlability and LLM discovery: URL in, crawler-policy and llms.txt evidence out, with regression monitoring available when needed.

A direct website preflight for AI crawlability and LLM discovery: URL in, crawler-policy and llms.txt evidence out, with regression monitoring available when needed.

Give it public URLs. Check robots.txt rules for major AI crawlers, llms.txt availability and discovery signals in one structured result.

Audit public website signals that communicate how AI-oriented crawlers may access content, and track adoption of the emerging llms.txt convention.

Key features

• Fetches and parses /robots.txt.
• Summarizes explicit rules for major AI crawler user agents.
• Distinguishes explicit allow, explicit disallow, and no dedicated policy.
• Checks /llms.txt and /.well-known/llms.txt.
• Reports one structured policy record per website with score and issues.

Input

{"urls":["https://www.python.org","https://example.com"]}

Output

Each dataset item contains the requested site, normalized origin, robots.txt status, AI crawler policy summary, llms.txt discovery status, content metadata, score, and issues.

Example

A site may return robotsFound: true, llmsTxtFound: false, and a crawlerPolicies object showing whether user agents such as GPTBot, ClaudeBot, Google-Extended, CCBot, and PerplexityBot have explicit allow or disallow directives.

Why use this Actor

1. AI-specific policy summary avoids hand-reading robots files across many domains.
2. llms.txt discovery tracks an emerging machine-readable publishing convention.
3. Absent policy is explicit, helping governance teams identify sites that need review.
4. No paid monitoring API or AI dependency, keeping recurring portfolio checks inexpensive.

Pricing

Pay per dataset result. One website policy audit creates one primary chargeable result. Pricing is designed for recurring multi-site monitoring.

Good use cases

• Publisher AI-access governance
• SEO and content policy audits
• Portfolio-wide crawler rule inventories
• Monitoring changes to robots.txt and llms.txt adoption

Limitations

• Published directives do not guarantee crawler behavior or legal compliance.
• AI crawler user-agent conventions can change and should be reviewed periodically.
• llms.txt remains an emerging convention; absence is informational, not automatically a defect.

Privacy and safety

The Actor reads only public HTTP(S) policy files, rejects private network targets, follows a small number of validated redirects, and requires no credentials.

Automation and recurring checks

This Actor is designed for repeatable Apify workflows. Save a task when you need to run the same check regularly, schedule it from Apify Console, or call the Actor through the Apify API. Each run writes structured results to the default dataset so the output can be exported to JSON, CSV, Excel, or consumed by another automation.

For reliable monitoring, start with a small input and confirm the expected output before increasing the batch size. Public websites can change behavior over time, so recurring users should review issue flags and HTTP failures rather than assuming an unavailable value means the same thing as a passing result. The Actor uses limited Apify permissions and does not require access to unrelated account data.

Reliability update — 2026-10-24

• Reject HTML soft-404s as llms.txt and expose effective wildcard crawler policy inheritance.