AI Agent Readiness Audit — llms.txt, robots.txt & Schema

Check whether AI agents can actually use a website. Audits robots.txt for AI-agent blocking (GPTBot, ClaudeBot, PerplexityBot and more), llms.txt presence, JSON-LD Product/Organization structured data, sitemap availability and summarisation signals. Returns a 0-100 score, a letter grade and a named finding for every deduction, so the result is an actionable fix list, not just a number. Built by PAOE.

Each item is processed individually and charged as its own event, so you pay only for what the run actually delivers. Results are written to the run's dataset as one JSON object per item, ready to download as JSON, CSV or Excel, or to pull through the Apify API.

What it checks

For each URL: whether robots.txt blocks named AI agents (GPTBot, ClaudeBot, PerplexityBot and others), whether an llms.txt file is published, whether the page carries JSON-LD Organization or Product structured data, whether a sitemap is advertised, and whether the page carries the summarisation signals assistants use (a meta description and a clear heading).

Pricing

Event	What it covers	Price
site-audited	site audited (primary)	$0.05 per event

Volume tiers reduce the price automatically on higher Apify plans: Bronze 15% off, Silver 25% off and Gold or above 40% off the listed free-tier price. The charge is per item processed, not per run.

Input

Field	Type	Required	Default	Description
urls	array	yes	["https://example.com"]	One or more site URLs. Each URL is one audited site and one charged event.
maxUrls	integer	no	10	Safety cap on how many URLs are audited in one run.

Example input:

{
  "urls": [
    "https://example.com"
  ],
  "maxUrls": 10
}

Output

One JSON object per item in the run's dataset. Every result carries the input it came from plus the fields this Actor measures, so the output can be joined back to your own data without guessing which row is which. The final dataset entry is a summary object with the run's totals.

Typical use cases

• Check whether AI crawlers and answer engines can actually read a site.
• Prepare a site for AI-agent traffic before competitors do.
• Produce a prioritised, per-item fix list for a web or SEO team.

Limitations

The sitemap is checked for availability and reported as a finding, but its URLs are not added automatically. Audited URLs come from the urls input, so you control exactly what is checked and charged. Sites behind a login, a paywall or an aggressive bot wall return unusable content and are reported as errors, not counted as successes.

Does it follow the sitemap?

Every limitation above is reported per item in a findings entry with a level of fail, warn, info or ok, a machine-readable code and a concrete action. If something cannot be checked it is reported as such rather than assumed to be fine.

Can it run on a schedule?

Yes. Save a task from this Actor with your inputs, then set a schedule on the task. Scheduled runs recur with the same inputs, which is the intended way to use it for ongoing monitoring.

Can I audit a whole site?

Pass the pages you want checked in urls, up to maxUrls. Every URL audited is one charged event, so the cost is exactly the number of pages you list.

Notes

If a site returns something unexpected, open an issue on the Actor's page with the URL and the input used, and it will be looked at.

Related Actors

• Bulk Broken Link & Redirect Audit

Keywords

ai agent readiness, llms.txt, robots.txt, GPTBot, JSON-LD, structured data, SEO audit, answer engine optimization, AEO.