AI Agent Readiness Audit — llms.txt, AI Crawlers & AI Shopping

AI Agent Readiness Audit: llms.txt, AI Crawlers & AI Shopping

More and more shoppers ask ChatGPT, Claude, Perplexity and Gemini what to buy. Those assistants can only recommend what they can read. This Actor checks a store or website the way an AI assistant sees it and gives it a 0–100 agent-readiness score, a grade and a prioritized fix list.

• AI crawler access: robots.txt rules for 11 AI crawlers and agents (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, CCBot…), split into search and assistant bots (they affect whether you're recommended) and training bots (a policy choice)
• llms.txt: whether the site offers the plain-text guide AI assistants use to understand it
• Readable without JavaScript: most AI crawlers don't run JavaScript, so a page built only in the browser looks empty to them
• Product data on real product pages: Product structured data, machine-readable price, currency and stock, GTIN/SKU, brand and reviews (it samples real product pages and skips category and marketing pages)
• Identity and discovery: Organization/Store structured data, XML sitemap
• Agentic checkout signal: whether the store advertises an agentic-commerce discovery link (such as Shopify's UCP), so AI agents can start a purchase
• Works on any platform: Shopify, WooCommerce, BigCommerce, custom. Sites that aren't stores get the general checks only.

Price: $0.03 per site. The prefilled example (2 sites) finishes in under a minute.

Use this when…

• you want to know if AI shopping assistants (ChatGPT, Claude, Perplexity, Gemini) can read a store's products, prices and stock
• you need to check llms.txt: whether a site has one
• you want an AI crawler audit: which AI bots (GPTBot, ClaudeBot, PerplexityBot…) robots.txt allows or blocks
• you're doing GEO / AI visibility checks for many client or prospect sites at once
• you need a 0–100 AI agent readiness score and a prioritized fix list per site

For AI agents

• Run signalloom/agent-readiness-audit (Apify MCP server call-actor, or the Apify API) with input like {"websites": ["ooni.com"]}
• Returns one JSON row per site in the default dataset: agentReadinessScore (0–100), grade, hasLlmsTxt, aiCrawlers (each bot allowed or blocked), readableWithoutJs, productData (price, stock, GTIN coverage), agenticCheckoutSignal, issues (fix list, errors first).
• Typical cost: 1 site = $0.03. 100 sites = $3.

Who is it for?

• E-commerce agencies: a concrete, sellable "AI readiness" check for every client and prospect
• Store owners and marketers: find out whether AI assistants can see your products, prices and stock
• SEO and GEO specialists: generative-engine optimization audits at scale
• SaaS and app makers: find stores that need help with structured data or AI visibility

How to use it

1. Add websites: www.nutribullet.com, https://your-store.com.
2. Optional: Product pages to sample (default 3).
3. Click Start. Sort by agentReadinessScore and work through each site's issues, which are sorted errors first.

Recipe: agency prospecting. Add 100 prospect stores, then filter to grade D or F, or to issueCodes containing PRICE_NOT_MACHINE_READABLE or AI_SEARCH_BLOCKED. Each one is a specific, fixable pitch.

Output

One record per site (real example, shortened):

{
  "domain": "nutribullet.com",
  "agentReadinessScore": 89,
  "grade": "A",
  "isStore": true,
  "hasLlmsTxt": false,
  "aiSearchCrawlersAllowed": true,
  "aiCrawlers": [
    { "agent": "OAI-SearchBot", "owner": "OpenAI", "purpose": "search", "allowed": true, "explicitRule": false }
  ],
  "readableWithoutJs": true,
  "structuredDataTypes": ["WebSite", "Organization"],
  "hasSitemap": true,
  "productPagesChecked": 3,
  "productData": { "productSchema": 1, "price": 1, "currency": 1, "availability": 1, "gtinOrSku": 1, "brand": 0, "reviews": 1 },
  "agenticCheckoutSignal": false,
  "issues": [
    { "severity": "warning", "code": "NO_LLMS_TXT", "message": "No /llms.txt: a plain-text guide that tells AI assistants what the site offers and where key pages are." },
    { "severity": "notice", "code": "NO_PRODUCT_BRAND", "message": "Brand missing from some product structured data." },
    { "severity": "notice", "code": "NO_AGENTIC_CHECKOUT_SIGNAL", "message": "No agentic-commerce discovery link (e.g. UCP), so AI agents get no standard way to start a checkout." }
  ]
}

productData values are the share of sampled product pages (0–1) that have each field. isStore is null when product pages exist but couldn't be read (for example, a maintenance or bot-check page). In that case the product checks are skipped and an issue explains why.

Score: 100 minus 15 per error, 7 per warning and 2 per notice. Grades: A ≥ 85, B ≥ 70, C ≥ 55, D ≥ 40, F below.

Pricing

• $0.03 per site audited
• Sites that can't be loaded, are blocked by bot protection, or whose robots.txt asks crawlers not to visit: free

Data handling

• Only public pages are read: the homepage, robots.txt, llms.txt, the sitemap and a few product pages. The Actor follows each site's robots.txt rules for general crawlers and never bypasses bot protection.
• No personal data is collected.

Limitations

• It checks a sample of product pages (default 3), not the whole catalog.
• "Readable without JavaScript" is measured from the HTML the server sends. Some AI agents with a full browser can read more.
• AI crawler rules come from robots.txt only. Firewalls or bot protection that block AI bots at the network level aren't visible.
• Agentic-commerce standards are new and changing fast. The checks will be updated as they settle.

FAQ

How do I check if a website has an llms.txt file?

Add the domain and run: hasLlmsTxt and hasLlmsFullTxt tell you, and a missing file shows up in the fix list. It checks a whole list of sites in one run.

Is my robots.txt blocking GPTBot, ClaudeBot or PerplexityBot?

aiCrawlers lists 11 AI crawlers and agents, whether each is allowed, and whether it is a search/assistant bot (affects being recommended) or a training bot (a policy choice).

Can ChatGPT and other AI assistants see my store's prices and stock?

The audit samples real product pages and reports in productData how many have machine-readable price, currency, availability, GTIN/SKU, brand and reviews. Missing ones become fix-list items.

How do I track AI readiness over time?

Schedule a monthly run on your own site or your clients' sites and compare agentReadinessScore and issueCodes between runs.

Feedback

Want another check added? Open an issue on the Issues tab.