AI Crawler Access Audit: robots.txt, llms.txt, AI Opt-Out
Pricing
from $1.60 / 1,000 domain auditeds
Check which AI crawlers each website allows or blocks: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 19 more, with the deciding robots.txt line. Plus llms.txt and AI training opt-out signals (TDMRep, noai). Bulk domains, change alerts, $2 per 1,000 domains. No login.
Domains or website addresses, one per line. Work emails such as info@acme.com also work. Up to 5,000 per run.
Get the robots.txt lines each site would need to add to reach this policy. Classic search (Googlebot, Bingbot) always stays allowed.
Look for /llms.txt and /llms-full.txt and return the title and link count of llms.txt.
Read /.well-known/tdmrep.json (TDM reservation under the EU text and data mining rules), the tdm-reservation header and meta tag, and noai or noimageai in the robots meta tag and X-Robots-Tag header of the home page.
Return only domains whose AI crawler rules, llms.txt or opt-out signals changed since the last run of this monitor. Unchanged domains are free. Schedule the Actor to get a change alert.
Keeps separate memories for separate lists. Letters, digits and "-".
More user-agent tokens to check, such as a new AI crawler. 23 AI and search crawlers are always checked.
