Go to example tasks
AI Crawler Access Audit: check GPTBot, ClaudeBot robots.txt
Check which AI crawlers each website allows or blocks, such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended, with the deciding robots.txt line. Also reports llms.txt and AI training opt-out signals (TDMRep, noai). Bulk domains and change detection supported. No login.
AI Crawler Access Audit: robots.txt, llms.txt, AI Opt-Outconserving_celerytop/ai-crawler-access-audit
Domain
Verdict
AI training
AI search
+9 fieldsTextNumberBooleanListObject
Input
Domains(required):nytimes.com+2
Suggest robots.txt lines for this policy:none
Check llms.txt and llms-full.txt:true
Check AI training opt-out signals:true
Only domains that changed:false
Monitor name:default
Sites at a time:5
Output fields
Domain
Verdict
AI training
AI search
Blocked crawlers
Partly blocked
llms.txt
TDM opt-out
noai tag
robots.txt
Changes
Status
Note
Sign up on Apify01
Create your Apify account to access the AI Crawler Access Audit: robots.txt, llms.txt, AI Opt-Out.
Start the run02
The Actor will start running based on the input automatically.
Receive the output03
Monitor the progress in real-time. You will be notified as soon as your dataset is complete and ready for review.
Integrate into your workflow04
The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow.
