AI Crawler Access Audit: check GPTBot, ClaudeBot robots.txt

Created by

Don Mangu

Actor

AI Crawler Access Audit: robots.txt, llms.txt, AI Opt-Out

Check which AI crawlers each website allows or blocks, such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended, with the deciding robots.txt line. Also reports llms.txt and AI training opt-out signals (TDMRep, noai). Bulk domains and change detection supported. No login.

Try for free

AI Crawler Access Audit: robots.txt, llms.txt, AI Opt-Outconserving_celerytop/ai-crawler-access-audit

Domain

Verdict

AI training

AI search

+9 fields

Input

Domains(required):nytimes.com+2

Suggest robots.txt lines for this policy:none

Check llms.txt and llms-full.txt:true

Check AI training opt-out signals:true

Only domains that changed:false

Monitor name:default

Sites at a time:5

Output fields

Domain

Verdict

AI training

AI search

Blocked crawlers

Partly blocked

llms.txt

TDM opt-out

noai tag

robots.txt

Changes

Status

Note

How it works

Sign up on Apify or log in

The AI Crawler Access Audit: robots.txt, llms.txt, AI Opt-Out Actor is ready to run with the inputs already set for "AI Crawler Access Audit: check GPTBot, ClaudeBot robots.txt". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "AI Crawler Access Audit: check GPTBot, ClaudeBot robots.txt" task

The AI Crawler Access Audit: robots.txt, llms.txt, AI Opt-Out Actor will start running based on the input automatically. You can also change all the input fields, like domains, suggest robots.txt lines for this policy, and more, to match your specific use case.

Receive the "AI Crawler Access Audit: check GPTBot, ClaudeBot robots.txt" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as domain, verdict, AI training, and more.