Check which AI crawlers your site and competitors allow

Created by

Leyten Data

Actor

Crawlability & AI Crawler Access Auditor

Per domain: robots.txt rules for GPTBot, ClaudeBot and CCBot, TDM opt-out, llms.txt, sitemaps and the anti-bot service in front.

Try for free

Crawlability & AI Crawler Access Auditorlwsdjfls/crawlability-ai-access-auditor

Domain

Verdict

AI training blocked

AI search blocked

+9 fields

Input

Domains(required):nos.nl+1

Pages to sample:5

Scan terms and legal pages for TDM clauses:true

Output fields

Domain

Verdict

AI training blocked

AI search blocked

Generic crawlers

TDM opt-out

Anti-bot

Challenged

llms.txt

Sitemap URLs

Last change

Summary

Since last run

How it works

Sign up on Apify or log in

The Crawlability & AI Crawler Access Auditor Actor is ready to run with the inputs already set for "Check which AI crawlers your site and competitors allow". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Check which AI crawlers your site and competitors allow" task

The Crawlability & AI Crawler Access Auditor Actor will start running based on the input automatically. You can also change all the input fields, like domains, pages to sample, and more, to match your specific use case.

Receive the "Check which AI crawlers your site and competitors allow" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as domain, verdict, AI training blocked, and more.