Which news sites block AI crawlers like GPTBot and ClaudeBot?

Created by

Rowcraft

Actor

Robots.txt and AI Crawler Access Checker API

The robots.txt policy of six large news websites for 27 AI and search crawlers, one row per website: which AI training bots, AI search bots and search engines are allowed or blocked, the sitemaps, llms.txt and the meta robots tag of the home page. Replace the list with your own competitors, clients or publishers.

Try for free

Robots.txt and AI Crawler Access Checker APIrowcraft/robots-ai-crawler-checker

Website checked (without www.)

GPTBot (OpenAI) allowed on every checked path; false = blocked on at least one; null = not in the list checked

ChatGPT-User (OpenAI) allowed on every checked path; false = blocked on at least one; null = not in the list checked

OAI-SearchBot (OpenAI) allowed on every checked path; false = blocked on at least one; null = not in the list checked

+16 fields

Input

Websites(required):bbc.com+5

Output fields

Website checked (without www.)

GPTBot (OpenAI) allowed on every checked path; false = blocked on at least one; null = not in the list checked

ChatGPT-User (OpenAI) allowed on every checked path; false = blocked on at least one; null = not in the list checked

OAI-SearchBot (OpenAI) allowed on every checked path; false = blocked on at least one; null = not in the list checked

ClaudeBot (Anthropic) allowed on every checked path; false = blocked on at least one; null = not in the list checked

Claude-User (Anthropic) allowed on every checked path; false = blocked on at least one; null = not in the list checked

Claude-SearchBot (Anthropic) allowed on every checked path; false = blocked on at least one; null = not in the list checked

PerplexityBot (Perplexity) allowed on every checked path; false = blocked on at least one; null = not in the list checked

Perplexity-User (Perplexity) allowed on every checked path; false = blocked on at least one; null = not in the list checked

Google-Extended (Google) allowed on every checked path; false = blocked on at least one; null = not in the list checked

Applebot-Extended (Apple) allowed on every checked path; false = blocked on at least one; null = not in the list checked

Amazonbot (Amazon) allowed on every checked path; false = blocked on at least one; null = not in the list checked

Amzn-SearchBot (Amazon) allowed on every checked path; false = blocked on at least one; null = not in the list checked

Amzn-User (Amazon) allowed on every checked path; false = blocked on at least one; null = not in the list checked

Meta-ExternalAgent (Meta) allowed on every checked path; false = blocked on at least one; null = not in the list checked

Meta-WebIndexer (Meta) allowed on every checked path; false = blocked on at least one; null = not in the list checked

DuckAssistBot (DuckDuckGo) allowed on every checked path; false = blocked on at least one; null = not in the list checked

MistralAI-User (Mistral) allowed on every checked path; false = blocked on at least one; null = not in the list checked

MistralAI-Index (Mistral) allowed on every checked path; false = blocked on at least one; null = not in the list checked

MistralAI-Training (Mistral) allowed on every checked path; false = blocked on at least one; null = not in the list checked

How it works

Sign up on Apify or log in

The Robots.txt and AI Crawler Access Checker API Actor is ready to run with the inputs already set for "Which news sites block AI crawlers like GPTBot and ClaudeBot?". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Which news sites block AI crawlers like GPTBot and ClaudeBot?" task

The Robots.txt and AI Crawler Access Checker API Actor will start running based on the input automatically. You can also change all the input fields, like websites, to match your specific use case.

Receive the "Which news sites block AI crawlers like GPTBot and ClaudeBot?" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as website checked (without www.), GPTBot (OpenAI) allowed on every checked path; false = blocked on at least one; null = not in the list checked, chatgpt-user (openai) allowed on every checked path; false = blocked on at least one; null = not in the list checked, and more.