AI crawler access of big online stores: who is blocked

Created by

Rowcraft

Actor

Robots.txt and AI Crawler Access Checker API

Which of 27 AI and search crawlers five large online stores allow on their home page and on a product search path, read from robots.txt by the RFC 9309 rules. The table lists the allowed and blocked crawlers per store and per path. Use it to compare how retailers treat AI shopping assistants and training bots.

Try for free

Robots.txt and AI Crawler Access Checker APIrowcraft/robots-ai-crawler-checker

Website checked (without www.)

Crawlers allowed on every checked path (names separated by commas, or null)

Crawlers blocked on at least one checked path (names separated by commas, or null)

Checked crawlers that have a group of their own in robots.txt (names separated by commas, or null)

+1 field

Input

Websites(required):amazon.com+4

Paths to check:/+1

Output fields

Website checked (without www.)

Crawlers allowed on every checked path (names separated by commas, or null)

Crawlers blocked on at least one checked path (names separated by commas, or null)

Checked crawlers that have a group of their own in robots.txt (names separated by commas, or null)

Every user agent named in robots.txt, in order of first appearance (up to 25, or null)

How it works

Sign up on Apify or log in

The Robots.txt and AI Crawler Access Checker API Actor is ready to run with the inputs already set for "AI crawler access of big online stores: who is blocked". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "AI crawler access of big online stores: who is blocked" task

The Robots.txt and AI Crawler Access Checker API Actor will start running based on the input automatically. You can also change all the input fields, like websites and paths to check, to match your specific use case.

Receive the "AI crawler access of big online stores: who is blocked" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as website checked (without www.), crawlers allowed on every checked path (names separated by commas, or null), crawlers blocked on at least one checked path (names separated by commas, or null), and more.