See Which AI Crawlers a Publisher Blocks
Mamba Labs
Reads robots.txt and well known files on a news or publisher domain to show which AI crawlers are allowed and which are blocked.
Input
Output fields
Sign up on Apify or log in
The Agent Accessibility Auditor Actor is ready to run with the inputs already set for "See Which AI Crawlers a Publisher Blocks". Sign up or log in and you'll land in Apify Console with a copy of this exact task.
Start the "See Which AI Crawlers a Publisher Blocks" task
The Agent Accessibility Auditor Actor will start running based on the input automatically. You can also change all the input fields, like domain, check machine readable endpoints, and more, to match your specific use case.
Receive the "See Which AI Crawlers a Publisher Blocks" output
Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as domain, agent readable tier, agent readable score, and more.
Integrate into your workflow
The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow. You can also run the "See Which AI Crawlers a Publisher Blocks" task via API.
