robots.txt Parser & AI Crawler Block Checker
Pricing
$11.00 / 1,000 robots.txt ai policy rows
Go to Apify Store
robots.txt Parser & AI Crawler Block Checker
Try Input: 1 domain ≈ USD 0.011/row. From USD 11/1k. Paste domain and Start.
robots.txt Parser & AI Crawler Block Checker
Pricing
$11.00 / 1,000 robots.txt ai policy rows
Try Input: 1 domain ≈ USD 0.011/row. From USD 11/1k. Paste domain and Start.
List of domains to analyze robots.txt for AI crawler policies. Max 500.
How to deliver results. 'dataset' saves to Apify Dataset, 'webhook' sends to a URL. In demoMode, delivery is always dataset.
Webhook URL to send results to (only used when delivery is 'webhook'). Works with Slack, Discord, or any HTTP endpoint.
Record key for change snapshots inside the fixed robotstxt-ai-checker-state Key-Value Store.
all emits every result. changes_only emits only changed results and emits zero rows and zero charges when nothing changed.
Maximum number of parallel requests. Higher = faster but may trigger rate limits.
If true, runs without saving results or sending webhooks. Useful for testing.
