Go to example tasks
Robots Rule Tester - Would Google Crawl This URL?
Google-spec robots.txt verdicts per URL + the exact rule that decided. Test Googlebot/GPTBot/any UA. $0.0008/URL, invalid free. API + MCP.
Robots Rule Tester - Would Google Crawl This URL?eliai/robots-rule-tester
URL
Ok
User Agent
Allowed
+7 fieldsTextNumberBooleanListObject
Input
Target URL:https://www.google.com/search
Target URLs (bulk):https://apify.com/store+1
User-agent / crawler:Googlebot
Max URLs:25
Output fields
URL
Ok
User Agent
Allowed
Matched Rule
Sitemaps
Crawl Delay
Robots Txt URL
Robots Status
Tested Path
Matched User Agent Group
Sign up on Apify01
Create your Apify account to access the Robots Rule Tester - Would Google Crawl This URL?.
Start the run02
The Actor will start running based on the input automatically.
Receive the output03
Monitor the progress in real-time. You will be notified as soon as your dataset is complete and ready for review.
Integrate into your workflow04
The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow.
