Robots.txt Tester & Checker - Would Google Crawl This URL? avatar

Robots.txt Tester & Checker - Would Google Crawl This URL?

Pricing

from $0.64 / 1,000 tested urls

Go to Apify Store
Robots.txt Tester & Checker - Would Google Crawl This URL?

Robots.txt Tester & Checker - Would Google Crawl This URL?

Tests URLs against their robots.txt with Google's matching rules: longest match, Allow wins ties, user-agent group selection (Googlebot, GPTBot...) and wildcards. Returns allowed or blocked, the exact rule that decided it and the declared sitemaps. For SEO specialists, publishers and developers.

Pricing

from $0.64 / 1,000 tested urls

Rating

0.0

(0)

Developer

Broke to Built

Broke to Built

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

14 days ago

Last modified

Features

Limited permissions

Share