Robots Audit: robots.txt Parser and AI Crawler Policy Check
Pricing
from $2.00 / 1,000 results
Robots Audit: robots.txt Parser and AI Crawler Policy Check
Parse robots.txt for up to 100 sites into one JSON record each: rule groups, sitemaps and crawl delays, plus an AI crawler policy audit across 17 bots like GPTBot, ClaudeBot and PerplexityBot. Each bot rated allowed, partial or blocked, with an open, selective or closed summary per site.
Pricing
from $2.00 / 1,000 results
Rating
0.0
(0)
Developer
Stive Hassan
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
4 days ago
Last modified
Categories
Share
Robots Audit
Parse robots.txt for any list of sites and get one clean JSON record each: rule groups, sitemaps, crawl delays, and an AI crawler policy audit covering 17 known AI bots (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider and more): allowed, partial, blocked, or no-rules per bot, plus an open / selective / closed summary per site.
What it does / What it does NOT do
- Fetches /robots.txt for up to 100 sites per run. Sites without one are reported as no-robots-txt (open by definition).
- Not supported: crawling pages, logins, or bypassing blocks. This tool reads policy; it does not help evade it.
Input / Output
Input: sites. Output: one dataset record per site with status
ok | no-robots-txt | blocked | error.
Pricing
Pay-per-event, per site audited. Free-plan runs are not charged by Apify.
Reliability
Fixture-tested nightly; live-verified weekly. Missing data is null, never fabricated.
