Bulk AI Crawler Access Checker
Pricing
from $3.00 / 1,000 domain checkeds
Bulk AI Crawler Access Checker
Audit AI crawler access across many domains at once. For each domain: robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, CCBot and more, plus llms.txt, ai.txt, RSL licensing — and a live User-Agent test that catches domains blocking bots their robots.txt actually allows.
Pricing
from $3.00 / 1,000 domain checkeds
Rating
0.0
(0)
Developer
srvScripts
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
a day ago
Last modified
Categories
Share
Bulk AI Crawler Access Checker (Apify Actor)
For each domain you give it, this Actor reports:
- robots.txt AI-bot policy — allowed/blocked for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot, Amazonbot, plus the Google-Extended / Applebot-Extended policy tokens.
- Live User-Agent response test — how the server actually answers each AI crawler's User-Agent vs a normal browser (catches firewall/CDN UA blocks robots.txt can't show).
- llms.txt, ai.txt, RSL / Content-Usage presence and validity.
Read-only: GET requests to the entered domains only; redirects are never followed; private/internal addresses are refused.
Input
Output (one dataset row per domain)
domain, robots_state, policy (per crawler), home_baseline, live (per crawler: policy + observed HTTP + flag), allowed_but_refused_count, llms_txt, ai_txt, rsl.
Monetization (you set this in the Apify Store UI)
The code calls Actor.charge({ eventName: 'domain-checked' }) once per domain that returns a result — so use pay-per-event and define an event named domain-checked. Market rate for this kind of bulk checker is about $1.50–$3.00 per 1,000 domains. Charging is skipped automatically in local/dev runs.
Honesty note (keep this in the Store description)
The live test runs from Apify's servers using crawler User-Agent strings — it shows how a site treats those User-Agents, not traffic from the crawler operators' own verified IP ranges. robots.txt is advisory; "policy" and "observed response" are reported separately.
Built by srvScripts — https://srvscripts.com/
