AI Visibility Panel

Apify Store title: AI Crawler Access & llms.txt Visibility Panel

Apify Store description:

See, across any list of websites, which AI crawlers each site blocks or allows and whether it has adopted the new llms.txt standard. The actor reads each site's public robots.txt and llms.txt and returns one clean row per domain: a per-crawler blocked/allowed map for GPTBot, ChatGPT-User, OAI-SearchBot (OpenAI), ClaudeBot, anthropic-ai, Claude-Web (Anthropic), CCBot (Common Crawl), Google-Extended, PerplexityBot, Bytespider, Amazonbot and Applebot-Extended, plus hasLlmsTxt, robotsHasAiRules, a per-vendor breakdown, and a summary like "blocks 3 of 12 AI crawlers". Zero-config default run scans ~15 well-known media, SaaS and retail sites immediately. 100% public data: no login, no API key, no PII.

Who buys this

• AI companies - see exactly who blocks your crawler; build a licensing / partnership target list.
• GEO / SEO agencies - audit whether client sites are visible to AI answer engines and have an llms.txt.
• AI-training / data teams - map crawlable vs blocked domains before a crawl.
• Publishers & media - benchmark your AI-access posture against peers.

Source + legality

GET https://{domain}/robots.txt and GET https://{domain}/llms.txt - two files every website publishes for crawlers to read. No authentication, no cookies, no PII. Public-data access is backed by hiQ v. LinkedIn and Meta v. Bright Data. A descriptive User-Agent is sent, redirects are followed, and one slow site never fails the run.

How it reads robots.txt

• User-agent tokens and field names are matched case-insensitively (real sites vary the case).
• A crawler is blocked when its ruleset disallows the site root (Disallow: /) and does not re-allow it (Allow: /).
• A blanket User-agent: * + Disallow: / counts as blocking all AI crawlers (via wildcard fallback).
• A crawler named with Allow: / (some sites explicitly allow GPTBot) is correctly reported as allowed.
• hasLlmsTxt is true only on HTTP 200 with a non-empty body, so soft-404 pages do not false-positive.

Zero-config default run

Leave the input empty: the actor scans a stable seed list (openai.com, anthropic.com, nytimes.com, theverge.com, reddit.com, github.com, shopify.com, medium.com, cnn.com, stackoverflow.com, wikipedia.org, substack.com, notion.so, hubspot.com, cloudflare.com) and returns one real row per domain, so the Store "try it" button works on first click. Point domains at your own list to scan any sites.

Output fields (one row per domain)

domain, robotsStatus, llmsStatus, hasRobotsTxt, robotsReadable, robotsHasAiRules, blanketBlockAll, hasLlmsTxt, llmsTxtBytes, aiCrawlerCount, blockedCount, allowedCount, blocks{crawler:bool}, blockedCrawlers[], vendors{vendor:allowed|partial|blocked}, summary, robotsUrl, llmsUrl, source, checkedAt

Run locally (no Apify account)

npm install
npm test --workspace actors/ai-visibility-panel
# or run the actor directly:
node actors/ai-visibility-panel/src/main.js

Proposed pricing (set in Apify at publish - CJ-gated)

Pay-Per-Event in the documented winning band:

• ~$0.004 per domain analyzed (one billed result row per domain)

Pricing is NOT set here - it is configured on the Apify Store at publish and requires CJ sign-off. No account, no publish, no spend happens in this repo.