AI Visibility & Bot Intelligence Suite - Dark Visitors Alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation, and AI shopping optimization. Monitor which AI crawlers visit your site, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents.

Features

• AI Bot Directory - Comprehensive database of 38+ AI crawlers with user agents, operators, and categories
• Robots.txt Analysis - Check which AI bots are blocked or allowed by your robots.txt
• LLMs.txt Check - Verify if a site has an LLMs.txt file for AI context
• LLMs.txt Generator - Auto-generate optimized LLMs.txt content for your site
• AI Shopping Optimizer - Analyze e-commerce sites for AI shopping agent compatibility
• Competitor AI Audit - Compare AI visibility settings across competitor sites
• Bot Response Tester - Test how sites respond to different AI bot user agents

Quick Start

{
  "urls": ["https://example.com"],
  "modules": ["robotsTxtAnalysis", "llmsTxtCheck"],
  "demoMode": false
}

Demo Mode

Set demoMode: true to test with sample data (no charges, no actual network requests). Perfect for testing the output format before running real analysis.

{
  "demoMode": true,
  "urls": ["https://example.com"],
  "modules": ["aiBotDirectory", "robotsTxtAnalysis"]
}

Input Parameters

Parameter	Type	Default	Description
urls	array	required	List of website URLs to analyze
modules	array	["robotsTxtAnalysis", "llmsTxtCheck"]	Analysis modules to run
runAllModules	boolean	false	Override module selection and run all modules
botCategories	array	[] (all)	Filter AI bot categories to analyze
sortBotsBy	string	"prevalence"	Sort bots by: name, category, operator, prevalence
generateLlmsTxt	boolean	false	Auto-generate LLMs.txt if site doesn't have one
customUserAgents	array	[]	Additional bot user agents to test
testPaths	array	["/", "/products", "/about"]	URL paths to test with bot user agents
outputFormat	string	"json"	Output format: json, markdown, html
demoMode	boolean	false	Run with sample data (no network requests)
webhookUrl	string	-	URL to receive notifications (Zapier, Make, n8n)
webhookOnIssues	boolean	true	Trigger webhook when issues are detected
webhookOnLowScore	boolean	true	Trigger webhook when score falls below threshold
lowScoreThreshold	integer	50	Alert threshold for low scores (0-100)

Available Modules

Module ID	Name	Description
aiBotDirectory	AI Bot Directory	Comprehensive database of AI crawlers and bots
robotsTxtAnalysis	Robots.txt Analysis	Check AI bot blocking/allowing status
llmsTxtCheck	LLMs.txt Check	Verify if site has LLMs.txt file
llmsTxtGenerator	LLMs.txt Generator	Generate optimized LLMs.txt for your site
aiShoppingOptimizer	AI Shopping Optimizer	Analyze e-commerce AI readiness
competitorAudit	Competitor AI Audit	Compare AI visibility across competitors
botResponseTester	Bot Response Tester	Test how site responds to AI bot user agents

Bot Categories

Category	Description
AI Assistant	ChatGPT, Claude, Gemini and other AI assistants
AI Search Crawler	Perplexity, You.com, Bing AI crawlers
AI Data Scraper	Bots that scrape data for AI training
AI Agent	Autonomous AI agents that browse the web
Undocumented AI Agent	AI bots without official documentation
Intelligence Archiver	Archive.org and similar services
AI Search Fetcher	Real-time content fetchers for AI search

Output Format

Robots.txt Analysis

{
  "module": "robotsTxtAnalysis",
  "url": "https://example.com",
  "status": "completed",
  "score": 75,
  "robotsTxtFound": true,
  "robotsTxtUrl": "https://example.com/robots.txt",
  "totalBotsChecked": 38,
  "botsBlocked": 12,
  "botsAllowed": 26,
  "aiBotsAnalysis": [
    {
      "botName": "GPTBot",
      "userAgent": "GPTBot",
      "operator": "OpenAI",
      "category": "AI Assistant",
      "isBlocked": true,
      "matchedRule": "User-agent: GPTBot\\nDisallow: /"
    }
  ],
  "recommendations": [
    "Consider allowing GPTBot if you want your content in ChatGPT responses"
  ]
}

LLMs.txt Check

{
  "module": "llmsTxtCheck",
  "url": "https://example.com",
  "status": "completed",
  "llmsTxtFound": false,
  "llmsTxtUrl": "https://example.com/llms.txt",
  "recommendations": [
    "Create an LLMs.txt file to provide context for AI systems"
  ]
}

AI Bot Directory

{
  "module": "aiBotDirectory",
  "status": "completed",
  "totalBots": 38,
  "categories": ["AI Assistant", "AI Search Crawler", "AI Data Scraper"],
  "bots": [
    {
      "name": "GPTBot",
      "userAgent": "GPTBot",
      "operator": "OpenAI",
      "category": "AI Assistant",
      "description": "OpenAI's web crawler for ChatGPT",
      "respectsRobotsTxt": true,
      "documentationUrl": "https://platform.openai.com/docs/gptbot"
    }
  ]
}

AI Shopping Optimizer

{
  "module": "aiShoppingOptimizer",
  "url": "https://example.com",
  "status": "completed",
  "score": 65,
  "platform": "shopify",
  "structuredDataFound": true,
  "schemaTypes": ["Product", "BreadcrumbList", "Organization"],
  "issues": [
    "Missing detailed product descriptions for AI comprehension"
  ],
  "recommendations": [
    "Add FAQ schema for common product questions",
    "Enhance product descriptions with AI-readable attributes"
  ]
}

AI Bot Database

The suite includes a comprehensive database of 38+ AI bots:

Bot Name	Operator	Category
GPTBot	OpenAI	AI Assistant
ChatGPT-User	OpenAI	AI Assistant
OAI-SearchBot	OpenAI	AI Search Crawler
Claude-Web	Anthropic	AI Assistant
anthropic-ai	Anthropic	AI Data Scraper
Google-Extended	Google	AI Data Scraper
Googlebot	Google	AI Search Crawler
Gemini	Google	AI Assistant
Bingbot	Microsoft	AI Search Crawler
bingbot	Microsoft	AI Search Crawler
PerplexityBot	Perplexity AI	AI Search Crawler
YouBot	You.com	AI Search Crawler
cohere-ai	Cohere	AI Data Scraper
CCBot	Common Crawl	AI Data Scraper
FacebookBot	Meta	AI Data Scraper
Meta-ExternalAgent	Meta	AI Agent
Bytespider	ByteDance	AI Data Scraper
Diffbot	Diffbot	AI Data Scraper
Applebot	Apple	AI Search Crawler
Applebot-Extended	Apple	AI Data Scraper
amazonbot	Amazon	AI Search Crawler
ClaudeBot	Anthropic	AI Assistant
AI2Bot	Allen AI	AI Data Scraper
Omgilibot	Omgili	AI Data Scraper
Timpibot	Timpi	AI Search Crawler
VelenPublicWebCrawler	Velen	AI Data Scraper
Webzio-Extended	Webz.io	AI Data Scraper
PanguBot	DeepSeek	AI Assistant
Scrapy	Scrapy	AI Data Scraper
iaskspider	iAsk	AI Search Crawler
Kangaroo Bot	Kangaroo	AI Agent
img2dataset	LAION	AI Data Scraper
ISSCyberRiskCrawler	ISS	Intelligence Archiver
AwarioSmartBot	Awario	AI Search Fetcher
DataForSeoBot	DataForSEO	AI Data Scraper
Nicecrawler	NiceCrawler	AI Search Crawler
FriendlyCrawler	FriendlyCrawler	AI Data Scraper
Sidetrade indexer bot	Sidetrade	AI Data Scraper

Webhook Integration (Zapier, Make, n8n)

Connect this actor to your automation workflows with webhook notifications.

Setup

{
  "urls": ["https://example.com"],
  "modules": ["robotsTxtAnalysis", "llmsTxtCheck"],
  "webhookUrl": "https://hooks.zapier.com/hooks/catch/xxx/yyy",
  "webhookOnIssues": true,
  "webhookOnLowScore": true,
  "lowScoreThreshold": 50
}

Webhook Payload

When triggered, the webhook receives:

{
  "actorId": "ecvImN3D2jfsZaHdt",
  "actorRunId": "abc123",
  "status": "alert",
  "alertType": "low_score",
  "timestamp": "2025-01-22T10:00:00Z",
  "summary": {
    "urlsAnalyzed": 5,
    "modulesRun": 2,
    "averageScore": 45,
    "totalIssues": 8,
    "totalRecommendations": 12
  },
  "alerts": [
    {
      "url": "https://example.com",
      "module": "robotsTxtAnalysis",
      "score": 35,
      "issues": 5,
      "message": "Low AI visibility score: 35% (threshold: 50%)"
    }
  ],
  "datasetUrl": "https://api.apify.com/v2/datasets/xxx/items"
}

Alert Types

Alert Type	Description
low_score	AI visibility score below threshold
issues_found	One or more AI visibility issues detected
blocked_bots	Important AI bots are being blocked

Integration Examples

Zapier: Use "Webhooks by Zapier" trigger with "Catch Hook"

Make (Integromat): Use "Webhooks" module with "Custom webhook"

n8n: Use "Webhook" trigger node

Custom: POST endpoint receives JSON payload

Pricing

This actor uses pay-per-event billing:

Event	Title	Description	Price
url-analyzed	URL Analyzed	Charged for each URL analyzed	$0.03
module-executed	Module Executed	Charged for each module run	$0.01
content-generated	Content Generated	Charged when LLMs.txt is generated	$0.05

Example costs:

• Analyze 10 URLs with 2 modules each: 10 × $0.03 + 20 × $0.01 = $0.50
• Generate LLMs.txt for 5 sites: 5 × $0.05 = $0.25

Demo mode is free (no charges).

Use Cases

• AI Compliance Audit - Check if your robots.txt properly manages AI crawler access
• LLMs.txt Implementation - Generate and validate LLMs.txt files for AI context
• Competitive Analysis - Compare how competitors handle AI bot access
• E-commerce AI Readiness - Optimize product pages for AI shopping agents
• AI Visibility Monitoring - Track which AI bots can access your content
• SEO for AI - Ensure your content is discoverable by AI-powered search engines

Comparison: Dark Visitors vs AI Visibility Suite

Feature	Dark Visitors	AI Visibility Suite
AI Bot Database	Yes	Yes (38+ bots)
Robots.txt Analysis	Yes	Yes
LLMs.txt Support	Limited	Full (check + generate)
AI Shopping Optimization	No	Yes
Competitor Audits	No	Yes
Bot Response Testing	No	Yes
API Access	Paid	Included
Pay-per-use	No	Yes

---

Built by John Rippy | johnrippy.link

2025 Zapier Automation Hero of the Year - Project Phoenix: A 95-step AI sales pipeline cutting development time by 50%.

---

Actor Arsenal