Pricing

from $0.01 / 1,000 results

Robots.txt Checker - CMS-Aware Analysis with AI Recommendations

The Robots.txt Checker provides comprehensive analysis of your robots.txt file: Syntax Validation CMS Detection - Identify WordPress, Shopify, Drupal,& 6+ other CMS platforms Best Practice Check Companion File Checks - sitemap.xml, llms.txt, security.txt AI Recommendations - CMS-specific suggestions

Pricing

from $0.01 / 1,000 results

Rating

0.0

(0)

Developer

The Howlers

Actor stats

Bookmarked

Total users

Monthly active users

20 days ago

Last modified

Robots.txt Checker with AI Recommendations

Analyze robots.txt files for syntax errors, best practices, and CMS-specific optimizations. Includes AI-powered recommendations, sitemap validation, and companion file checks (llms.txt, security.txt).

BYOK (Bring Your Own Key) -- you provide your own API credentials.

Before You Start

This actor requires your own API credentials to fetch real data.

Where to get your key: Your Anthropic API key for enhanced AI recommendations. Without key, uses rule-based recommendations.

Quick Start

Run with real data

{
  "url": "https://example.com",
  "urls": "https://example.com",
  "includeSitemapCheck": true,
  "includeLlmsTxtCheck": true,
  "includeSecurityTxtCheck": false,
  "includeAIRecommendations": true,
  "anthropicApiKey": "YOUR_API_KEY_HERE"
}

Input Parameters

Parameter	Type	Default	Required	Description
`url`	string	-	No	Single URL to analyze (will check /robots.txt at domain root)
`urls`	array	-	No	Multiple URLs to analyze (batch processing)
`includeSitemapCheck`	boolean	`true`	No	Verify that declared sitemaps exist and are valid
`includeLlmsTxtCheck`	boolean	`true`	No	Check for llms.txt (AI training opt-out file)
`includeSecurityTxtCheck`	boolean	`false`	No	Check for .well-known/security.txt
`includeAIRecommendations`	boolean	`true`	No	Generate AI-powered optimization recommendations
`anthropicApiKey`	string	-	Yes*	Your Anthropic API key for enhanced AI recommendations. Without key, uses rule-based recommendations.
`webhookUrl`	string	-	No	URL to receive results when analysis completes (Zapier, Make, n8n)

Pricing

This actor uses pay-per-event billing:

Event	Description	Price
Domain Analysis	Complete robots.txt analysis including syntax validation, best practices check, and CMS detection	$0.05
Companion File Check	Check for llms.txt, security.txt, and sitemap.xml companion files	$0.02

Troubleshooting

"API key is required"

Add your API key in the input

"API error 403" or "Unauthorized"

Your API key is invalid, expired, or doesn't have access to this specific API endpoint. Double-check your key and account permissions.

"API error 429" or "Rate limit"

Too many requests. Wait a minute and try again, or reduce the number of items per run.

No results or empty dataset

Check the run log for error messages. Common causes:

Invalid input format (check the examples above)
API key without proper permissions
The target data doesn't exist or is too small to track

How do I test without an API key?

Built by John Rippy | Actor Arsenal

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

LLMs.txt AI Discovery Auditor

glowing_glove/llms-txt-ai-discovery-auditor

Check websites for llms.txt, llms-full.txt, robots AI crawler rules, sitemap hints, and AI search readiness gaps.

Ushba Khan

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Website Tech Audit Scraper: robots.txt, Sitemaps & Meta

arman-bd/website-tech-audit-scraper

Audit any site's public technical files: robots.txt rules, sitemap discovery and URL counts, ads.txt sellers, security.txt contacts, meta tags, structured data and the detected tech stack with evidence.

Arman Hossain

AI Training Blocker Checker

bipbip-apis/ai-training-blocker-checker

Check robots.txt, llms.txt, sitemap hints, and AI crawler blocking rules for websites and domains.

Daniel Christensen

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

Robots.txt Auditor

junipr/robots-txt-auditor

Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

junipr

Robots.txt Validator

predictable_function/my-actor-3

List of website base URLs whose robots.txt files will be validated

riya rawat

5.0

AI Crawler Policy Audit — robots.txt, llms.txt, noai

multiplex/ai-crawl-policy

See exactly which AI crawlers a site allows. Per domain it reads robots.txt for 25 named AI agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot…), plus ai.txt, llms.txt, noai meta, X-Robots-Tag and tdmrep.json, and quotes the rule behind every answer. Alerts on policy changes.

Daniel James

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.