Robots.txt Checker with AI Recommendations

Analyze robots.txt files for syntax errors, best practices, and CMS-specific optimizations. Includes AI-powered recommendations, sitemap validation, and companion file checks (llms.txt, security.txt).

BYOK (Bring Your Own Key) -- you provide your own API credentials.

---

Before You Start

This actor requires your own API credentials to fetch real data.

Where to get your key: Your Anthropic API key for enhanced AI recommendations. Without key, uses rule-based recommendations.

You can test with Demo Mode first (free, no key needed) to see the output format before committing.

---

Quick Start

Test with Demo Mode (free, no API key needed)

{
  "demoMode": true,
  "url": "https://example.com",
  "urls": "https://example.com"
}

Run with real data

{
  "demoMode": false,
  "url": "https://example.com",
  "urls": "https://example.com",
  "includeSitemapCheck": true,
  "includeLlmsTxtCheck": true,
  "includeSecurityTxtCheck": false,
  "includeAIRecommendations": true,
  "anthropicApiKey": "YOUR_API_KEY_HERE"
}

---

Input Parameters

Parameter	Type	Default	Required	Description
demoMode	boolean	true	No	Run with sample data to test the actor (free, no real URL fetching)
url	string	-	No	Single URL to analyze (will check /robots.txt at domain root)
urls	array	-	No	Multiple URLs to analyze (batch processing)
includeSitemapCheck	boolean	true	No	Verify that declared sitemaps exist and are valid
includeLlmsTxtCheck	boolean	true	No	Check for llms.txt (AI training opt-out file)
includeSecurityTxtCheck	boolean	false	No	Check for .well-known/security.txt
includeAIRecommendations	boolean	true	No	Generate AI-powered optimization recommendations
anthropicApiKey	string	-	Yes*	Your Anthropic API key for enhanced AI recommendations. Without key, uses rule-based recommendations.
webhookUrl	string	-	No	URL to receive results when analysis completes (Zapier, Make, n8n)

*Required when Demo Mode is off.

---

Pricing

This actor uses pay-per-event billing:

Event	Description	Price
Domain Analysis	Complete robots.txt analysis including syntax validation, best practices check, and CMS detection	$0.05
Companion File Check	Check for llms.txt, security.txt, and sitemap.xml companion files	$0.02

Demo mode is free -- no charges for sample data.

---

Troubleshooting

"API key is required"

You have Demo Mode turned off but didn't provide an API key. Either:

• Turn Demo Mode on to test with sample data
• Add your API key in the input

"API error 403" or "Unauthorized"

Your API key is invalid, expired, or doesn't have access to this specific API endpoint. Double-check your key and account permissions.

"API error 429" or "Rate limit"

Too many requests. Wait a minute and try again, or reduce the number of items per run.

No results or empty dataset

Check the run log for error messages. Common causes:

• Invalid input format (check the examples above)
• API key without proper permissions
• The target data doesn't exist or is too small to track

How do I test without an API key?

Enable Demo Mode in the input. This returns realistic sample data so you can verify the output format works for your workflow.

---

Built by John Rippy | Actor Arsenal