📄 llms.txt Generator & Validator

Generate llms.txt files from your sitemap and validate AI search engine visibility. Checks robots.txt for AI bot access (OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended), scores your AI discoverability 0–100, and produces a ready-to-deploy llms.txt file.

📋 Table of Contents

• What is this?
• How to use
• Input
• Output
• JSON output example
• How much does it cost?
• Use cases
• Integrations
• Is it legal?
• Troubleshooting
• FAQ
• Tech stack
• Feedback & issues

🤔 What is this?

llms.txt is a new standard (llmstxt.org) that tells AI search engines — ChatGPT Search, Perplexity, Claude, Gemini — what your website is about and which pages matter most. It's like robots.txt but for AI citations.

This Actor:

1. Generates llms.txt from your sitemap — fetches all URLs, extracts page titles/descriptions, categorizes into sections, writes spec-compliant file
2. Validates existing llms.txt — checks structure, H1/blockquote presence, line count, link depth
3. Checks AI bot access in robots.txt — verifies OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended are allowed (not blocked)
4. Scores AI discoverability 0–100 across 8 categories

🚀 How to use

Step 1: Enter your URL

Provide your website root URL (e.g., https://example.com). The Actor auto-discovers your sitemap.

Step 2: Provide site name + description

Enter a human-readable site name and one-sentence description. These become the H1 header and blockquote in your llms.txt.

Step 3: Run

Click Start. The Actor:

• Fetches + parses your sitemap
• Extracts page metadata via headless browser
• Generates llms.txt with categorized sections
• Checks robots.txt for AI bot access
• Checks for AI discovery endpoints (/.well-known/ai.txt, /ai/summary.json)
• Scores your AI visibility 0–100

Step 4: Download

• Key-value store: llms.txt — Ready-to-deploy file
• Key-value store: OUTPUT — Full analysis JSON with score + recommendations
• Dataset — Summary row with score, grade, issues

📥 Input

Field	Type	Default	Description
url	string	required	Root URL (e.g., https://example.com)
siteName	string	hostname	Human-readable site name
siteDescription	string	"Website at..."	One-sentence description
maxUrls	integer	50	Max URLs in llms.txt (5–500, keep under 200)
validateOnly	boolean	false	Only check existing llms.txt + AI bot access. Don't generate.

📤 Output

Field	Type	Description
llmsTxt	string	Generated llms.txt content
llmsFullTxt	string	Same as llmsTxt (downloadable)
robotsAnalysis	object	AI bot access: which bots allowed/blocked
existingLlmsTxt	object	Analysis of existing llms.txt (if found)
aiDiscovery	object	AI discovery endpoints check
sitemapEntries	array	URLs with title, description, section
score	integer	AI visibility score 0–100
grade	string	Critical / Foundation / Good / Excellent
recommendations	array	Prioritized action items

📄 JSON output example

{
  "websiteUrl": "https://example.com",
  "generatedAt": "2026-07-04T04:30:00.000Z",
  "llmsTxt": "# Example Site\n\n> A website about...\n\n## Home\n\n- [Home](https://example.com): Main page\n\n## Blog\n\n- [Latest Posts](https://example.com/blog): Blog index\n\n## Links\n\n- [XML Sitemap](https://example.com/sitemap.xml)\n- [robots.txt](https://example.com/robots.txt)",
  "sitemapUrlsFound": 47,
  "sitemapUrlsIncluded": 47,
  "robotsAnalysis": {
    "aiBots": {
      "OAI-SearchBot": true,
      "PerplexityBot": true,
      "ClaudeBot": true,
      "Google-Extended": true,
      "GPTBot": false,
      "anthropic-ai": false
    },
    "blockingTraining": ["GPTBot", "anthropic-ai"],
    "allowingCitation": ["OAI-SearchBot", "PerplexityBot", "ClaudeBot", "Google-Extended"]
  },
  "existingLlmsTxt": {
    "exists": false,
    "issues": ["llms.txt not found"]
  },
  "aiDiscovery": {
    "hasAiTxt": false,
    "hasSummaryJson": false,
    "hasFaqJson": false
  },
  "score": 50,
  "grade": "Foundation",
  "recommendations": [
    "Create llms.txt file — this Actor can generate one for you",
    "Add /.well-known/ai.txt for AI agent discovery",
    "Add /ai/summary.json for AI summary access"
  ]
}

💰 How much does it cost?

$2 per website. Includes sitemap fetch, page metadata extraction, llms.txt generation, robots.txt analysis, AI discovery check, and scoring. No subscription.

Cost examples

Use case	Sites	Cost
Single site llms.txt	1	$2
Agency (10 clients)	10	$20
SEO audit portfolio (50 sites)	50	$100

🎯 Use cases

• AI SEO preparation — Get your site ready for ChatGPT Search, Perplexity, and Google AI Overviews citations
• Periodic audits — Track your AI visibility score over time
• Agency deliverable — Generate llms.txt for clients as part of SEO package
• Competitive analysis — Run on competitor sites to see their AI visibility setup
• Pre-launch check — Verify AI bot access before deploying a new site

🔗 Integrations

AI agents (MCP)

Works with Apify MCP server (https://mcp.apify.com). AI agents can run this Actor to check any website's AI visibility.

Chain with ADA Scanner

Run alongside our ADA Compliance Checker for a full AI+accessibility audit.

Automation

• Zapier — Generate llms.txt on new site deployment → push to GitHub
• Make — Schedule monthly AI visibility audits → email report
• Custom — Use Apify webhooks to trigger CI/CD llms.txt deployment

⚖️ Is it legal?

Yes. This Actor fetches public web pages (sitemap.xml, robots.txt) and generates a static text file. It does not modify, hack, or breach any website. llms.txt is an open standard (llmstxt.org).

🔧 Troubleshooting

Problem	Cause	Fix
0 URLs found	Sitemap not found or empty	Check for sitemap at /sitemap.xml or /sitemap_index.xml. Ensure robots.txt has Sitemap directive
Score is 0	No AI bot access + no llms.txt	Allow AI citation bots in robots.txt. Generate llms.txt
Slow generation	Many URLs to extract	Reduce maxUrls (default 50). Each URL requires browser page load for metadata
Empty llms.txt	Sitemap found but no URLs	Check sitemap XML is valid. Try validateOnly: true first

❓ FAQ

🛠️ Tech stack

• Apify SDK — Platform integration
• Playwright — Headless Chromium for page metadata extraction
• llmstxt.org — llms.txt specification

💬 Feedback & issues

• Apify Console — Issues tab on this Actor's page
• Review — Rate on Store page