Website llms.txt, AI Bot Permissions & Manifest Extractor

An autonomous, ultra-fast Apify Actor that discovers, extracts, and validates website /llms.txt, /llms-full.txt, AI crawler permissions (GPTBot, ClaudeBot, PerplexityBot), and AI manifest metadata for any domain.

Essential for AI developer agents, RAG document ingestion pipelines, SEO teams auditing AI crawler access, and autonomous web research bots.

---

🤖 Features

• Automated /llms.txt Discovery: Probes /llms.txt, /llms-full.txt, and /.well-known/llms.txt endpoints in parallel.
• AI Bot Access Auditing: Deeply analyzes robots.txt directives specifically for major AI crawlers:
  • GPTBot (OpenAI)
  • ClaudeBot (Anthropic)
  • PerplexityBot (Perplexity)
  • CCBot (Common Crawl)
  • Google-Extended (Google Gemini)
  • Applebot-Extended (Apple Intelligence)
• Structured Markdown Parser: Converts raw llms.txt files into clean structured JSON with titles, summaries, documentation section headers, and direct URLs.
• Sitemap Extraction: Identifies all XML sitemaps referenced in robots.txt.
• Zero Browser Overhead: Sub-second HTTP execution (< 300ms per domain) with zero compute waste.

---

📥 Input Parameters

Field	Type	Default	Description
urls	Array	["https://docs.anthropic.com", "https://apify.com"]	List of target domains or URLs to inspect.
checkAiBots	Boolean	true	Analyze robots.txt for AI crawler permissions.
fetchFullLlmsTxt	Boolean	true	Attempt to fetch /llms-full.txt if available.
timeoutSecs	Integer	10	Timeout per domain in seconds.

---

📤 Output Format

{
  "domain": "docs.anthropic.com",
  "baseUrl": "https://docs.anthropic.com",
  "hasLlmsTxt": true,
  "llmsTxtUrl": "https://docs.anthropic.com/llms.txt",
  "llmsTxtLength": 1420,
  "hasLlmsFullTxt": false,
  "hasRobotsTxt": true,
  "sitemaps": ["https://docs.anthropic.com/sitemap.xml"],
  "aiBotPermissions": {
    "GPTBot": {
      "status": "ALLOWED",
      "source": "ALLOWED_DIRECTIVE"
    },
    "ClaudeBot": {
      "status": "ALLOWED",
      "source": "ALLOWED_DIRECTIVE"
    },
    "PerplexityBot": {
      "status": "ALLOWED",
      "source": "ALLOWED_DIRECTIVE"
    }
  },
  "llmsManifest": {
    "title": "Anthropic Documentation",
    "summary": "Official guides and reference docs for Claude 3.7",
    "sectionsCount": 3,
    "sections": [
      {
        "title": "Getting Started",
        "links": [
          {
            "title": "Quickstart Guide",
            "url": "https://docs.anthropic.com/quickstart",
            "description": "Build your first Claude app in 5 minutes"
          }
        ]
      }
    ]
  }
}

---

🤖 AI Agent Integration (TypeScript / Python)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('Richigga/website-llms-txt-manifest-extractor').call({
  urls: ['https://docs.anthropic.com', 'https://openai.com']
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].aiBotPermissions);

---

💰 Unit Economics

• Pricing Model: Pay Per Event / Pay Per URL ($0.002 per domain)
• External Cost: $0.00 (Direct public HTTP endpoints)
• Compute Time: < 300ms (~0.00005 Apify CU)
• Gross Margin: > 97%

---

📄 License

Developed autonomously under Apache-2.0.