llms.txt Generator + Validator

Format submitted public resources into llms.txt, then check the file shape before publishing it.

This Actor takes site details, resource links, and optional user-chosen crawler rules. It generates llms.txt and an optional robots.txt snippet, then returns validation findings and an Evidence Pack.

What it checks

• top-level site heading
• Markdown resource-link shape
• absolute HTTP(S) resource URLs
• duplicate resource URLs
• submitted crawler-rule formatting
• crawler tokens outside the built-in reference list

Outputs

• generated llms.txt
• optional robots-snippet.txt
• REPORT.json, REPORT.html, and REPORT.md
• dataset finding rows

The Actor does not fetch the deployed site or test live crawler behavior.

Boundaries

llms.txt is treated as an emerging convention. This Actor formats and validates submitted inputs; it does not:

• choose a crawler policy for you
• provide legal or SEO advice
• promise that any AI or crawler service reads or follows the file
• predict traffic, indexing, ranking, citations, or AI answers
• publish or modify files on your site

Review the generated files before deployment.

Pricing

Pay per event:

Event	Price
llms-txt-report-generated	$0.30 per completed report
llms-txt-item-checked	$0.02 per generated or validated item

The standard sample contains five billable items and costs $0.40. With the current input limits, the theoretical maximum is 201 items and $4.32 per run. Input-validation failures are not charged. Set an Apify max cost per run if you need a lower account-level limit.

Data handling

Generated files and validation findings are written to normal Apify run storage. Submitted Actor input is also subject to Apify run storage and retention settings. Do not submit confidential policy text or private resource URLs.

Support boundary

Support covers reproducible Actor defects, documented input-format questions, missing generated files, and a mismatch between documented and recorded billing events.

Support does not cover policy selection, SEO or legal advice, editing site content, deploying files, custom templates, custom integrations, or evaluating AI-platform outcomes.

Related Actors

• AI Crawler Policy Preflight: inspect crawler-facing files after publication
• AI Crawler Policy Drift Monitor: track later changes to published crawler policy
• AI Visibility Signal Report: inventory llms.txt alongside other public-site signals

Minimal input

{
  "siteName": "Example Docs",
  "siteUrl": "https://example.com",
  "siteDescription": "Documentation and public resources for Example Docs.",
  "policyNotes": "This file lists the public resources that Example Docs wants to make easy to inspect.",
  "entries": [
    {
      "title": "Documentation",
      "url": "https://example.com/docs",
      "description": "Primary documentation entry point.",
      "section": "Docs"
    },
    {
      "title": "API Reference",
      "url": "https://example.com/api",
      "description": "Reference pages for public API endpoints.",
      "section": "Docs",
      "optional": true
    }
  ],
  "crawlerRules": [
    {
      "userAgent": "GPTBot",
      "directive": "Disallow",
      "path": "/"
    },
    {
      "userAgent": "OAI-SearchBot",
      "directive": "Allow",
      "path": "/"
    }
  ],
  "includeRobotsSnippet": true
}

API and scheduled use

Run the same validated input from Apify Console, the Actor API, CLI, schedules, or integrations. The run writes machine-readable findings to the default dataset and the complete report to the run storage.

Common input errors

Start with the published example and keep siteName, siteUrl, siteDescription, policyNotes, entries in the documented JSON shape. Malformed JSON, unsupported URL schemes, missing required values, and inputs above the stated limits are rejected rather than repaired silently.