llms.txt Validator & Generator

Check whether a website publishes a valid /llms.txt, find out what is wrong with it, and get a draft llms.txt when it has none.

For each site you enter, this Actor can:

• Validate /llms.txt against the llmstxt.org format (v2, August 2026): H1 title first, optional > summary blockquote, no headings in the preamble, ## sections with - [name](url): notes link lists, and an Optional section placed last. It also checks HTTP status, content type, size, UTF-8, redirects, relative, invalid and duplicate links, raw HTML, and unclosed code fences.
• Check links: every listed URL is requested with HEAD, falling back to GET, up to a cap. Broken links are reported, along with whether the links point to markdown or plain text, which the spec recommends.
• Score the file 0–100 with a letter grade, give pass/warn/fail for each check, and list fix suggestions in order of importance.
• Check /llms-full.txt (optional): whether it exists, its status, size and content type, and whether it starts with an H1.
• Generate a draft llms.txt. It is built from the homepage title and meta description, plus the titles and meta descriptions of a capped sample of pages taken from the site's sitemap. Pages are grouped into ## sections, and blog, news, legal and changelog pages go under ## Optional. The draft is saved as a text file in the key-value store and included in the dataset item.

It only makes plain HTTP requests (no browser, no proxy) to public URLs.

What it fetches (and what it doesn't)

• llms.txt files: /llms.txt at the site's origin, or the exact llms.txt URL you enter (e.g. https://fastht.ml/docs/llms.txt), plus the sibling llms-full.txt if enabled.
• Linked URLs: the links listed in that llms.txt (link check), up to maxLinksToCheck.
• For generation: robots.txt, the sitemap(s) (from robots.txt Sitemap: lines, else /sitemap.xml; sitemap indexes and gzip are supported; at most 6 sitemap files), the homepage, and at most maxSitemapPages pages.
• robots.txt is respected for page sampling and link checks. The crawler token is LlmsTxtValidator; if robots.txt has no group for it, the * group applies.
  • Disallowed pages and links are skipped and reported as skipped.
  • If a site's robots.txt is unreachable (5xx or network error), no pages are sampled (RFC 9309).
  • Pages with meta robots noindex are left out of the draft.
  • Login, account, admin, cart and checkout paths are never sampled.
• Politeness:
  • One request per host at a time.
  • At least requestDelayMs (default 1000 ms) between requests to the same host.
  • A robots.txt Crawl-delay is honoured, up to 10 s.
  • User-Agent: LlmsTxtValidator/0.1 (Apify actor llms-txt-validator-generator; polite, 1 request per host at a time).
• Safety: private, local and reserved IP addresses are refused, including via redirects and DNS. Only http/https on default ports is allowed, and every response has a size cap.

Input

Field	Type	Default	Notes
startUrls	array of URLs	(required)	Up to 500 sites per run. Each entry is reduced to its origin; enter a full llms.txt URL to validate a file in a subpath. Duplicates are skipped.
mode	validate / generate / both	both
checkLinks	boolean	true	HEAD, then GET fallback (16 KB cap).
maxLinksToCheck	integer 0–200	20	Sampled round-robin across sections.
checkLlmsFullTxt	boolean	true	Informational; not part of the score.
includeRawLlmsTxt	boolean	false	Adds the first 100 KB of the file to the item.
maxSitemapPages	integer 1–100	15	Pages fetched for the draft.
requestDelayMs	integer 250–30000	1000	Per host.
maxConcurrency	integer 1–10	3	Sites processed in parallel; each host still gets one request at a time.

Example:

{
  "startUrls": [{ "url": "https://llmstxt.org" }, { "url": "https://vercel.com" }],
  "mode": "both",
  "maxSitemapPages": 15,
  "maxLinksToCheck": 20
}

Output

Dataset: one item per site. Table views are Overview, Checks & fixes and Generated drafts.

• summary: flat key numbers (state, score, grade, pass/warn/fail counts, links checked, broken links, llms-full.txt state, draft status).
• validation: HTTP details, score, grade, checks[] (id, title, status = pass/warn/fail/skip, weight, message, fix), suggestions[], the parsed title, summary and sections, and linkCheck (totals, brokenLinks[], per-link results[]).
• llmsFullTxt: state (present / absent / html-response / empty / error), HTTP status, size, notes.
• generation: status (generated / skipped / failed) with reason, the draft text llmsTxt, its sections, sitemap stats, pages sampled, used and skipped (with reasons), selfCheckScore (the draft run through this Actor's own validator), and kvKey / kvUrl.
• robotsTxt, requestsMade, chargedEvents (the pay-per-event events this item is billable for; the actual charge is made by Apify, e.g. nothing is billed while the Actor has no PPE pricing), error, checkedAt.

Key-value store: each draft is saved as llms-txt_<domain> (plus _<path> for subpath files) as text/plain. A SUMMARY record holds the run totals.

How the score works

Each check has a weight. A pass earns the full weight, a warn earns half, a fail earns nothing, and a skip is left out.

• Heaviest checks: existence (30), H1 first (20), links resolve (15) and list format (12).
• Missing or HTML response: if /llms.txt is missing or returns an HTML page (a soft 404), the score is 0.
• Grades: A ≥ 90, B ≥ 80, C ≥ 65, D ≥ 50, F below 50.
• Not an official score: it is this Actor's own heuristic. llmstxt.org has no official scoring.

Real example output

These results come from a local run on 2026-09-25 around 20:52 PT (npm run local, same code, default caps: 15 pages, 20 links, 1 s delay). Sites: llmstxt.org, docs.anthropic.com, vercel.com, fastht.ml/docs/llms.txt, www.gnu.org. The full output is in local-output.json in the source.

Site	llms.txt	Score	Pass/Warn/Fail	Links checked (broken)	llms-full.txt	Draft (sitemap URLs → pages used)	Requests
llmstxt.org	200, 637 B	100 A	19/0/0	3 (0)	404	7 → 6	14
docs.anthropic.com	200 via 2 redirects to platform.claude.com, 69,098 B, 637 links	90 A	15/5/0	20 of 637 (0)	200, over 1 MB cap	sitemap is on another host → homepage links, 15 used	43
vercel.com	200, 4,766 B	94 A	18/2/0	20 of 24 (0)	200 but HTML	7,375 → 15	40
fastht.ml/docs/llms.txt	200, 4,796 B	100 A	20/0/0	18 (0), 2 skipped by robots.txt	404	38 → 15	41
www.gnu.org	404	0 F	0/0/1	none	404	5,333 → 15 (5 of 20 sampled were 404)	27

What the warnings were:

• docs.anthropic.com:
  • Cross-origin redirect.
  • No > summary.
  • 133 of 770 list items without a link.
  • 2 sections without links.
  • 12 H3 sub-headings inside sections.
• vercel.com: 8 of 32 list items without a link, in 2 prose sections ("When to use Vercel", "How agents should use Vercel").

Dataset item for vercel.com (excerpt; long arrays shortened with "…"):

{
  "input": "vercel.com",
  "domain": "vercel.com",
  "origin": "https://vercel.com",
  "mode": "both",
  "summary": {
    "llmsTxtState": "present",
    "score": 94,
    "grade": "A",
    "passed": 18,
    "warnings": 2,
    "failed": 0,
    "linksChecked": 20,
    "brokenLinks": 0,
    "llmsFullTxtState": "html-response",
    "draftStatus": "generated",
    "draftLinks": 15,
    "sitemapUrlsFound": 7375,
    "draftUrl": null
  },
  "validation": {
    "url": "https://vercel.com/llms.txt",
    "finalUrl": "https://vercel.com/llms.txt",
    "httpStatus": 200,
    "contentType": "text/plain; charset=utf-8",
    "bytes": 4766,
    "estimatedTokens": 1192,
    "state": "present",
    "score": 94,
    "grade": "A",
    "passed": 18,
    "warnings": 2,
    "failed": 0,
    "title": "Vercel",
    "sectionCount": 9,
    "linkCount": 24,
    "checks": [
      {
        "id": "list-format",
        "title": "List items use `- [name](url): notes`",
        "status": "warn",
        "weight": 12,
        "message": "8 of 32 list item(s) are malformed, e.g. line 9: List item has no markdown link [name](url).",
        "fix": "Each list entry in a `##` section should be `- [Link name](https://url): optional notes`."
      },
      {
        "id": "section-links",
        "title": "Every H2 section contains at least one link",
        "status": "warn",
        "weight": 5,
        "message": "Section(s) without links: \"When to use Vercel\" (line 7), \"How agents should use Vercel\" (line 15).",
        "fix": "Add `- [name](url)` links to these sections, or move their prose into the details area above the first `##`."
      },
      "… 18 more checks with status \"pass\" …"
    ],
    "suggestions": [
      "Each list entry in a `##` section should be `- [Link name](https://url): optional notes`.",
      "Add `- [name](url)` links to these sections, or move their prose into the details area above the first `##`."
    ],
    "linkCheck": {
      "totalUniqueLinks": 24,
      "checked": 20,
      "ok": 20,
      "broken": 0,
      "skipped": 0,
      "notChecked": 4,
      "results": [
        {
          "url": "https://vercel.com/get-started.md",
          "href": "https://vercel.com/get-started.md",
          "section": "Agent setup",
          "line": 23,
          "outcome": "ok",
          "httpStatus": 200,
          "method": "HEAD",
          "finalUrl": "https://vercel.com/get-started.md",
          "contentType": "text/markdown; charset=utf-8",
          "redirects": 0,
          "llmFriendly": true,
          "reason": null
        },
        "… 19 more …"
      ]
    }
  },
  "llmsFullTxt": {
    "url": "https://vercel.com/llms-full.txt",
    "finalUrl": "https://vercel.com/llms-full.txt",
    "httpStatus": 200,
    "state": "html-response",
    "contentType": "text/html; charset=utf-8",
    "bytes": 1048576,
    "truncated": true,
    "estimatedTokens": null,
    "title": null,
    "notes": [
      "Returned an HTML page instead of a text file (probably a soft 404)."
    ]
  },
  "generation": {
    "status": "generated",
    "kvKey": "llms-txt_vercel.com",
    "title": "Vercel",
    "summary": "The autonomous stack for every app and agent.",
    "sections": [
      {
        "name": "Docs",
        "links": 2
      },
      {
        "name": "Templates",
        "links": 2
      },
      {
        "name": "Academy",
        "links": 2
      },
      {
        "name": "Ai Gateway",
        "links": 2
      },
      {
        "name": "Connect",
        "links": 2
      },
      {
        "name": "Pages",
        "links": 1
      },
      {
        "name": "Optional",
        "links": 4
      }
    ],
    "linkCount": 15,
    "sitemap": {
      "sources": [
        "https://vercel.com/sitemap.xml"
      ],
      "filesFetched": 1,
      "urlsFound": 7375,
      "truncated": false,
      "fallbackToHomepageLinks": false
    },
    "pagesSampled": 15,
    "pagesUsed": 15,
    "selfCheckScore": 100,
    "llmsTxt": "(full draft text, shown below)"
  },
  "robotsTxt": {
    "url": "https://vercel.com/robots.txt",
    "httpStatus": 200,
    "state": "ok",
    "sitemaps": [
      "https://vercel.com/sitemap.xml"
    ],
    "crawlDelaySeconds": null
  },
  "requestsMade": 40,
  "chargedEvents": [
    "site-validated",
    "llms-txt-generated"
  ],
  "checkedAt": "2026-09-26T03:52:11.589Z"
}

Generated draft for vercel.com (key-value record llms-txt_vercel.com):

# Vercel

> The autonomous stack for every app and agent.

## Docs

- [Vercel Documentation](https://vercel.com/docs): Find guides and reference documentation for building, deploying, and managing applications with Vercel, including the CLI, SDKs, and APIs.
- [Vercel CLI Overview](https://vercel.com/docs/cli): Learn how to use the Vercel command-line interface (CLI) to manage and configure your Vercel Projects from the command line.

## Templates

- [Templates](https://vercel.com/templates): Jumpstart your app development process with pre-built solutions from Vercel and our community.
- [Go Templates](https://vercel.com/templates/go): Jumpstart your app development process with pre-built solutions from Vercel and our community.

## Academy

- [Vercel Academy](https://vercel.com/academy): Go from beginner to expert by learning the ins and outs of Vercel, Next.js, Turborepo, AI SDK and more to build fully functional apps that use all the latest features.
- [Builders Guide to the AI SDK](https://vercel.com/academy/ai-sdk): Build production-ready AI features with the AI SDK & Next.js. Learn LLMs, prompting, extraction, streaming, & more.

## Ai Gateway

- [AI Gateway](https://vercel.com/ai-gateway): Use hundreds of AI models without managing rate limits and API keys through Vercel AI Gateway.
- [Browse AI Gateway Models](https://vercel.com/ai-gateway/models): Compare capabilities, pricing, and performance across leading AI providers.

## Connect

- [Connect](https://vercel.com/connect): Securely connect agents to the tools, data, and services they need to take action.
- [X Integration](https://vercel.com/connect/x): Use X with AI apps and agents through Vercel Connect. Follow a complete setup guide for secure runtime access.

## Pages

- [CQRS for read-heavy applications](https://vercel.com/i/cqrs): Learn how to judge whether CQRS earns its complexity, when a simpler pattern solves the same read/write problem, and how the split maps to serverless.

## Optional

- [Blog](https://vercel.com/blog): Blog posts from the Vercel team and community. Discover the latest in web development.
- [Next.js](https://vercel.com/blog/next): We're very proud to open-source Next.js, a small framework for server-rendered universal JavaScript webapps.
- [Changelog](https://vercel.com/changelog): The Vercel changelog: stay up to date with the latest platform changes, new features, product improvements, and release notes for developers.
- [Introducing npm i chat - One codebase, every chat platform](https://vercel.com/changelog/chat-sdk): Chat SDK is now open source and available in public beta. It's a TypeScript library for building chat bots that work across Slack, Microsoft Teams, Google Chat, Discord, GitHub, and Linear — from a…

Generated draft for www.gnu.org, a site without an llms.txt:

# The GNU Operating System and the Free Software Movement

> Since 1983, developing the free Unix style operating system GNU, so that computer users can have the freedom to share and improve the software they use.

## Education

- [Frequently Asked Questions About Free Software and Education](http://www.gnu.org/education/edu-faq.html)
- [Why Educational Institutions Should Use and Teach Free Software](http://www.gnu.org/education/edu-why.html)

## Graphics

- [Dog Cartoon](http://www.gnu.org/graphics/dog.html)
- [GNU/Linux buttons by Vladimir Tsarkov](http://www.gnu.org/graphics/BVBN.html)

## Licenses

- [The BSD License Problem](http://www.gnu.org/licenses/bsd.html)
- [Design Science License](http://www.gnu.org/licenses/dsl.html)

## Philosophy

- [The X Window System Trap](http://www.gnu.org/philosophy/x.html)
- [The Ballad of Dennis Karjala](http://www.gnu.org/philosophy/bdk.html)

## Server

- [Tasks to do for www.gnu.org](http://www.gnu.org/server/tasks.html)
- [About GNU Servers](http://www.gnu.org/server/server.html)

## Pages

- [About GNU](http://www.gnu.org/gnu/gnu.html)
- [Laugh along with GNU](http://www.gnu.org/fun/humor.html)
- [List of 1999 FSF Supporters](http://www.gnu.org/thankgnus/1999.html)
- [GNU/Linux Distros](http://www.gnu.org/distros/distros.html)
- [Additions to the Malware Section](http://www.gnu.org/proprietary/all.html)

These drafts are starting points. Section names come from URL paths (hence "Ai Gateway"), and link names and notes are the pages' own titles and meta descriptions. Review them, point the links at markdown versions of pages where you have them, and trim the lists before publishing.

Pricing

Pay per event. You pay only for results that land in your dataset:

Event	When
site-validated	Once per site whose llms.txt URL returned an HTTP response (including 404, meaning "no llms.txt found") and was analysed.
llms-txt-generated	Once per site for which a draft was generated and saved.

Not charged: invalid inputs, DNS failures, timeouts, refused private addresses, and generation skipped because of robots.txt.

• Prices are set by the Actor owner and shown on the Actor's Store page. The proposed prices are USD 0.004 per validated site and USD 0.02 per generated draft; see PRICING.md in the source for the benchmark.
• Spending limit: if you set a maximum cost per run, the Actor stops starting new sites when the limit is reached and finishes normally. In both mode it only generates when both events still fit.

Limits

• Plain HTTP only: no JavaScript rendering. Sites that build their <title> and meta description client-side give weaker drafts.
• The draft uses titles and meta descriptions only. It does not read page bodies, does not write prose for the details section, and does not create .md versions of pages.
• The link check is capped (maxLinksToCheck, max 200), so large files are only sampled. HEAD responses are trusted; GET is used only when HEAD fails or returns an error status.
• Caps:
  • llms.txt: 5 MB
  • llms-full.txt: 1 MB read (larger files are reported as truncated)
  • Each sitemap: 10 MB
  • 6 sitemap files and 20,000 sitemap URLs per site
  • 500 sites per run
• Sitemap URLs on a different host than the site are ignored; the Actor falls back to the homepage's same-origin links. This happened for docs.anthropic.com, whose sitemap is on platform.claude.com.
• The format checks follow llmstxt.org v2. The spec is informal markdown, so some checks are recommendations (warn), not hard rules. Only these fail: a missing file, a missing H1, bad UTF-8, more than half the list items malformed, or more than 20% of checked links broken.

FAQ

Is a 404 on /llms.txt charged? Yes, as site-validated. Finding out that a site has no llms.txt is the result, and the item includes the fix. DNS errors and timeouts are not charged.

Does it follow robots.txt? Yes, for everything it crawls: sampled pages and link checks. The llms.txt, llms-full.txt, robots.txt and sitemap files themselves are fetched directly, because they are the public files being audited.

Why was a link "skipped"? The linked host's robots.txt disallows it for this crawler, or that robots.txt was unreachable. For example, gist.githubusercontent.com links in the fastht.ml file were skipped.

Does llms-full.txt affect the score? No. It is a widely used convention but not part of the llmstxt.org spec, so it is reported separately.

Can I validate an llms.txt that is not at the site root? Yes. Enter the full URL, e.g. https://fastht.ml/docs/llms.txt. Spec v2 lets a file sit at any path and cover the pages under it.

Does the Actor log in anywhere or use my cookies? No. It only makes anonymous requests to public URLs.