llms.txt Auditor: AI Search Readiness Checker

Check any website's llms.txt for AI crawlers and answer engines. Finds missing or broken files, invalid structure and dead links, gives each site a readiness score and suggests fixes. Bulk-friendly, from $0.003 per site.

Checks a list of websites for llms.txt readiness. For each origin it fetches /llms.txt and /llms-full.txt, validates the index file against the llms.txt spec, optionally HEAD-checks the linked URLs, and reports whether robots.txt blocks common AI crawlers.

Input: startUrls[] (default https://example.com). Optional: checkLinks (default on), maxLinksToCheck (default 25), draftStarterLlmsTxt (default off), maxConcurrency (default 5), requestTimeoutSecs (default 20). The default site needs no credentials, so the store health run can finish on an empty or missing /llms.txt.

Output: one dataset item per site

• hasLlmsTxt, hasLlmsFullTxt, specValid
• issues[] (code, severity, file, message)
• linkCount, brokenLinks (each { url, status, error, file })
• sizeBytes (combined size of the accepted llms files)
• robotsAiDirectives for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended (blocked, matchedGroup, rule)
• readinessScore (0-100) and scoreLabel: the MasterInvictus AI readiness score (not an official standard)
• fixSuggestions
• starterLlmsTxt when draftStarterLlmsTxt is on and /llms.txt is missing (homepage <title>, meta description, and the first <nav> / header / main links; no LLM call)

Run summary is stored in the key-value store as SUMMARY (sites checked and errored, average MasterInvictus AI readiness score (not an official standard), spec-valid count, crawlers blocked). It is not a dataset row.

specValid is true only when /llms.txt exists and matches the spec: an H1 title first (the only required section), a blockquote summary directly under it is checked and recommended, every H2 is a markdown link list (- [name](url) plus optional : notes), and every link is an absolute http(s) URL or a relative path. /llms-full.txt is fetched and parsed the same way; structural mismatches are warnings, because the expanded file is often concatenated documentation.

Score (100 before penalties): llms.txt present 30, H1 15, blockquote summary 15, H2 link section 10, syntactically valid links 10, llms-full.txt present 10, AI crawlers allowed 10. Checked links that fail subtract up to 15. Unreachable domains get readinessScore: null and are not charged.

Fails loudly: the run exits non-zero when startUrls is empty or invalid, or when no domain could be checked.

Starter draft: with draftStarterLlmsTxt: true, a missing /llms.txt produces starterLlmsTxt from the homepage title, meta description, and main nav. Nothing is sent to an LLM.

Pricing (pay-per-event)

Event	Price
run-started: custom flat event, charged once per run and only after the first site is checked successfully. A run that checks no site is not billed (not Apify's memory-scaled apify-actor-start)	$0.01
site-audited: once per domain checked successfully. Failed or unreachable domains are not charged	$0.003

When publishing, register run-started and site-audited as custom events at these prices. Do not enable the synthetic start event, which is priced per GB of memory.

---

Built and maintained by MasterInvictus.