LLMs.txt Discovery Auditor

Audit a website for /llms.txt and /llms-full.txt AI discovery readiness. Returns a readiness score, markdown structure summary, and actionable recommendations. The llms.txt convention lets websites expose clean markdown for AI crawlers and LLMs.

Use cases

• GEO/AI-SEO consultants - audit client sites for AI crawler readiness
• Documentation teams - verify /llms.txt is properly structured and linked
• B2B SaaS marketers - ensure product docs are discoverable by AI search
• DevRel teams - check that documentation is AI-accessible
• AI search consultants - benchmark competitor AI discovery readiness

Input

Field	Type	Required	Default	Description
startUrl	string	yes	-	Public website URL to audit
checkFullFile	boolean	no	true	Also check /llms-full.txt
timeoutSeconds	integer	no	10	Per-request timeout (3-30 seconds)

Example input

{
  "startUrl": "https://example.com",
  "checkFullFile": true,
  "timeoutSeconds": 10
}

Output

A single dataset item with the full audit:

Field	Type	Description
inputUrl	string	The URL provided
siteOrigin	string	Normalized site origin
ok	boolean	Whether any llms.txt file was found
score	integer	AI discovery readiness score (0-100)
checkedAt	string	ISO timestamp
files	array	Per-file audit results (see below)
recommendations	array	Actionable recommendations

files array

Each entry contains:

Field	Type	Description
path	string	File path checked (/llms.txt or /llms-full.txt)
present	boolean	Whether the file exists and has content
status	integer	HTTP status code
finalUrl	string	Final URL after redirects
contentType	string	HTTP content-type
byteLength	integer	File size in bytes
truncated	boolean	Whether the file exceeded the 1 MB limit
title	string	H1 title extracted from the markdown
headingCount	integer	Number of headings in the file
headings	array	First 20 headings with level and text
linkCount	integer	Number of markdown links
links	array	First 100 links with text, URL, and domain
uniqueLinkedDomains	array	Unique domains linked from the file
likelyDocsLinks	array	Links matching docs/api/guide/reference patterns
lineCount	integer	Total line count
warnings	array	Issues found (missing file, empty, HTML instead of markdown, no links, etc.)

Score breakdown

Condition	Points
/llms.txt present	+45 (baseline)
H1 title present	+10
2+ headings	+10
5+ links	+15
2+ documentation links	+10
/llms-full.txt present	+10
Maximum	100

Example output

{
  "inputUrl": "https://example.com",
  "siteOrigin": "https://example.com",
  "ok": true,
  "score": 70,
  "checkedAt": "2026-07-06T12:00:00.000Z",
  "files": [
    {
      "path": "/llms.txt",
      "present": true,
      "status": 200,
      "title": "Example",
      "headingCount": 3,
      "linkCount": 8,
      "uniqueLinkedDomains": ["docs.example.com", "api.example.com", "example.com", "github.com"],
      "likelyDocsLinks": [
        { "text": "Documentation", "url": "https://docs.example.com", "domain": "docs.example.com" },
        { "text": "API Reference", "url": "https://api.example.com", "domain": "api.example.com" }
      ],
      "warnings": []
    },
    {
      "path": "/llms-full.txt",
      "present": false,
      "status": 404,
      "warnings": ["/llms-full.txt is missing"]
    }
  ],
  "recommendations": [
    "Add /llms-full.txt for complete long-form context when practical"
  ]
}

Security

• Only public HTTP/HTTPS URLs are accepted
• SSRF protection: localhost, private IPv4/IPv6, and DNS-resolving-to-private IPs are blocked
• URLs with embedded credentials are rejected
• Redirects are manually revalidated before following
• No browser automation, no proxies

Pricing

Pay per event:

Event	Price
Actor start	$0.005
Site audited	$0.01

A single-site audit costs approximately $0.015.

FAQ

What is llms.txt? llms.txt is a convention where websites expose a markdown file at /llms.txt to help AI crawlers and LLMs understand their content. It serves a similar role to robots.txt but for AI models.

What is the difference between /llms.txt and /llms-full.txt? /llms.txt is a concise overview with curated links. /llms-full.txt is the complete documentation in markdown. The actor checks both by default.

What does the readiness score mean? The score (0-100) is based on file presence, content quality (headings, links, title), and whether the response is markdown rather than HTML. Higher scores indicate better AI discovery readiness.

Can I audit multiple sites in one run? This actor audits one site per run. For bulk audits, schedule multiple runs.