llms.txt Generator & AI Crawler Audit
Pricing
Pay per event
Check AI crawler access, llms.txt, sitemaps and sampled page metadata. Generate an llms.txt draft from discovered URLs and page metadata.
Row type
type
Optional
site for a site summary, page for a sampled page. Fields belonging to the other row type are absent.
Status
status
Optional
ok means processing succeeded, even when findings exist. Only ok page rows incur Page events. A site row is robots_only when robots.txt was read, so crawler verdicts are complete, but no page could be sampled. Failures include invalid_url, private_host, http_403, http_404, network_error, bot_challenge (a challenge or interstitial page served instead of the resource), robots_denied, crawl_delay_required, unsupported_content, response_too_large, resource_limit, time_limit, budget_exhausted and no_results.
Input
input
Optional
Original site URL supplied by the caller. Duplicate origins use the first supplied URL. Empty for an empty-batch no_results row.
Source
source
Optional
Audited site origin on summaries or final resource URL on sampled pages. When the supplied URL redirects to another site that is audited instead, the summary's source is that landed origin. Input failures retain the supplied URL.
Fetched at
fetchedAt
Optional
ISO 8601 UTC page observation time, retained on cached resources. Site summaries use output production time; unfetched failures use attempt time.
Reason
reason
Optional
Plain-language explanation for unsuccessful rows; null on success.
Page title
title
Optional
HTML title on page rows; null when absent, on Markdown pages or on failure. Absent on site rows.
Meta description
description
Optional
HTML meta description on page rows; null when missing, on Markdown pages or on failure. Absent on site rows.
Canonical URL
canonical
Optional
Resolved canonical link on page rows; null when absent, invalid or on failure. Absent on site rows.
Heading outline
headings
Optional
Page heading objects with level (1-6) and text, in document order; repeated headings remain. Empty when none or on failure. Absent on site rows.
[ { "level": 1, "text": "Python source" }]JSON-LD types
jsonLdTypes
Optional
Distinct sorted @type strings from page JSON-LD, including nested graphs. Empty when none or on failure; absent on site rows.
[ "WebPage"]Markdown URL
markdownUrl
Optional
Confirmed Markdown representation URL on page rows. The supplied page can itself be Markdown; otherwise at most one .md twin is probed. Null means none detected or page failure, not proof that no Markdown version exists.
Findings
findings
Optional
Issue objects containing code and fix, in check order. Empty when no issues are recorded. Findings can occur on ok rows and do not make a successful page free.
[ { "code": "missing_canonical", "fix": "Add a canonical link identifying the preferred URL." }]Crawler access
crawlers
Optional
Site rows only. Per crawler: allowed and rule give the robots.txt verdict for the exact supplied URL, or for the landing page when the audit moved to the site it redirects to; liveStatus and liveBlocked give the result of one live GET of that page sent with the crawler's published user agent. Order follows the configured crawler list; empty for inputs not audited. Neither proves how the vendor's real crawler is treated.
[ { "name": "GPTBot", "allowed": true, "rule": null, "liveStatus": 403, "liveBlocked": true }]Robots status
robotsStatus
Optional
Site rows only. Fetch status for robots.txt, or not_checked before an audit. http_404 means no published restriction; other fetch failures prevent sampling.
llms.txt status
llmsTxtStatus
Optional
Site rows only. Fetch status for /llms.txt, or not_checked before an audit.
llms.txt valid
llmsTxtValid
Optional
Site rows only. True or false for structural llms.txt linting; null when unavailable. Checks a leading H1 and Markdown links in optional H2 sections, not content quality. An HTML page served at /llms.txt is false with finding llms_txt_html.
llms-full.txt status
llmsFullStatus
Optional
Site rows only. Fetch status for /llms-full.txt, or not_checked before an audit.
llms-full.txt valid
llmsFullValid
Optional
Site rows only. Whether /llms-full.txt is nonempty non-HTML text. Null when unavailable. An HTML page served there is false with finding llms_full_html. This is not validation against a separate full-text schema.
Sitemaps
sitemapUrls
Optional
Successfully parsed same-origin sitemap URLs in inspection order on site rows. Empty when none. Includes inspected index children; does not imply complete sitemap coverage.
[ "https://llmstxt.org/sitemap.xml"]Discovered pages
discoveredPages
Optional
Number of unique URLs discovered for this site, at most 1,000: the supplied page plus same-origin sitemap URLs, or same-origin links on the supplied page when no sitemap yields any. Zero before discovery; not the site's total page count.
Audited pages
sampledPages
Optional
Number of successfully audited pages for this site. Excludes failed or robots-denied pages; findings do not exclude a successful page.
Generated llms.txt
generatedLlmsTxt
Optional
Site-row llms.txt draft using up to 200 discovered URLs, grouped by first path section. Uses sampled metadata and confirmed Markdown URLs; unsampled labels come from paths. Empty for inputs not audited. Does not convert page bodies to Markdown.
Upstream requests
requests
Optional
Network attempts for this site, including redirects, transient retries and live crawler checks. Cache hits are excluded; zero before fetching.
Downloaded bytes
bytes
Optional
Decoded response-body bytes consumed for this site, excluding cache hits. Zero before fetching.