llms.txt Generator & AI Crawler Audit

Check a website's published crawler rules, and whether its CDN lets those crawlers in, before deciding what your agent can read. Audit robots.txt, llms.txt, sitemaps and sampled page metadata, then get an llms.txt draft. Each successfully audited page costs $0.009, plus $0.00005 per run start at default memory. Generation uses discovered URLs and metadata; it makes no runtime LLM calls and does not convert page bodies to Markdown.

Try it

{"urls":["https://llmstxt.org"],"maxPagesPerSite":5}

An empty input {} runs the same audit, sampling up to five pages. No upstream API key is needed.

Use from an agent

Add https://mcp.apify.com?tools=miacx/llms-txt-generator to your MCP client and sign in to Apify. Call the dedicated Actor tool, then use get-dataset-items to read the completed run's dataset. Or set APIFY_TOKEN to your Apify API token and call the API for JSON rows:

curl --fail-with-body 'https://api.apify.com/v2/actors/miacx~llms-txt-generator/run-sync-get-dataset-items' \
  -H "Authorization: Bearer $APIFY_TOKEN" -H 'Content-Type: application/json' \
  -d '{"urls":["https://llmstxt.org"],"maxPagesPerSite":5}'

What you get

The dataset contains one site summary per unique origin and one page row per sampled page. Read type, then status. An ok page is billable even when it contains findings; site summaries incur no Page events. A site row is robots_only when robots.txt was read but no page could be sampled: its crawler verdicts and live checks are complete, and each page row keeps its own error. Use generatedLlmsTxt on site rows for the draft and findings for issue codes and suggested fixes.

Selected fields from parsing the recorded tests/fixtures/resources.json page fixture:

{
  "type": "page",
  "status": "ok",
  "input": "https://llmstxt.org",
  "source": "https://llmstxt.org/core.html",
  "fetchedAt": "2026-10-31T02:07:54.430Z",
  "reason": null,
  "title": "Python source – llms-txt",
  "description": "Source code for llms_txt Python module, containing helpers to create and use llms.txt files",
  "canonical": null,
  "jsonLdTypes": []
}

Page rows also contain headings in document order, a confirmed markdownUrl or null, and findings. Missing metadata is null; missing collections are empty. An HTML page gets at most one Markdown-twin probe. A null URL means none detected, not proof none exists.

Site rows contain crawler decisions, resource statuses, sitemap URLs, discovered and successful sample counts, request counts and bytes. crawlers entries contain name, allowed and a one-based matching robots rule line, or null. allowed: null means the policy could not be read. Each entry also reports a live check: liveStatus is the HTTP status of one GET of the supplied page sent with that crawler's published user agent, and liveBlocked is true for any 4xx except 404 and 410 (such as 402 pay-per-crawl or 403), a 503 or a CDN challenge. Both are null when nothing was sent: the vendor publishes no user agent (ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, Bytespider), robots.txt disallows the crawler or this audit, or the request failed. allowed: true with liveBlocked: true adds a crawler_edge_blocked finding: robots.txt invites the crawler, but the CDN or firewall turns it away.

{ "name": "GPTBot", "allowed": true, "rule": null, "liveStatus": 403, "liveBlocked": true }

A live check shows how the site answers that user agent from Apify's network, not how it treats the vendor's own IP ranges. llmsTxtValid and llmsFullValid are null when their resources are unavailable. Fields belonging to the other row type are absent.

Page fetchedAt records the original observation time, including on cache hits. Site fetchedAt records summary production time. reason is null on success.

Input

Field	Default	Accepted input
urls	["https://llmstxt.org"]	Up to five public HTTP(S) URL strings. An empty list returns one no_results site row.
maxPagesPerSite	5	Integer 1–25. Includes the supplied page; same-origin sitemap URLs supply the rest, or same-origin links on the supplied page when no sitemap yields any.

Omitted fields use their defaults. Duplicate origins share one audit within a run, using the first supplied URL as the starting page. A supplied URL that redirects to another origin, such as https://docs.bsky.app/ to https://bsky.network/, is audited where it lands unless its own origin's sitemap lists pages. The site row's source is then the landed origin, and its robots.txt, sitemaps and llms.txt are the ones checked. Crawler decisions describe that exact URL, not every path on the site. Invalid URLs become individual site rows; malformed batch shapes fail input validation.

Pricing

One Page event costs $0.009, including metadata checks and up to one Markdown-twin probe. Five successfully audited pages cost $0.04505, including $0.00005 per run start at default 128 MB memory. One thousand Page events cost $9, plus run starts.

Pages with findings count. Site summaries, failed pages and no-result rows incur no Page events. Duplicate origins share work within the run. Cached successful pages still incur Page events. The $0.00005 run start applies even with no billable pages at default memory. Apify counts one start event per GB of memory, with a minimum of one event.

Limits and data freshness

• Each run allows five sites, four concurrent upstream loads, 180 network attempts, 25 seconds of fetch work and 8 MiB total response bytes. Live crawler checks add at most 14 header-only requests per site, outside the 180.
• Resources cap at 512 KiB. Each site inspects up to two sitemaps and four index children at one level. When no sitemap yields a same-origin URL, links on the supplied page that robots.txt permits are discovered instead.
• Discovery retains up to 1,000 URLs; the generated draft includes up to 200, grouped by first path section. Sampled pages that answered 404 or 410 are left out.
• Draft labels use sampled metadata where available and paths otherwise. Confirmed Markdown links take precedence. A page title such as Blog - AT Protocol is labelled Blog when the draft's H1 is AT Protocol.
• The draft's H1 is the supplied page's title. If that page fails, it is a site name most sampled titles share, such as Nature in Mice age slower | Nature, else the hostname.
• No JavaScript rendering, compressed sitemap expansion or cross-origin sitemap crawling runs. HTML soft-404 pages may appear successful.

The llms.txt check lints a leading H1 and links in optional H2 sections against the proposal. The llms-full.txt check only tests for available, nonempty non-HTML text. An HTML page served at either path, such as a catch-all soft 404, is reported as llms_txt_html or llms_full_html rather than as malformed Markdown. Crawler decisions report declared rules; they do not prove vendor enforcement or search-ranking effects.

Robots 404 means no published restriction. Other robots failures prevent page sampling. The audit respects rules for mia.cx-apify; a positive crawl-delay skips sampling within this synchronous budget.

Successful resources can be reused for up to one hour from the named store llms-txt-audit-cache. Shorter upstream freshness and cache restrictions apply. Cache snapshots retain at most 32 resources and 1 MiB per site; records remain until overwritten or deleted.

If a row fails

Status	Next step
invalid_url, private_host	Supply a public HTTP(S) URL without credentials or custom ports.
robots_denied, crawl_delay_required	Inspect the published policy; sampling was skipped.
http_403, http_404, network_error	Check the resource URL and public availability.
bot_challenge	The site served a bot challenge (Cloudflare, AWS WAF or similar) instead of the resource; it was not audited or billed.
unsupported_content, response_too_large	Supply a supported HTML or Markdown page within the size cap.
resource_limit, time_limit, budget_exhausted	Reduce sites or pages, or increase the Page charge budget where applicable.
robots_only	Use crawlers as usual; check page rows for why no page could be sampled.
no_results	Check page rows and findings; no sampled page succeeded.

For workflows exceeding the synchronous API's 300-second wait, use asynchronous runs.