llms.txt Generator & Checker - llms-full.txt, AI SEO (GEO)

What does llms.txt Generator do?

It creates an llms.txt file for any website, plus an optional llms-full.txt, so AI assistants and AI search engines can understand the site quickly. It can also validate the llms.txt files that websites already publish, in bulk. This is part of GEO (Generative Engine Optimization): making your content easy for ChatGPT, Claude, Perplexity and other AI tools to find and cite.

llms.txt is a proposed standard: a Markdown file at https://yoursite.com/llms.txt with the site's name, a short summary and a curated list of its important pages.

What the Actor does in generate mode (default):

• 🔎 Finds the pages: reads your sitemaps (via robots.txt and common locations, or the sitemap URL you give) or crawls from the home page if there is no sitemap; when a sitemap lists only a handful of pages, pages found by crawling are added (source shows sitemap + crawl)
• 🧭 Picks the key pages first: the home page and top-level pages, up to your page limit; optionally skips pages your robots.txt disallows
• 🗂️ Groups pages into sections by URL path (Docs, Blog, Pricing…; a leading version or language folder such as /3/ or /en/ is skipped), in the order you choose, and moves secondary pages (privacy, terms, login, tags) to an ## Optional section, as the proposal suggests
• 🏷️ Clean titles and descriptions: removes repeated title suffixes (" | Docs"), ignores site-wide default descriptions, uses the H1 when a page's title is just the site name
• ✍️ Your wording or AI: set your own site name and summary, or let AI write the one-paragraph summary from your home page
• 🤖 Optional AI descriptions for pages that have no meta description
• 📍 One section only: enter a URL with a path (e.g. https://docs.apify.com/academy) and only pages under that path are included; the home page is still read for the site name and summary. If that page links to nothing under its own path (docs sites that give every page its own folder, such as /en/getting-started/), the folder one level up is used (/en/, never the whole site) and SUMMARY scopeWidened says so. The result shows the scope used. If the website redirects to another host (e.g. docs.pydantic.dev/latest/ → pydantic.dev/docs/validation/latest/…), the new host and the matching section are used, and the result shows finalUrl and redirectedFrom
• 📚 llms-full.txt: the full text of every included page in one file, with navigation, footers and cookie notices removed by default
• 🔗 Direct download links: the first result row and the SUMMARY record carry llmsTxtUrl and llmsFullTxtUrl; the first row also holds the llms.txt text itself in llmsTxt (cut at 200,000 characters with llmsTxtTruncated: true), so API and n8n users need no second download
• 🧮 Token count per file: llmsTxtTokens and llmsFullTxtTokens (first row and SUMMARY) estimate how many tokens each file takes in an AI model's context (about 4 characters per token, 1 per Chinese, Japanese or Korean character), so you can see whether llms-full.txt fits before you use it
• 🔣 Safe links: ) and spaces in page URLs are percent-encoded (Mercury_(planet) → Mercury_%28planet%29), so links in the file never break

In validate mode, for each website it:

• 📥 fetches /llms.txt and /llms-full.txt (HTML "not found" pages served with 200 count as missing)
• 📐 checks the format: one # Title first, a > summary, ## Section headings, links written as - [name](url): notes, absolute URLs, empty sections, content type
• 🔗 requests a sample of the links to find broken ones
• 📏 estimates the size in tokens: llmsTxtTokens for llms.txt and llmsFullTxt.tokens for llms-full.txt (from its size)
• 🧮 returns a score (0-100) and a list of issues, each with a code, severity and message

How much does it cost?

Event	Price
Indexed page (generate)	$2.00 / 1,000 pages
AI page description (optional)	$5.00 / 1,000 descriptions
Validated website (validate)	$2.00 / 1,000 websites

No start fee. The site summary and llms-full.txt are included at no extra cost. A typical 50-page site costs about $0.10. Pages that cannot be read and unreachable websites are not charged. Higher Apify plans get volume discounts.

For comparison, other llms.txt generators in the Apify Store charge $0.004 to $0.005 per page, and llms.txt checkers $0.0005 to $0.01 per site, some with a start fee (checked September 2026).

Control your cost

• Charged: each page read into llms.txt ($0.002), each AI page description if you turn that option on ($0.005), and each validated website ($0.002).
• Free: pages that cannot be read or that redirect to another website, the same page reached through two URLs, unreachable websites, invalid input lines, the site summary and llms-full.txt. Every row has charged: true or false.
• Mistakes in the input end the run normally (status SUCCEEDED, no Actor failure): you get one free row with errorType: "invalid_input" that says what to fix (empty or invalid website, invalid sitemap URL, no valid website to validate). If the website cannot be reached the row has errorType: "network"; HTTP 404 gives not_found; a site that blocks us gives blocked; a robots.txt that disallows every selected page (with Respect robots.txt on) gives robots_disallowed. None of these is charged.
• At the start, the log and status message show the plan: number of pages (or websites) × price = the most the run can cost, compared with your maximum charge per run (set in the run options). Max pages is the main cost control in generate mode.
• When that maximum is reached, the run stops reading new pages and still writes llms.txt from the pages already read. The status message says how many were left out, and SUMMARY.notProcessed lists them (count and up to 100 URLs or inputs).
• If Apify restarts the run (server migration or Resurrect), pages already indexed are reused from the run's storage and not read or charged again (SUMMARY.resumedPages). In validate mode, websites already validated are skipped (SUMMARY.resumedSkipped).
• Time limits (Advanced settings, default 120 seconds each): a page that takes longer to read is left out and listed in SUMMARY.timedOutPages; in validate mode, a website that takes longer gets a row with errorType: "timeout". Neither is charged.

How to use it

Generate an llms.txt

1. Enter your website (e.g. https://example.com). To describe one section only, include its path (e.g. https://docs.apify.com/academy). Your own Include URL patterns take priority over the path.
2. Set Max pages (30–100 is typical for llms.txt).
3. Optional: set your own site name, summary, section order or sitemap URL.
4. Click Start.
5. Open the Output tab (or the first result row) and download llms.txt and llms-full.txt. Upload them to the root of your website: in WordPress with a file manager or an llms.txt plugin, in Webflow or Shopify through your hosting's file or redirect settings.

Input example

{
    "website": "python.org",
    "maxPages": 12,
    "siteName": "Python",
    "summary": "The official home of the Python programming language: downloads, documentation, community and news.",
    "sectionOrder": ["Downloads", "About"],
    "respectRobotsTxt": true
}

Output example

Real output of that input (September 2026, shortened):

# Python

> The official home of the Python programming language: downloads, documentation, community and news.

## About

- [Applications for Python](https://www.python.org/about/apps/)
- [Quotes about Python](https://www.python.org/about/quotes/)

## Main

- [Welcome to Python.org](https://www.python.org/): The official home of the Python Programming Language
- [3.14.7 Documentation](https://docs.python.org/3/): The official Python documentation.
- [Python Job Board](https://www.python.org/jobs/)

## Community

- [Internet Relay Chat](https://www.python.org/community/irc/)

The first dataset row points to the files; the other rows list every included page with its section and where its description came from (page or ai):

{
    "type": "files",
    "title": "llms.txt (download)",
    "llmsTxtUrl": "https://api.apify.com/v2/key-value-stores/<store>/records/llms.txt",
    "llmsFullTxtUrl": "https://api.apify.com/v2/key-value-stores/<store>/records/llms-full.txt",
    "siteName": "Python",
    "summarySource": "input",
    "pageCount": 11,
    "source": "crawl"
}

Validate existing llms.txt files

Choose Validate existing llms.txt files and list the websites. Lines with several domains separated by commas or spaces are split. A line that is not a website gets its own row with errorType: "invalid_input" (not charged) and the other sites are still validated. Every row has input, your original line. A site that refuses our request for /llms.txt (HTTP 401, 403, 418 or 429, or a bot-check page) is reported with errorType: "blocked" and not charged, instead of being scored as missing; the single Website field (prefilled for generate mode) is ignored when Websites to validate has entries.

Input example (validate)

{ "mode": "validate", "websites": ["docs.anthropic.com", "developers.cloudflare.com", "python.org", "vercel.com"], "linkSampleSize": 8 }

Output example (validate)

Real results (September 2026, shortened):

[
    { "website": "https://developers.cloudflare.com", "found": true, "score": 100, "linkCount": 107, "linksChecked": 8, "brokenLinks": [],
      "issues": [{ "code": "llms-full-txt", "severity": "info", "message": "llms-full.txt found." }] },
    { "website": "https://docs.anthropic.com", "found": true, "score": 90, "linkCount": 517,
      "issues": [
          { "code": "no-summary", "severity": "warning", "message": "No \"> summary\" blockquote after the title. A one-line summary helps AI tools understand the site quickly." },
          { "code": "notes-separator", "severity": "info", "message": "125 link(s) add notes with something other than \": \" ..." }
      ] },
    { "website": "https://python.org", "found": false, "score": 0,
      "issues": [{ "code": "missing", "severity": "error", "message": "/llms.txt answered HTTP 404. Create a Markdown file at /llms.txt with your site name, a one-line summary and links to key pages." }] }
]

Scoring: each error costs 30 points, each warning 10, broken links 5 each (at most 30); info items are free. A missing file scores 0. Sites without an llms.txt are sales leads for agencies: generate one for them with this Actor.

The SUMMARY record has status (SUCCESS, PARTIAL_RESULTS, FAILED, NO_RESULTS or LIMIT_REACHED), costPlan, notProcessed and, in generate mode, the download links and scope. Failed items carry an errorType.

Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/llms-txt-generator) to Claude, Cursor or any MCP client, then ask e.g. "Create an llms.txt for example.com with the Docs section first" or "Check which of these 20 sites have a valid llms.txt."

{ "website": "example.com", "maxPages": 30 }

Failed items are not charged and carry an errorType.

Use it from code and integrations

Run it from your own code with the Apify API. This call waits for the run and returns the results as JSON (replace YOUR_TOKEN with your Apify API token):

curl -X POST "https://api.apify.com/v2/acts/tidytools~llms-txt-generator/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"website":"https://docs.apify.com","maxPages":30}'

The synchronous endpoint waits up to 5 minutes. For bigger runs, start the run with POST https://api.apify.com/v2/acts/tidytools~llms-txt-generator/runs and read the dataset when it finishes, or use the apify-client package for JavaScript or Python.

Schedules and integrations: run it daily or weekly with Apify Schedules, get a webhook when a run finishes, or send the results to Zapier, Make, n8n, Google Sheets, Slack and other apps with Apify integrations. Results can be exported as JSON, CSV, Excel or XML.

FAQ

Does the Actor collect usage data? The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

Will llms.txt improve my rankings? Not directly. llms.txt is a proposal, not an official standard; AI companies decide whether and how to use it. It is cheap to add and helps AI tools that do read it.

Can I edit the result? Yes. Set the site name, summary and section order in the input, or edit the downloaded file before uploading it.

I regenerate llms.txt on a schedule (n8n, Make, API). Why does the file change every time? The AI summary is worded differently on each run. Set your own summary (and site name) in the input: the file then changes only when your pages change. Read the text from llmsTxt in the first dataset row and write it to your site or repository.

My site already has an llms.txt. Does generate mode read it? No. Generate mode builds a new file from your pages (sitemap or crawl) and never reads or overwrites the one on your site. To check the existing file, use validate mode.

Does it run JavaScript or read pages behind a login? Pages are read with plain requests first and a real browser as a fallback, so most JavaScript sites work (for example react.dev). Pages behind a login or bot protection are skipped and not charged.

A big site gives only 100 pages. Why? Max pages (default 100, up to 2,000) is the limit and also your cost control; the home page and shallow paths come first. Raise it, or use Include URL patterns to describe one section.

Can it check an existing llms.txt instead of generating one? Yes. Set What to do to Validate existing llms.txt files (many websites) and list the sites in Websites to validate: for each one it fetches /llms.txt and /llms-full.txt, checks the format, requests a sample of links (Links to test per site (validate mode)) and returns a score (0-100) with a list of issues. Each validated website costs $2.00 per 1,000, and unreachable websites, websites that block the request and invalid lines are not charged.

Tips

• Review the file before publishing it: llms.txt should describe what you want AI tools to know first.
• Use exclude patterns to leave out archives, tags or duplicate language versions. A pattern ending in /** (for example **/legal/**) also leaves out the folder page itself (/legal).
• Advanced settings: Plain HTTP requests from and Proxy help with sites that block data-center requests (proxy usage is billed to your Apify account).
• Check which AI crawlers can actually read your site with our AI Crawler Access Checker, or get a full AI search readiness score with GEO Readiness Audit.

Related tools (by TidyTools)

Tool	Use it to
GEO Readiness Audit	Score how well AI search can reach and read your site (firewall test, content without JavaScript, llms.txt, structured data), with an HTML report and an optional check of whether Google AI Overviews cite you
AI Crawler Access Checker	See which of 26 AI crawlers your robots.txt allows, read Cloudflare Content Signals, and track changes across thousands of domains
Website Change Monitor	Get an alert when a page changes, for example your published llms.txt or a competitor's
Sitemap URL Extractor	List every URL a site publishes in its sitemaps, to see which pages your llms.txt should cover
SEO Audit Crawler	Audit titles, links, speed and structured data across the whole site

Limitations

• llms.txt is a proposal, not an official standard; AI companies decide whether and how to use it.
• Pages that block automated access are skipped.
• Only public pages; no logins.

Support

Open an issue in the Issues tab with the website and your input. Issues are checked regularly.