# llms.txt Generator & Checker - llms-full.txt, AI SEO (GEO) (`tidytools/llms-txt-generator`) Actor

Generate llms.txt and llms-full.txt for any website from its sitemap or a crawl, with sections and AI descriptions, or validate existing llms.txt files in bulk. $2 per 1,000 pages or sites.

- **URL**: https://apify.com/tidytools/llms-txt-generator.md
- **Developed by:** [Yukai Lin](https://apify.com/tidytools) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does llms.txt Generator do?

It creates an **llms.txt** file for any website, plus an optional **llms-full.txt**, so AI assistants and AI search engines can understand the site quickly. It can also **validate the llms.txt files that websites already publish**, in bulk. This is part of **GEO (Generative Engine Optimization)**: making your content easy for ChatGPT, Claude, Perplexity and other AI tools to find and cite.

[llms.txt](https://llmstxt.org) is a proposed standard: a Markdown file at `https://yoursite.com/llms.txt` with the site's name, a short summary and a curated list of its important pages.

What the Actor does in **generate** mode (default):

- 🔎 **Finds the pages**: reads your sitemaps (via robots.txt and common locations, or the **sitemap URL** you give) or crawls from the home page if there is no sitemap; when a sitemap lists only a handful of pages, pages found by crawling are added (`source` shows `sitemap + crawl`)
- 🧭 **Picks the key pages first**: the home page and top-level pages, up to your page limit; optionally skips pages your robots.txt disallows
- 🗂️ **Groups pages into sections** by URL path (Docs, Blog, Pricing…; a leading version or language folder such as `/3/` or `/en/` is skipped), in the **order you choose**, and moves secondary pages (privacy, terms, login, tags) to an `## Optional` section, as the proposal suggests
- 🏷️ **Clean titles and descriptions**: removes repeated title suffixes (" | Docs"), ignores site-wide default descriptions, uses the H1 when a page's title is just the site name
- ✍️ **Your wording or AI**: set your own **site name** and **summary**, or let AI write the one-paragraph summary from your home page
- 🤖 **Optional AI descriptions** for pages that have no meta description
- 📍 **One section only**: enter a URL with a path (e.g. `https://docs.apify.com/academy`) and only pages under that path are included; the home page is still read for the site name and summary. If that page links to nothing under its own path (docs sites that give every page its own folder, such as `/en/getting-started/`), the folder one level up is used (`/en/`, never the whole site) and SUMMARY `scopeWidened` says so. The result shows the `scope` used. If the website **redirects to another host** (e.g. `docs.pydantic.dev/latest/` → `pydantic.dev/docs/validation/latest/…`), the new host and the matching section are used, and the result shows `finalUrl` and `redirectedFrom`
- 📚 **llms-full.txt**: the full text of every included page in one file, with navigation, footers and cookie notices removed by default
- 🔗 **Direct download links**: the first result row and the `SUMMARY` record carry `llmsTxtUrl` and `llmsFullTxtUrl`; the first row also holds the llms.txt text itself in `llmsTxt` (cut at 200,000 characters with `llmsTxtTruncated: true`), so API and n8n users need no second download
- 🧮 **Token count per file**: `llmsTxtTokens` and `llmsFullTxtTokens` (first row and `SUMMARY`) estimate how many tokens each file takes in an AI model's context (about 4 characters per token, 1 per Chinese, Japanese or Korean character), so you can see whether llms-full.txt fits before you use it
- 🔣 **Safe links**: `)` and spaces in page URLs are percent-encoded (`Mercury_(planet)` → `Mercury_%28planet%29`), so links in the file never break

In **validate** mode, for each website it:

- 📥 fetches `/llms.txt` and `/llms-full.txt` (HTML "not found" pages served with 200 count as missing)
- 📐 checks the format: one `# Title` first, a `> summary`, `## Section` headings, links written as `- [name](url): notes`, absolute URLs, empty sections, content type
- 🔗 requests a sample of the links to find broken ones
- 📏 estimates the size in tokens: `llmsTxtTokens` for llms.txt and `llmsFullTxt.tokens` for llms-full.txt (from its size)
- 🧮 returns a **score (0-100)** and a list of **issues**, each with a code, severity and message

### How much does it cost?

| Event | Price |
|---|---|
| Indexed page (generate) | **$2.00 / 1,000 pages** |
| AI page description (optional) | $5.00 / 1,000 descriptions |
| Validated website (validate) | **$2.00 / 1,000 websites** |

**No start fee.** The site summary and llms-full.txt are included at no extra cost. A typical 50-page site costs about **$0.10**. Pages that cannot be read and unreachable websites are not charged. Higher Apify plans get volume discounts.

For comparison, other llms.txt generators in the Apify Store charge $0.004 to $0.005 per page, and llms.txt checkers $0.0005 to $0.01 per site, some with a start fee (checked September 2026).

### Control your cost

- **Charged**: each page read into llms.txt ($0.002), each AI page description if you turn that option on ($0.005), and each validated website ($0.002).
- **Free**: pages that cannot be read or that redirect to another website, the same page reached through two URLs, unreachable websites, invalid input lines, the site summary and llms-full.txt. Every row has `charged: true` or `false`.
- **Mistakes in the input end the run normally** (status SUCCEEDED, no Actor failure): you get one free row with `errorType: "invalid_input"` that says what to fix (empty or invalid website, invalid sitemap URL, no valid website to validate). If the website cannot be reached the row has `errorType: "network"`; HTTP 404 gives `not_found`; a site that blocks us gives `blocked`; a robots.txt that disallows every selected page (with **Respect robots.txt** on) gives `robots_disallowed`. None of these is charged.
- At the start, the log and status message show the plan: number of pages (or websites) × price = the most the run can cost, compared with your **maximum charge per run** (set in the run options). **Max pages** is the main cost control in generate mode.
- When that maximum is reached, the run stops reading new pages and still writes llms.txt from the pages already read. The status message says how many were left out, and `SUMMARY.notProcessed` lists them (count and up to 100 URLs or inputs).
- **If Apify restarts the run** (server migration or Resurrect), pages already indexed are reused from the run's storage and not read or charged again (`SUMMARY.resumedPages`). In validate mode, websites already validated are skipped (`SUMMARY.resumedSkipped`).
- **Time limits** (Advanced settings, default 120 seconds each): a page that takes longer to read is left out and listed in `SUMMARY.timedOutPages`; in validate mode, a website that takes longer gets a row with `errorType: "timeout"`. Neither is charged.

### How to use it

#### Generate an llms.txt

1. Enter your **website** (e.g. `https://example.com`). To describe one section only, include its path (e.g. `https://docs.apify.com/academy`). Your own *Include URL patterns* take priority over the path.
2. Set **Max pages** (30–100 is typical for llms.txt).
3. Optional: set your own **site name**, **summary**, **section order** or **sitemap URL**.
4. Click **Start**.
5. Open the **Output** tab (or the first result row) and download **llms.txt** and **llms-full.txt**. Upload them to the root of your website: in WordPress with a file manager or an llms.txt plugin, in Webflow or Shopify through your hosting's file or redirect settings.

##### Input example

```json
{
    "website": "python.org",
    "maxPages": 12,
    "siteName": "Python",
    "summary": "The official home of the Python programming language: downloads, documentation, community and news.",
    "sectionOrder": ["Downloads", "About"],
    "respectRobotsTxt": true
}
```

##### Output example

Real output of that input (September 2026, shortened):

```markdown
## Python

> The official home of the Python programming language: downloads, documentation, community and news.

### About

- [Applications for Python](https://www.python.org/about/apps/)
- [Quotes about Python](https://www.python.org/about/quotes/)

### Main

- [Welcome to Python.org](https://www.python.org/): The official home of the Python Programming Language
- [3.14.7 Documentation](https://docs.python.org/3/): The official Python documentation.
- [Python Job Board](https://www.python.org/jobs/)

### Community

- [Internet Relay Chat](https://www.python.org/community/irc/)
```

The first dataset row points to the files; the other rows list every included page with its section and where its description came from (`page` or `ai`):

```json
{
    "type": "files",
    "title": "llms.txt (download)",
    "llmsTxtUrl": "https://api.apify.com/v2/key-value-stores/<store>/records/llms.txt",
    "llmsFullTxtUrl": "https://api.apify.com/v2/key-value-stores/<store>/records/llms-full.txt",
    "siteName": "Python",
    "summarySource": "input",
    "pageCount": 11,
    "source": "crawl"
}
```

#### Validate existing llms.txt files

Choose **Validate existing llms.txt files** and list the websites. Lines with several domains separated by commas or spaces are split. A line that is not a website gets its own row with `errorType: "invalid_input"` (not charged) and the other sites are still validated. Every row has `input`, your original line. A site that refuses our request for `/llms.txt` (HTTP 401, 403, 418 or 429, or a bot-check page) is reported with `errorType: "blocked"` and not charged, instead of being scored as missing; the single **Website** field (prefilled for generate mode) is ignored when **Websites to validate** has entries.

##### Input example (validate)

```json
{ "mode": "validate", "websites": ["docs.anthropic.com", "developers.cloudflare.com", "python.org", "vercel.com"], "linkSampleSize": 8 }
```

##### Output example (validate)

Real results (September 2026, shortened):

```json
[
    { "website": "https://developers.cloudflare.com", "found": true, "score": 100, "linkCount": 107, "linksChecked": 8, "brokenLinks": [],
      "issues": [{ "code": "llms-full-txt", "severity": "info", "message": "llms-full.txt found." }] },
    { "website": "https://docs.anthropic.com", "found": true, "score": 90, "linkCount": 517,
      "issues": [
          { "code": "no-summary", "severity": "warning", "message": "No \"> summary\" blockquote after the title. A one-line summary helps AI tools understand the site quickly." },
          { "code": "notes-separator", "severity": "info", "message": "125 link(s) add notes with something other than \": \" ..." }
      ] },
    { "website": "https://python.org", "found": false, "score": 0,
      "issues": [{ "code": "missing", "severity": "error", "message": "/llms.txt answered HTTP 404. Create a Markdown file at /llms.txt with your site name, a one-line summary and links to key pages." }] }
]
```

Scoring: each error costs 30 points, each warning 10, broken links 5 each (at most 30); info items are free. A missing file scores 0. Sites without an llms.txt are sales leads for agencies: generate one for them with this Actor.

The `SUMMARY` record has `status` (`SUCCESS`, `PARTIAL_RESULTS`, `FAILED`, `NO_RESULTS` or `LIMIT_REACHED`), `costPlan`, `notProcessed` and, in generate mode, the download links and `scope`. Failed items carry an `errorType`.

### Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/llms-txt-generator) to Claude, Cursor or any MCP client, then ask e.g. "Create an llms.txt for example.com with the Docs section first" or "Check which of these 20 sites have a valid llms.txt."

```json
{ "website": "example.com", "maxPages": 30 }
```

Failed items are not charged and carry an `errorType`.

### Use it from code and integrations

Run it from your own code with the Apify API. This call waits for the run and returns the results as JSON (replace `YOUR_TOKEN` with your Apify API token):

```bash
curl -X POST "https://api.apify.com/v2/acts/tidytools~llms-txt-generator/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"website":"https://docs.apify.com","maxPages":30}'
```

The synchronous endpoint waits up to 5 minutes. For bigger runs, start the run with `POST https://api.apify.com/v2/acts/tidytools~llms-txt-generator/runs` and read the dataset when it finishes, or use the `apify-client` package for JavaScript or Python.

**Schedules and integrations:** run it daily or weekly with Apify Schedules, get a webhook when a run finishes, or send the results to Zapier, Make, n8n, Google Sheets, Slack and other apps with Apify integrations. Results can be exported as JSON, CSV, Excel or XML.

### FAQ

**Does the Actor collect usage data?** The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

**Will llms.txt improve my rankings?** Not directly. llms.txt is a proposal, not an official standard; AI companies decide whether and how to use it. It is cheap to add and helps AI tools that do read it.

**Can I edit the result?** Yes. Set the site name, summary and section order in the input, or edit the downloaded file before uploading it.

**I regenerate llms.txt on a schedule (n8n, Make, API). Why does the file change every time?** The AI summary is worded differently on each run. Set your own **summary** (and **site name**) in the input: the file then changes only when your pages change. Read the text from `llmsTxt` in the first dataset row and write it to your site or repository.

**My site already has an llms.txt. Does generate mode read it?** No. Generate mode builds a new file from your pages (sitemap or crawl) and never reads or overwrites the one on your site. To check the existing file, use validate mode.

**Does it run JavaScript or read pages behind a login?** Pages are read with plain requests first and a real browser as a fallback, so most JavaScript sites work (for example react.dev). Pages behind a login or bot protection are skipped and not charged.

**A big site gives only 100 pages. Why?** **Max pages** (default 100, up to 2,000) is the limit and also your cost control; the home page and shallow paths come first. Raise it, or use *Include URL patterns* to describe one section.

**Can it check an existing llms.txt instead of generating one?** Yes. Set **What to do** to *Validate existing llms.txt files (many websites)* and list the sites in **Websites to validate**: for each one it fetches `/llms.txt` and `/llms-full.txt`, checks the format, requests a sample of links (**Links to test per site (validate mode)**) and returns a score (0-100) with a list of issues. Each validated website costs $2.00 per 1,000, and unreachable websites, websites that block the request and invalid lines are not charged.

### Tips

- Review the file before publishing it: llms.txt should describe what you want AI tools to know first.
- Use **exclude patterns** to leave out archives, tags or duplicate language versions. A pattern ending in `/**` (for example `**/legal/**`) also leaves out the folder page itself (`/legal`).
- **Advanced settings**: *Plain HTTP requests from* and *Proxy* help with sites that block data-center requests (proxy usage is billed to your Apify account).
- Check which AI crawlers can actually read your site with our **AI Crawler Access Checker**, or get a full AI search readiness score with **GEO Readiness Audit**.

### Related tools (by TidyTools)

| Tool | Use it to |
|---|---|
| **GEO Readiness Audit** | Score how well AI search can reach and read your site (firewall test, content without JavaScript, llms.txt, structured data), with an HTML report and an optional check of whether Google AI Overviews cite you |
| **AI Crawler Access Checker** | See which of 26 AI crawlers your robots.txt allows, read Cloudflare Content Signals, and track changes across thousands of domains |
| **Website Change Monitor** | Get an alert when a page changes, for example your published llms.txt or a competitor's |
| **Sitemap URL Extractor** | List every URL a site publishes in its sitemaps, to see which pages your llms.txt should cover |
| **SEO Audit Crawler** | Audit titles, links, speed and structured data across the whole site |

### Limitations

- llms.txt is a proposal, not an official standard; AI companies decide whether and how to use it.
- Pages that block automated access are skipped.
- Only public pages; no logins.

### Support

Open an issue in the **Issues** tab with the website and your input. Issues are checked regularly.

# Actor input Schema

## `website` (type: `string`):

Main input in generate mode (the default): the website to describe, e.g. https://example.com or example.com. A path (e.g. https://docs.example.com/academy) limits llms.txt to that section (when that page links to nothing under its own path, e.g. docs sites that give every page its own folder, the folder one level up is used; never the whole site); the home page is still read for the site name and summary. In validate mode it is added to the list below.

## `mode` (type: `string`):

Generate builds llms.txt and llms-full.txt from a site's pages. Validate checks the llms.txt that websites already publish (format and a sample of links) and scores it. Values: generate = Generate llms.txt for one website; validate = Validate existing llms.txt files (many websites).

## `websites` (type: `array`):

Validate mode only: websites whose /llms.txt should be checked, one per line (up to 5,000). Lines with several domains separated by commas or spaces are split; lines that are not websites get an error row and are not charged.

## `maxPages` (type: `integer`):

How many pages to include. The home page and top-level pages are picked first.

## `includeFullText` (type: `boolean`):

llms-full.txt contains the full Markdown text of every included page (no extra charge).

## `cleanFullText` (type: `boolean`):

Keep only the main content of each page in llms-full.txt (menus, footers, cookie notices and "Skip to content" links are removed). Turn off if a site loses real content.

## `aiSiteSummary` (type: `boolean`):

Write the summary line of llms.txt with AI from the home page content (no extra charge). Off = use the home page's meta description.

## `aiPageDescriptions` (type: `boolean`):

Pages without a meta description get a one-line AI-written description (charged as an extra event).

## `siteName` (type: `string`):

Use this as the llms.txt title (# line) instead of the name detected from the home page.

## `summary` (type: `string`):

Use this as the summary line (> line). When set, no AI summary is written.

## `sectionOrder` (type: `array`):

Optional section names to put first, in this order, e.g. Docs, Products, Blog. Other sections follow; Optional stays last.

## `sitemapUrl` (type: `string`):

Optional. Read pages from this sitemap (or sitemap index) instead of discovering sitemaps automatically.

## `includeUrlPatterns` (type: `array`):

Optional glob patterns; only matching pages are included, e.g. https://example.com/docs/\*\*. When set, a path in "Website" does not limit the scope.

## `excludeUrlPatterns` (type: `array`):

Optional glob patterns to leave out, e.g. **/tag/**

## `respectRobotsTxt` (type: `boolean`):

Leave out pages that the site's robots.txt disallows for crawlers (the "\*" rules).

## `linkSampleSize` (type: `integer`):

How many links from each llms.txt are requested to find broken ones. 0 = format checks only.

## `maxConcurrency` (type: `integer`):

How many pages are processed at the same time.

## `httpVia` (type: `string`):

Some sites block requests from data centers. Auto retries from a second network before falling back to a real browser. If both are refused, Auto also tries our second server (Oracle, different IP) before a real browser.

## `pageTimeoutSecs` (type: `integer`):

Generate mode: a page that takes longer to read (its title, description and text, all fallbacks included) is left out, listed in SUMMARY.timedOutPages and not charged. 0 = no limit.

## `siteTimeoutSecs` (type: `integer`):

Validate mode: a website whose llms.txt, llms-full.txt and link sample take longer than this is reported with errorType "timeout" and not charged. 0 = no limit.

## `proxyConfiguration` (type: `object`):

Only used for requests sent from Apify's network. Apify proxy usage is billed to your Apify account.

## Actor input object example

```json
{
  "website": "https://docs.apify.com",
  "mode": "generate",
  "maxPages": 30,
  "includeFullText": true,
  "cleanFullText": true,
  "aiSiteSummary": true,
  "aiPageDescriptions": false,
  "respectRobotsTxt": false,
  "linkSampleSize": 10,
  "maxConcurrency": 8,
  "httpVia": "auto",
  "pageTimeoutSecs": 120,
  "siteTimeoutSecs": 120
}
```

# Actor output Schema

## `llms` (type: `string`):

No description

## `llmsFull` (type: `string`):

No description

## `pages` (type: `string`):

No description

## `validation` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "website": "https://docs.apify.com",
    "maxPages": 30
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidytools/llms-txt-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "website": "https://docs.apify.com",
    "maxPages": 30,
}

# Run the Actor and wait for it to finish
run = client.actor("tidytools/llms-txt-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "website": "https://docs.apify.com",
  "maxPages": 30
}' |
apify call tidytools/llms-txt-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidytools/llms-txt-generator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/egcg4cyQFYEB7CfPM/builds/Tb6lt4MbnKdepEJX5/openapi.json