# 🤖 llms.txt Generator & Auditor - Make Sites AI-Readable (`that_red_bird/llms-txt-generator`) Actor

⚡ Generate a spec-shaped llms.txt for any site in one run, and audit an existing one. ✅ Sitemap discovery, clean markdown extraction, optional llms-full.txt. ✅ Coverage %, missing pages, stale URLs and spec issues. Reads only the site itself — nothing to block it.

- **URL**: https://apify.com/that\_red\_bird/llms-txt-generator.md
- **Developed by:** [mohamed alaya](https://apify.com/that_red_bird) (community)
- **Categories:** AI, SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## llms.txt Generator & Auditor

Make any site readable by AI assistants — generate a spec-shaped **`llms.txt`** in one run, and
audit an existing one for coverage and correctness.

`llms.txt` is the emerging convention that tells AI crawlers what a site is and which pages matter,
the way `robots.txt` tells search crawlers what to skip. Anthropic, Cloudflare and Perplexity already
publish one. Most sites do not, and hand-writing it is tedious and goes stale the moment you publish.

### What it does

**Generate** — discovers your pages (sitemap first, including sitemap indexes; falls back to
following same-origin links), pulls each page's title and description, groups them into sections by
site structure, and writes a valid `llms.txt` to the key-value store.

**Audit** — fetches your existing `llms.txt` and reports:

- **coverage %** — how many published pages are actually listed
- **missing URLs** — pages you publish but never told the AI about
- **stale URLs** — entries pointing at pages that no longer exist
- **spec issues** — missing `# title` line, missing `> summary`, no links at all

**`llms-full.txt`** (optional) — the long-form companion with each page's full content as clean
markdown: the file an assistant ingests directly. Headings, code blocks with language tags, tables,
lists and links are preserved; nav, footers, cookie banners and ad slots are stripped.

### Input

```json
{
  "siteUrl": "https://docs.example.com",
  "mode": "both",
  "maxPages": 200,
  "includePatterns": ["/docs/", "/guides/"],
  "excludePatterns": ["/tag/", "/author/"],
  "generateFullText": true
}
```

### Output

A `result` row (site name, discovery method, page counts, the generated file, the full audit), one
`page` row per page (url, title, description, language, word count, markdown), and an `error` row for
anything that failed. The files themselves are saved to the key-value store as **`llms.txt`** and
**`llms-full.txt`**, ready to upload to your web root.

### Why this is reliable

It reads **your own site**. There is no anti-bot, no login, no rate-limit war and no third party that
can cut you off — which is not true of most scrapers. If every page fails to load the run fails
loudly rather than handing you an empty file that looks like a result.

### Typical uses

Publishing `llms.txt` for a docs site or product site · keeping it current as a scheduled job so it
never drifts from what you actually publish · auditing client sites as an agency deliverable ·
producing clean markdown corpora for your own RAG pipeline.

# Actor input Schema

## `siteUrl` (type: `string`):

The site to analyse, e.g. https://docs.example.com. Pages are discovered from its sitemap, or by following same-origin links if there is no sitemap.

## `mode` (type: `string`):

generate = build a new llms.txt · audit = check the existing one for coverage and spec issues · both = do both.

## `maxPages` (type: `integer`):

Upper bound on pages to include. Documentation sites are usually well covered by 100-300. Kept small by default so a first run finishes quickly; raise it for a fuller crawl.

## `useSitemap` (type: `boolean`):

Prefer the site's sitemap (including sitemap indexes) for discovery. Falls back to following links automatically if none is found.

## `includePatterns` (type: `array`):

Keep only URLs containing one of these substrings, e.g. /docs/, /guides/.

## `excludePatterns` (type: `array`):

Drop URLs containing any of these, e.g. /tag/, /author/, ?page=.

## `siteName` (type: `string`):

The H1 line of the generated file. Derived from the first page title when empty.

## `summary` (type: `string`):

The blockquote summary line. Derived from the site's meta description when empty.

## `generateFullText` (type: `boolean`):

Produce the long-form companion file containing each page's full markdown, which is what an LLM ingests directly.

## `maxFullTextChars` (type: `integer`):

Hard cap so a large site cannot produce an unusable multi-megabyte file.

## `concurrency` (type: `integer`):

How many pages to fetch in parallel.

## `proxyConfiguration` (type: `object`):

Optional Apify proxy configuration.

## Actor input object example

```json
{
  "siteUrl": "https://docs.apify.com",
  "mode": "both",
  "maxPages": 20,
  "useSitemap": true,
  "generateFullText": false,
  "maxFullTextChars": 500000,
  "concurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `downloadCsv` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `count` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrl": "https://docs.apify.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("that_red_bird/llms-txt-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "siteUrl": "https://docs.apify.com" }

# Run the Actor and wait for it to finish
run = client.actor("that_red_bird/llms-txt-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrl": "https://docs.apify.com"
}' |
apify call that_red_bird/llms-txt-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,that_red_bird/llms-txt-generator"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/C2iKcpGZuhFQk4lg9/builds/ekxCC6CfNJr2ez8RF/openapi.json