llms.txt Generator

Generate an llms.txt file from any website sitemap. This Apify Actor crawls all URLs from a sitemap, extracts page titles and meta descriptions, and creates a Markdown-formatted file that helps LLMs understand your website's content.

What is llms.txt?

The llms.txt file is a standardized way to provide LLMs (Large Language Models) with information about your website. It follows a simple Markdown format:

# Website Name

Brief description of the website.

## Pages

- [Page Title](https://example.com/page): Page description
- [Another Page](https://example.com/another): Another description

Features

• Crawls XML sitemaps (including sitemap index files with nested sitemaps)
• Extracts page titles from <title> tags
• Extracts descriptions from <meta name="description"> or <meta property="og:description">
• Supports glob patterns for URL filtering (include/exclude)
• Respects robots.txt directives
• Configurable concurrency and request limits
• Progress tracking with status updates

Input

Field	Type	Required	Default	Description
sitemapUrl	string	Yes	-	URL of the XML sitemap to crawl
maxConcurrency	integer	No	5	Maximum concurrent requests (1-50)
maxRequestsPerCrawl	integer	No	1000	Maximum pages to crawl (0 = unlimited)
respectRobotsTxt	boolean	No	true	Honor robots.txt restrictions
includeUrlPatterns	array	No	["**"]	Glob patterns for URLs to include
excludeUrlPatterns	array	No	[]	Glob patterns for URLs to exclude

Example Input

{
  "sitemapUrl": "https://example.com/sitemap.xml",
  "maxConcurrency": 5,
  "maxRequestsPerCrawl": 500,
  "includeUrlPatterns": ["**/blog/**", "**/docs/**"],
  "excludeUrlPatterns": ["**/tag/**", "**/author/**"],
  "respectRobotsTxt": true
}

URL Pattern Examples

• ** - Match all URLs
• **/blog/** - Match URLs containing /blog/
• **/docs/* - Match direct children of /docs/
• **/*.html - Match URLs ending with .html

Output

The Actor produces two outputs:

1. llms.txt File (Key-Value Store)

The generated llms.txt file is stored in the default Key-Value Store and can be downloaded directly:

https://api.apify.com/v2/key-value-stores/{storeId}/records/llms.txt

2. Crawl Results (Dataset)

The Dataset contains a single item with crawl statistics:

{
  "llmsTxtUrl": "https://api.apify.com/v2/key-value-stores/{storeId}/records/llms.txt",
  "statistics": {
    "totalDiscovered": 150,
    "totalAfterFiltering": 120,
    "successCount": 118,
    "errorCount": 2,
    "robotsSkippedCount": 0,
    "limitSkippedCount": 0,
    "startedAt": "2024-01-15T10:00:00.000Z",
    "finishedAt": "2024-01-15T10:01:30.000Z",
    "durationMs": 90000
  },
  "errors": [
    { "url": "https://example.com/broken", "message": "404 Not Found" }
  ]
}

Local Development

Prerequisites

• Node.js 18+
• npm

Setup

# Install dependencies
npm install

# Create input file
mkdir -p storage/key_value_stores/default
cat > storage/key_value_stores/default/INPUT.json << 'EOF'
{
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "maxConcurrency": 5,
  "maxRequestsPerCrawl": 50
}
EOF

# Run locally
npm run start:dev

Available Scripts

Command	Description
npm run start:dev	Run Actor locally with tsx
npm run build	Compile TypeScript
npm run lint	Run ESLint
npm run lint:fix	Fix ESLint issues
npm test	Run tests

Project Structure

.actor/
  actor.json              # Actor configuration
  input_schema.json       # Input validation schema
  output_schema.json      # Output schema definition
  dataset_schema.json     # Dataset structure
  key_value_store_schema.json  # KV store structure
src/
  main.ts                 # Entry point and orchestration
  types.ts                # TypeScript interfaces
  services/
    crawler.ts            # CheerioCrawler configuration
    sitemap.ts            # Sitemap loading utilities
    url-filter.ts         # Glob-based URL filtering
    llms-txt.ts           # llms.txt generation
  utils/
    constants.ts          # Default values and config
storage/                  # Local storage (dev only)

Deploy to Apify

Using Git

1. Push your code to a Git repository
2. Go to Apify Console
3. Click "Link Git Repository"
4. Select your repository

Using CLI

# Login to Apify
apify login

# Deploy
apify push

Technologies

• Apify SDK - Actor lifecycle and storage
• Crawlee - Web scraping framework
• CheerioCrawler - Fast HTML crawler
• picomatch - Glob pattern matching

Resources

• llms.txt Specification
• Apify Actors Documentation
• Crawlee Documentation

License

Apache-2.0