llms.txt Generator — Make Any Website LLM-Ready

Generate an llms.txt for any website in one click. llms.txt is the emerging standard (llmstxt.org) — think "robots.txt for LLMs" — a clean Markdown map that helps ChatGPT, Claude, and AI agents understand and navigate your site. This actor crawls a site and builds that file for you, plus an optional llms-full.txt containing every page's full clean content.

No more hand-writing llms.txt. Point it at a homepage and download the result.

What it does

• Crawls your site (same-domain, depth- and page-limited) and extracts each page's title and description.
• Builds llms.txt — an H1 site name, a summary blockquote, and a linked, described page index in the standard format.
• Optional llms-full.txt — concatenates every page's full content (clean Markdown) for full-context ingestion.
• Downloadable output — the files are saved to the run's key-value store; the page inventory goes to the dataset.

Use cases

• Make your site AI-friendly — publish llms.txt so LLMs cite and use your content accurately.
• RAG ingestion — llms-full.txt is a single clean corpus of your whole site.
• SEO/AEO — get ahead of AI-search (answer-engine optimization) with a machine-readable site map.
• Agencies — generate llms.txt for every client site in seconds.

Input

Field	Description
startUrls	The site to process (usually the homepage).
maxPages	Cap on pages crawled and listed (main cost lever).
maxDepth	How many link-hops to follow.
onlySameDomain	Keep the crawl on the start URL's domain.
includeFullText	Also build llms-full.txt with full page content.

Output

• Key-value store: llms.txt (always) and llms-full.txt (if enabled) — ready to download and drop at your site root.
• Dataset: one record per crawled page (url, title, description) — the inventory used to build the file.

Example llms.txt:

# Apify Documentation

> Check the performance of your Actors, validate data quality, and receive alerts.

## Pages

- [Apify Documentation](https://docs.apify.com): Learn how to build, run, and scale Actors.
- [Academy](https://docs.apify.com/academy): Free web scraping and automation courses.

Pricing

Pay-per-result: charged per page crawled — no monthly fee, no charge if the site can't be crawled.

Notes

• Works on server-rendered HTML (the content crawlers see first). JavaScript-only content is on the roadmap.
• Crawls public pages only — it does not log in or bypass access controls.

FAQ

What is llms.txt? A proposed standard file that gives LLMs a curated, Markdown overview of your site — see llmstxt.org.

Difference between the two files? llms.txt is a concise link index; llms-full.txt includes the full text of every page.

Where do I put the file? At your site root: https://yoursite.com/llms.txt.