LLMs.txt & llms-full.txt Generator for Any Website

What does LLMs.txt Generator do?

LLMs.txt Generator creates a spec-compliant llms.txt and llms-full.txt for any website in one click. This is the emerging standard that tells ChatGPT, Claude, Perplexity, Gemini and other AI search engines what your site is about and which pages matter. Paste your homepage URL and the Actor automatically finds your sitemap (or crawls internal links), extracts every page's title, meta description and full Markdown content, and outputs two ready-to-upload files.

Run it on the Apify platform to get API access, scheduling (regenerate your llms.txt weekly so it never goes stale), integrations (Make, Zapier, webhooks) and run monitoring out of the box.

Why generate an llms.txt file?

• AI search visibility (GEO/AEO): a growing share of searches happens inside AI assistants. llms.txt is how you tell them what your business does, in a format they can actually ingest.
• Control your AI narrative: curate which pages AI models read first: your pricing, your docs, your best content, instead of random crawled pages.
• llms-full.txt for deep context: the full-content version gives AI tools your complete site content as clean Markdown, perfect for RAG pipelines and AI agents.
• SEO agencies & freelancers: generate llms.txt files for every client site in bulk via the API and charge for AI-readiness as a service.
• Zero setup: no API keys, no code, no config. One URL in, two files out.

How to use LLMs.txt Generator

1. Click Try for free and open the Actor in Apify Console.
2. Paste your website URL (e.g. https://yoursite.com).
3. Optionally set Max pages or include/exclude URL patterns (e.g. only /blog/).
4. Click Start. The run takes seconds to a few minutes depending on site size.
5. Download llms.txt and llms-full.txt from the run's Storage → Key-value store tab (direct links appear in the Output tab).
6. Upload both files to your website root: https://yoursite.com/llms.txt.

💡 Tip: schedule the Actor to run weekly so your llms.txt stays in sync with your content.

Input

Only one field is required. Everything else has sensible defaults. See the Input tab for all options.

{
    "websiteUrl": "https://yoursite.com",
    "maxPages": 50,
    "includeFullText": true
}

Output

The generated files land in the key-value store; one dataset row is pushed per processed page. You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Example llms.txt output:

# Acme Analytics

> Acme Analytics is a privacy-first web analytics platform for small teams.

## Main Pages

- [Acme Analytics](https://acme.io): Privacy-first analytics for small teams.
- [Pricing](https://acme.io/pricing): Simple plans starting free.

## Docs

- [Getting Started](https://acme.io/docs/getting-started): Install the snippet in five minutes.

Example dataset item:

{
    "url": "https://acme.io/docs/getting-started",
    "title": "Getting Started",
    "description": "Install the snippet in five minutes.",
    "section": "Docs",
    "wordCount": 843,
    "status": "OK"
}

Data fields

Field	Description
url	Page URL included in the generated files
title	Page title used in llms.txt
description	Meta description (or first-paragraph fallback)
section	llms.txt section, derived from the URL path
wordCount	Words extracted for llms-full.txt
status	OK or ERROR

How much does it cost to generate an llms.txt file?

You pay a small fee per page processed. A typical 50-page site costs just a few cents. Free plan users can process up to 10 pages per run, enough to generate a valid llms.txt for a small site. Subscribe to an Apify plan to unlock up to 500 pages per run, scheduling and full API access.

Tips and advanced options

• Sitemap-first: sites with a sitemap.xml get the fastest, most complete results. No sitemap? The Actor falls back to crawling internal links automatically.
• Use Include URL patterns (/docs/, /blog/) to build a focused llms.txt for one content area.
• Use Exclude URL patterns (/tag/, /cart) to keep noise out.
• Lower Concurrency if the target site rate-limits requests.
• If a site blocks direct traffic, enable Apify Proxy in the input (datacenter or residential) or plug in your own proxies.

FAQ

What is llms.txt?

llms.txt is a proposed web standard (llmstxt.org): a Markdown file at your site root that gives large language models a curated index of your site: what it is, and which pages matter, with one-line descriptions.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a short curated index of links with descriptions. llms-full.txt contains the full Markdown content of every page, so AI tools can ingest your whole site in one file.

Does llms.txt help with ChatGPT and Perplexity visibility?

It gives AI crawlers a clean, curated view of your site instead of letting them guess from raw HTML. Combined with solid content and schema markup, it is a core part of Generative Engine Optimization (GEO).

Can I automate this from an AI agent?

Yes. Call this Actor from an AI agent via the Apify MCP server, or through the Apify API from any workflow tool (Make, Zapier, n8n).

Is crawling my own website legal?

Yes. This Actor only reads publicly accessible pages of the site you point it at, and is typically used by site owners and their agencies on their own properties.

Support and custom solutions

Found a bug or need a feature? Open an issue in the Issues tab.

💼 Need a custom scraper or automation? Reach out at youssefarhan24@gmail.com 🌐 Got a site in mind? Suggest it here ⚡ Unlock full limits with an Apify plan