# LLMs.txt & llms-full.txt Generator for Any Website (`fayoussef/llms-txt-generator`) Actor Generate a spec-compliant llms.txt and llms-full.txt for any website in one click. Crawls your sitemap or site, extracts titles, descriptions and full Markdown content, and outputs ready-to-upload files that make your site visible to ChatGPT, Claude, Perplexity and other AI search engines. - **URL**: https://apify.com/fayoussef/llms-txt-generator.md - **Developed by:** [youssef farhan](https://apify.com/fayoussef) (community) - **Categories:** AI, Automation, SEO tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $4.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### What does LLMs.txt Generator do? **LLMs.txt Generator** creates a **spec-compliant `llms.txt` and `llms-full.txt`** for any website in one click. This is the emerging standard that tells **ChatGPT, Claude, Perplexity, Gemini and other AI search engines** what your site is about and which pages matter. Paste your homepage URL and the Actor automatically finds your **sitemap** (or crawls internal links), extracts every page's **title, meta description and full Markdown content**, and outputs two ready-to-upload files. Run it on the Apify platform to get **API access, scheduling** (regenerate your llms.txt weekly so it never goes stale), **integrations** (Make, Zapier, webhooks) and run monitoring out of the box. ### Why generate an llms.txt file? - **AI search visibility (GEO/AEO)**: a growing share of searches happens inside AI assistants. `llms.txt` is how you tell them what your business does, in a format they can actually ingest. - **Control your AI narrative**: curate which pages AI models read first: your pricing, your docs, your best content, instead of random crawled pages. - **llms-full.txt for deep context**: the full-content version gives AI tools your complete site content as clean Markdown, perfect for RAG pipelines and AI agents. - **SEO agencies & freelancers**: generate llms.txt files for every client site in bulk via the API and charge for AI-readiness as a service. - **Zero setup**: no API keys, no code, no config. One URL in, two files out. ### How to use LLMs.txt Generator 1. Click **Try for free** and open the Actor in Apify Console. 2. Paste your **website URL** (e.g. `https://yoursite.com`). 3. Optionally set **Max pages** or include/exclude URL patterns (e.g. only `/blog/`). 4. Click **Start**. The run takes seconds to a few minutes depending on site size. 5. Download **llms.txt** and **llms-full.txt** from the run's **Storage ā†’ Key-value store** tab (direct links appear in the Output tab). 6. Upload both files to your website root: `https://yoursite.com/llms.txt`. šŸ’” **Tip:** schedule the Actor to run weekly so your llms.txt stays in sync with your content. ### Input Only one field is required. Everything else has sensible defaults. See the Input tab for all options. ```json { "websiteUrl": "https://yoursite.com", "maxPages": 50, "includeFullText": true } ```` ### Output The generated files land in the key-value store; one dataset row is pushed per processed page. You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. Example `llms.txt` output: ``` ## Acme Analytics > Acme Analytics is a privacy-first web analytics platform for small teams. ### Main Pages - [Acme Analytics](https://acme.io): Privacy-first analytics for small teams. - [Pricing](https://acme.io/pricing): Simple plans starting free. ### Docs - [Getting Started](https://acme.io/docs/getting-started): Install the snippet in five minutes. ``` Example dataset item: ```json { "url": "https://acme.io/docs/getting-started", "title": "Getting Started", "description": "Install the snippet in five minutes.", "section": "Docs", "wordCount": 843, "status": "OK" } ``` ### Data fields | Field | Description | |-------|-------------| | `url` | Page URL included in the generated files | | `title` | Page title used in llms.txt | | `description` | Meta description (or first-paragraph fallback) | | `section` | llms.txt section, derived from the URL path | | `wordCount` | Words extracted for llms-full.txt | | `status` | `OK` or `ERROR` | ### How much does it cost to generate an llms.txt file? You pay a small fee **per page processed**. A typical 50-page site costs just a few cents. **Free plan users can process up to 10 pages per run**, enough to generate a valid llms.txt for a small site. Subscribe to an [Apify plan](https://apify.com/pricing?fpr=youssef) to unlock up to 500 pages per run, scheduling and full API access. ### Tips and advanced options - **Sitemap-first**: sites with a sitemap.xml get the fastest, most complete results. No sitemap? The Actor falls back to crawling internal links automatically. - Use **Include URL patterns** (`/docs/`, `/blog/`) to build a focused llms.txt for one content area. - Use **Exclude URL patterns** (`/tag/`, `/cart`) to keep noise out. - Lower **Concurrency** if the target site rate-limits requests. - If a site blocks direct traffic, enable **Apify Proxy** in the input (datacenter or residential) or plug in your own proxies. ### FAQ #### What is llms.txt? `llms.txt` is a proposed web standard (llmstxt.org): a Markdown file at your site root that gives large language models a curated index of your site: what it is, and which pages matter, with one-line descriptions. #### What is the difference between llms.txt and llms-full.txt? `llms.txt` is a short curated index of links with descriptions. `llms-full.txt` contains the **full Markdown content of every page**, so AI tools can ingest your whole site in one file. #### Does llms.txt help with ChatGPT and Perplexity visibility? It gives AI crawlers a clean, curated view of your site instead of letting them guess from raw HTML. Combined with solid content and schema markup, it is a core part of Generative Engine Optimization (GEO). #### Can I automate this from an AI agent? Yes. Call this Actor from an AI agent via the **Apify MCP server**, or through the Apify API from any workflow tool (Make, Zapier, n8n). #### Is crawling my own website legal? Yes. This Actor only reads publicly accessible pages of the site you point it at, and is typically used by site owners and their agencies on their own properties. ### Support and custom solutions Found a bug or need a feature? Open an issue in the **Issues** tab. šŸ’¼ Need a custom scraper or automation? Reach out at **youssefarhan24@gmail.com** šŸŒ Got a site in mind? [Suggest it here](https://automationbyexperts.com/apify) āš” Unlock full limits with an [Apify plan](https://apify.com/pricing?fpr=youssef) # Actor input Schema ## `websiteUrl` (type: `string`): The website to generate llms.txt and llms-full.txt for. Just paste your homepage URL ā€” the Actor finds your sitemap (or crawls internal links) automatically. āš” Unlock full pages and higher limits by subscribing to an [Apify plan](https://apify.com/pricing?fpr=youssef). šŸ’¼ Need a custom solution? Reach out at youssefarhan24@gmail.com šŸŒ Got a site in mind? [Suggest it here](https://automationbyexperts.com/apify) ## `maxPages` (type: `integer`): Maximum number of pages to include in the generated files. Free plan is capped at 10 pages; paid plans can go up to 500. ## `includeFullText` (type: `boolean`): Also generate llms-full.txt with the complete Markdown content of every page (not just the link index). Recommended ā€” this is the file AI tools ingest for deep context. ## `includePatterns` (type: `array`): Only include URLs containing at least one of these substrings (e.g. `/blog/`, `/docs/`). Leave empty to include all pages. ## `excludePatterns` (type: `array`): Skip URLs containing any of these substrings (e.g. `/tag/`, `?page=`, `/cart`). ## `concurrency` (type: `integer`): How many pages to fetch in parallel. Lower this if the target site rate-limits you. ## `proxyConfiguration` (type: `object`): Use Apify Proxy (datacenter or residential) or your own proxies for sites that block direct traffic. Leave disabled for a direct connection. ## Actor input object example ```json { "websiteUrl": "https://books.toscrape.com", "maxPages": 50, "includeFullText": true, "concurrency": 10, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `llmsTxt` (type: `string`): The generated llms.txt file ā€” download and upload it to your website root. ## `llmsFullTxt` (type: `string`): The generated llms-full.txt file with full Markdown content of every page. ## `pages` (type: `string`): One row per crawled page with title, description, section and word count. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "websiteUrl": "https://books.toscrape.com", "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("fayoussef/llms-txt-generator").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`šŸ’¾ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // šŸ“š Want to learn more šŸ“–? Go to ā†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "websiteUrl": "https://books.toscrape.com", "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("fayoussef/llms-txt-generator").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("šŸ’¾ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # šŸ“š Want to learn more šŸ“–? Go to ā†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "websiteUrl": "https://books.toscrape.com", "proxyConfiguration": { "useApifyProxy": false } }' | apify call fayoussef/llms-txt-generator --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=fayoussef/llms-txt-generator", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "LLMs.txt & llms-full.txt Generator for Any Website", "description": "Generate a spec-compliant llms.txt and llms-full.txt for any website in one click. Crawls your sitemap or site, extracts titles, descriptions and full Markdown content, and outputs ready-to-upload files that make your site visible to ChatGPT, Claude, Perplexity and other AI search engines.", "version": "1.0", "x-build-id": "zVzXcmeX6Lrni4foR" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/fayoussef~llms-txt-generator/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-fayoussef-llms-txt-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/fayoussef~llms-txt-generator/runs": { "post": { "operationId": "runs-sync-fayoussef-llms-txt-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/fayoussef~llms-txt-generator/run-sync": { "post": { "operationId": "run-sync-fayoussef-llms-txt-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "websiteUrl" ], "properties": { "websiteUrl": { "title": "Website URL", "type": "string", "description": "The website to generate llms.txt and llms-full.txt for. Just paste your homepage URL ā€” the Actor finds your sitemap (or crawls internal links) automatically.\n\nāš” Unlock full pages and higher limits by subscribing to an [Apify plan](https://apify.com/pricing?fpr=youssef).\nšŸ’¼ Need a custom solution? Reach out at youssefarhan24@gmail.com\nšŸŒ Got a site in mind? [Suggest it here](https://automationbyexperts.com/apify)" }, "maxPages": { "title": "Max pages", "minimum": 1, "maximum": 500, "type": "integer", "description": "Maximum number of pages to include in the generated files. Free plan is capped at 10 pages; paid plans can go up to 500.", "default": 50 }, "includeFullText": { "title": "Generate llms-full.txt (full page content)", "type": "boolean", "description": "Also generate llms-full.txt with the complete Markdown content of every page (not just the link index). Recommended ā€” this is the file AI tools ingest for deep context.", "default": true }, "includePatterns": { "title": "Include URL patterns", "type": "array", "description": "Only include URLs containing at least one of these substrings (e.g. `/blog/`, `/docs/`). Leave empty to include all pages.", "items": { "type": "string" } }, "excludePatterns": { "title": "Exclude URL patterns", "type": "array", "description": "Skip URLs containing any of these substrings (e.g. `/tag/`, `?page=`, `/cart`).", "items": { "type": "string" } }, "concurrency": { "title": "Concurrency", "minimum": 1, "maximum": 30, "type": "integer", "description": "How many pages to fetch in parallel. Lower this if the target site rate-limits you.", "default": 10 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Use Apify Proxy (datacenter or residential) or your own proxies for sites that block direct traffic. Leave disabled for a direct connection.", "default": { "useApifyProxy": false } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```