# LLMs.txt Generator (`onescales/the-llms-txt-generator`) Actor

The most powerful LLMs.txt Generator tool online. Generates LLMs.txt , llms-full.txt and markdown .md files within seconds! Get your website discovered, and recommended by ChatGPT, Claude, Google Gemini, Perplexity, Grok, and every AI. (Great for AEO, AIO, GEO, AI SEO) Made by Hi LLMs

- **URL**: https://apify.com/onescales/the-llms-txt-generator.md
- **Developed by:** [One Scales](https://apify.com/onescales) (community)
- **Categories:** AI, Agents, SEO tools
- **Stats:** 129 total users, 2 monthly users, 87.5% runs succeeded, 12 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $4.00 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## The LLMS.TXT Generator | Hi LLMS

The easiest and most powerful tool online to generate llms.txt , llms-full.txt and markdown .md files within seconds! Get your website discovered, understood, and recommended by ChatGPT, Claude, Google Gemini, Perplexity, Grok, and every AI. Transform your site into an AI-friendly powerhouse today!

[![Watch the video](https://img.youtube.com/vi/63a3XTI8uNY/maxresdefault.jpg)](https://www.youtube.com/watch?v=63a3XTI8uNY)

### What is llms.txt?

llms.txt is a standard file format that helps AI models understand and navigate your website's content. It provides a structured overview of your site's key pages with descriptions, making it easier for LLMs to reference your content accurately.

Most other llms.txt generator miss the part about markdown. They summarize your pages in a short sentence in llms.txt but they DON'T provide AI's the full content of each and every page.

**We got your covered. We do it all!**
**How it Works:**

1. Fill out the domain of your website (i.e. onescales.com). We will find your sitemap and a list of all your pages on your site.
2. Specify the url of where you will upload your markdown and llms.txt file.
3. Click on start
4. Once run is finished, check out the output tab, Scroll to the bottom and click on the link on row llms.zip
5. Unzip the zip file and upload it to your directory of where you sepcified your markdown files.
6. Enjoy better visibility from Ai's

### Features

- **Flexible URL Input**: Accept domains, sitemaps, or specific page lists
- **Automatic Sitemap Detection**: Finds sitemap.xml or sitemap_index.xml automatically
- **Smart Categorization**: Automatically organizes URLs based on your site type (SaaS, E-commerce, Service, Media/Blog, Other)
- **Content Extraction**: Uses Mozilla Readability to extract clean content from each page
- **Markdown Conversion**: Converts page content to markdown format
- **Resume/Retry Capability**: Continue from failed runs and merge results
- **Multiple Outputs**: Generates llms.txt, llms-full.txt, individual markdown files, and a complete zip package
- **Marketing Efforts**: Helps with your SEO, GEO, AIO, AEO optimization efforts

### How It Works

#### URL Input Options

The actor accepts three types of input in the **URL(s)** field:

##### 1. Domain Name (Sitemap Auto-Discovery)

```
apple.com
example.org
```

##### 2. Full Sitemap URL

```
https://apple.com/sitemap.xml
https://example.com/sitemap_index.xml
```

##### 3. Specific Page URLs (List)

```
https://apple.com/about
https://apple.com/pricing
https://apple.com/contact
```

### Resume/Retry Feature

If for any reason any of the pages fail to process, just use the retry tab and specify the Run ID of your past failed run with the error urls and it will re-rerun.

### Input Requirements

#### Required Fields:

- **URL(s): Domain, Sitemap URL or List of Pages**: See URL Input Options above
- **Website/Business Name**: Your site name (max 500 characters)
- **Website/Business Description**: Brief description of your site (max 800 characters)
- **Website/Business Category**: Choose from SaaS, E-commerce, Service, Media/Blog, or Other

#### Optional Fields:

**Resume/Retry Section:**

- **Add to Previous llms.txt (if prior run failed)**: Enter Run ID to resume/merge

**Advanced Settings:**

- **Maximum Number of Pages**: Limit the number of pages to process
- **Pause between pages**: Add delay between requests (1-100 seconds)
- **URL Format in llms.txt**: Original URLs or Markdown URLs
- **URL Path for Markdown Files**: Custom path for .md files
- **Proxy Configuration**: Configure proxy settings (defaults to Apify proxy)

### Output

#### Dataset Table Columns:

| Column | Description |
|--------|-------------|
| **URL** | Page URL or file name |
| **Status** | `Complete` or `Error` |
| **Run ID** | The run ID that created this file (use for retry) |
| **Failed URLs** | List of failed URLs (shown in llms.txt row only) |
| **Comments** | Error details if status is Error |
| **Content** | Preview of the content |
| **File Size (bytes)** | Size of the file |
| **URL to Download** | Direct download link |

#### Generated Files:

1. **llms.txt**: Main index file with categorized URLs
2. **llms-full.txt**: Complete content archive
3. **Individual .md files**: Clean markdown for each page
4. **llms.zip**: Everything packaged together
5. **instructions.txt**: Installation guide (in zip)

### Use Cases

- Make your website AI-friendly (SEO, GEO, AIO, AEO)
- Improve how LLMs reference your content
- Create structured documentation for AI models
- Generate clean markdown archives of your site
- Maintain up-to-date llms.txt files with retry capability

### FAQs

Q. Can i Set an Automatic Schedule (for example, once a week)
A. Yes, just click on top right hand side on 3 dots and "schedule actor" and select your timing.

Q. How to Integrate with my Website (shopify, wordpress, drupal, magento, bigcommerce, wix, weebly, squarespace, etc)
A. We are in process of creating tutorials and apps. Will update you once ready.

Q. How to Integrate with automation (zapier, n8n, make.com, ifttt)
A. We are in process of creating tutorials and apps. Will update you once ready.

Q. How to report a bug, suggest or feature request?
A. Fill out the form below. We are here to help:
https://docs.google.com/forms/d/e/1FAIpQLSfsKyzZ3nRED7mML47I4LAfNh_mBwkuFMp1FgYYJ4AkDRgaRw/viewform?usp=dialog

### Roadmap

We plan to improve our app with the following in the future:

- Excluding URLs
- Additional Website Categories
- How to Integrate with Automations (Zapier, n8n, make.com, shopify, wordpress)
- Better Error Handling
- Token Size for Files
- Hash for Files
- Reference for Meta Tags (rel alternate type text/markdown)
- Possible AI support for better content generation
- Improved speed of crawling
- Markdown file folder structure support
- Support for js heavy sites (with playwright)

See updates at https://hillms.com/

### Release Notes

- V1: First version: November 18th, 2025
- V2: Faster Processing, 2nd Processing for Smaller Pages : November 19th, 2025

### Support

Questions or need additional features? Fill out the form at:
https://docs.google.com/forms/d/e/1FAIpQLSfsKyzZ3nRED7mML47I4LAfNh_mBwkuFMp1FgYYJ4AkDRgaRw/viewform?usp=dialog

### Related Keywords

actor, AI, API, apify, at scale, auditor, automated, automation, batch, bulk, checker, converter, crawler, cron, CSV, dataset, detector, downloader, Excel, export, exporter, extractor, fetcher, finder, free tool, generator, Google Sheets, HTML, ifttt, instant, JSON, lookup, make, make.com, maker, mass, MCP, monitor, n8n, no-code, no API key required, parser, PDF, pipeline, report, scanner, schedule, scheduled, scraper, spreadsheet, tool, validator, verifier, webhook, workflow, XML, zapier

# Actor input Schema

## `url` (type: `string`):

Enter one of the following:

• A domain without https (e.g., apple.com or example.org) - app will find your sitemap
• A sitemap URL (e.g., https://apple.com/sitemap.xml) - app will parse it
• A list of specific page URLs (one per line) - app will process only those pages

Examples:
apple.com
https://apple.com/sitemap.xml
https://apple.com/about
https://apple.com/pricing

## `websitename` (type: `string`):

The name of your website or business (max 500 characters). This goes on the top of the llms.txt file.

## `websitedescription` (type: `string`):

A short summary of your website or business containing key information (max 800 characters).  This goes on the top of the llms.txt file.

## `websitecategory` (type: `string`):

Select the category that best describes your website (required). This setting will help organize the llms.txt file categories.

## `maxItems` (type: `integer`):

Maximum number of pages to process (leave empty for unlimited)

## `pauseBetweenPages` (type: `integer`):

Add a delay between processing each page to avoid getting blocked or reduce server load (optional, 1-100 seconds)

## `urlFormat` (type: `string`):

This setting will change the urls referenced inside the llms.txt. We strongly recommend using the Markdown URLs option and specifying the url path of where you are storing your .md markdown files from this app output.

## `markdownUrlPath` (type: `string`):

The URL path where you'll upload the generated .md files on your server (e.g., https://example.com/llms-content/ or https://example.com/docs/). The app will append the filename to this path. Only used when 'Markdown URLs' is selected.

## `proxyConfiguration` (type: `object`):

Proxy settings for the scraper (recommended to leave default)

## `previousRunId` (type: `string`):

Use this section ONLY IF your previous run failed and you want to retry the failed pages and merge with previous results.

ALL YOU HAVE TO DO:

1. Take the Run ID from your previous run that failed (found in the browser URL or in the 'Run ID' column of the output table).
2. Specify only the urls that failed.
3. The app will rerun ONLY the URLs you specify below and merge them with all the successful pages from the previous run.

## Actor input object example

```json
{
  "url": "duckduckgo.com",
  "websitename": "Example Inc",
  "websitedescription": "Example Inc is a website designed to provide services and products to consumers.",
  "websitecategory": "other",
  "pauseBetweenPages": 2,
  "urlFormat": "markdown",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `generatedFiles` (type: `string`):

All generated llms.txt files, markdown content, and downloadable package

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "duckduckgo.com",
    "websitename": "Example Inc",
    "websitedescription": "Example Inc is a website designed to provide services and products to consumers.",
    "websitecategory": "other",
    "pauseBetweenPages": 2,
    "urlFormat": "markdown",
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("onescales/the-llms-txt-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "duckduckgo.com",
    "websitename": "Example Inc",
    "websitedescription": "Example Inc is a website designed to provide services and products to consumers.",
    "websitecategory": "other",
    "pauseBetweenPages": 2,
    "urlFormat": "markdown",
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("onescales/the-llms-txt-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "duckduckgo.com",
  "websitename": "Example Inc",
  "websitedescription": "Example Inc is a website designed to provide services and products to consumers.",
  "websitecategory": "other",
  "pauseBetweenPages": 2,
  "urlFormat": "markdown",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call onescales/the-llms-txt-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,onescales/the-llms-txt-generator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Q06qO6KftC0d9I0SX/builds/OaGSALfhU2siRRcvh/openapi.json