# llms.txt Generator + Validator (`checksmithcats/llms-txt-generator-validator`) Actor

Creates llms.txt and robots.txt snippets from submitted resources and validates their basic shape before the user publishes any files.

- **URL**: https://apify.com/checksmithcats/llms-txt-generator-validator.md
- **Developed by:** [Checksmith Cats](https://apify.com/checksmithcats) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / llms.txt report generated

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## llms.txt Generator + Validator

Format submitted public resources into `llms.txt`, then check the file shape before publishing it.

This Actor takes site details, resource links, and optional user-chosen crawler rules. It generates `llms.txt` and an optional `robots.txt` snippet, then returns validation findings and an Evidence Pack.

### What it checks

- top-level site heading
- Markdown resource-link shape
- absolute HTTP(S) resource URLs
- duplicate resource URLs
- submitted crawler-rule formatting
- crawler tokens outside the built-in reference list

### Outputs

- generated `llms.txt`
- optional `robots-snippet.txt`
- `REPORT.json`, `REPORT.html`, and `REPORT.md`
- dataset finding rows

The Actor does not fetch the deployed site or test live crawler behavior.

### Boundaries

`llms.txt` is treated as an emerging convention. This Actor formats and validates submitted inputs; it does not:

- choose a crawler policy for you
- provide legal or SEO advice
- promise that any AI or crawler service reads or follows the file
- predict traffic, indexing, ranking, citations, or AI answers
- publish or modify files on your site

Review the generated files before deployment.

### Pricing

Pay per event:

| Event | Price |
|---|---:|
| `llms-txt-report-generated` | $0.30 per completed report |
| `llms-txt-item-checked` | $0.02 per generated or validated item |

The standard sample contains five billable items and costs **$0.40**. With the current input limits, the theoretical maximum is 201 items and **$4.32** per run. Input-validation failures are not charged. Set an Apify max cost per run if you need a lower account-level limit.

### Data handling

Generated files and validation findings are written to normal Apify run storage. Submitted Actor input is also subject to Apify run storage and retention settings. Do not submit confidential policy text or private resource URLs.

### Support boundary

Support covers reproducible Actor defects, documented input-format questions, missing generated files, and a mismatch between documented and recorded billing events.

Support does not cover policy selection, SEO or legal advice, editing site content, deploying files, custom templates, custom integrations, or evaluating AI-platform outcomes.

### Related Actors

- **AI Crawler Policy Preflight**: inspect crawler-facing files after publication
- **AI Crawler Policy Drift Monitor**: track later changes to published crawler policy
- **AI Visibility Signal Report**: inventory llms.txt alongside other public-site signals

### Minimal input

```json
{
  "siteName": "Example Docs",
  "siteUrl": "https://example.com",
  "siteDescription": "Documentation and public resources for Example Docs.",
  "policyNotes": "This file lists the public resources that Example Docs wants to make easy to inspect.",
  "entries": [
    {
      "title": "Documentation",
      "url": "https://example.com/docs",
      "description": "Primary documentation entry point.",
      "section": "Docs"
    },
    {
      "title": "API Reference",
      "url": "https://example.com/api",
      "description": "Reference pages for public API endpoints.",
      "section": "Docs",
      "optional": true
    }
  ],
  "crawlerRules": [
    {
      "userAgent": "GPTBot",
      "directive": "Disallow",
      "path": "/"
    },
    {
      "userAgent": "OAI-SearchBot",
      "directive": "Allow",
      "path": "/"
    }
  ],
  "includeRobotsSnippet": true
}
```

### API and scheduled use

Run the same validated input from Apify Console, the Actor API, CLI, schedules, or integrations. The run writes machine-readable findings to the default dataset and the complete report to the run storage.

### Common input errors

Start with the published example and keep `siteName`, `siteUrl`, `siteDescription`, `policyNotes`, `entries` in the documented JSON shape. Malformed JSON, unsupported URL schemes, missing required values, and inputs above the stated limits are rejected rather than repaired silently.

# Actor input Schema

## `siteName` (type: `string`):

The site or product name used as the top-level llms.txt heading.

## `siteUrl` (type: `string`):

Optional public site URL to include in the generated llms.txt.

## `siteDescription` (type: `string`):

Optional short description, written into the llms.txt blockquote.

## `policyNotes` (type: `string`):

Optional user-provided policy note. This Actor formats it as text; it does not choose the policy for you.

## `entries` (type: `array`):

Resources to list in llms.txt. The Actor formats each item as a Markdown link.

## `crawlerRules` (type: `array`):

Optional user-chosen Allow/Disallow rules to format as a robots.txt snippet.

## `llmsTxtText` (type: `string`):

Optional existing llms.txt content to validate instead of only validating the generated text.

## `includeRobotsSnippet` (type: `boolean`):

Generate a robots.txt snippet from submitted crawlerRules.

## Actor input object example

```json
{
  "siteName": "Example Docs",
  "entries": [
    {
      "title": "Documentation",
      "url": "https://example.com/docs",
      "description": "Primary documentation entry point.",
      "section": "Docs"
    }
  ],
  "crawlerRules": [
    {
      "userAgent": "GPTBot",
      "directive": "Disallow",
      "path": "/"
    }
  ],
  "includeRobotsSnippet": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `report` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteName": "Example Docs",
    "entries": [
        {
            "title": "Documentation",
            "url": "https://example.com/docs",
            "description": "Primary documentation entry point.",
            "section": "Docs"
        }
    ],
    "crawlerRules": [
        {
            "userAgent": "GPTBot",
            "directive": "Disallow",
            "path": "/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("checksmithcats/llms-txt-generator-validator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "siteName": "Example Docs",
    "entries": [{
            "title": "Documentation",
            "url": "https://example.com/docs",
            "description": "Primary documentation entry point.",
            "section": "Docs",
        }],
    "crawlerRules": [{
            "userAgent": "GPTBot",
            "directive": "Disallow",
            "path": "/",
        }],
}

# Run the Actor and wait for it to finish
run = client.actor("checksmithcats/llms-txt-generator-validator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteName": "Example Docs",
  "entries": [
    {
      "title": "Documentation",
      "url": "https://example.com/docs",
      "description": "Primary documentation entry point.",
      "section": "Docs"
    }
  ],
  "crawlerRules": [
    {
      "userAgent": "GPTBot",
      "directive": "Disallow",
      "path": "/"
    }
  ]
}' |
apify call checksmithcats/llms-txt-generator-validator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,checksmithcats/llms-txt-generator-validator"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/OodJAmcufkoQldzzq/builds/nl2LVl0Pt3ifQUtmP/openapi.json