# llms.txt Generator & AI Crawler Audit (`miacx/llms-txt-generator`) Actor

Check AI crawler access, llms.txt, sitemaps and sampled page metadata. Generate an llms.txt draft from discovered URLs and page metadata.

- **URL**: https://apify.com/miacx/llms-txt-generator.md
- **Developed by:** [mia.cx](https://apify.com/miacx) (community)
- **Categories:** SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## llms.txt Generator & AI Crawler Audit

Check a website's published crawler rules, and whether its CDN lets those crawlers in, before deciding what your agent can read.
Audit robots.txt, llms.txt, sitemaps and sampled page metadata, then get an llms.txt draft.
Each successfully audited page costs $0.009, plus $0.00005 per run start at default memory.
Generation uses discovered URLs and metadata; it makes no runtime LLM calls and does not convert page bodies to Markdown.

### Try it

```json
{"urls":["https://llmstxt.org"],"maxPagesPerSite":5}
```

An empty input `{}` runs the same audit, sampling up to five pages. No upstream API key is needed.

### Use from an agent

Add `https://mcp.apify.com?tools=miacx/llms-txt-generator` to your MCP client and sign in to Apify.
Call the dedicated Actor tool, then use `get-dataset-items` to read the completed run's dataset.
Or set `APIFY_TOKEN` to your Apify API token and call the API for JSON rows:

```sh
curl --fail-with-body 'https://api.apify.com/v2/actors/miacx~llms-txt-generator/run-sync-get-dataset-items' \
  -H "Authorization: Bearer $APIFY_TOKEN" -H 'Content-Type: application/json' \
  -d '{"urls":["https://llmstxt.org"],"maxPagesPerSite":5}'
```

### What you get

The dataset contains one `site` summary per unique origin and one `page` row per sampled page.
Read `type`, then `status`. An `ok` page is billable even when it contains findings; site summaries incur no Page events.
A site row is `robots_only` when robots.txt was read but no page could be sampled: its crawler verdicts and live checks are complete, and each page row keeps its own error.
Use `generatedLlmsTxt` on site rows for the draft and `findings` for issue codes and suggested fixes.

Selected fields from parsing the recorded `tests/fixtures/resources.json` page fixture:

```json
{
  "type": "page",
  "status": "ok",
  "input": "https://llmstxt.org",
  "source": "https://llmstxt.org/core.html",
  "fetchedAt": "2026-10-31T02:07:54.430Z",
  "reason": null,
  "title": "Python source – llms-txt",
  "description": "Source code for llms_txt Python module, containing helpers to create and use llms.txt files",
  "canonical": null,
  "jsonLdTypes": []
}
```

Page rows also contain headings in document order, a confirmed `markdownUrl` or null, and `findings`.
Missing metadata is null; missing collections are empty.
An HTML page gets at most one Markdown-twin probe. A null URL means none detected, not proof none exists.

Site rows contain crawler decisions, resource statuses, sitemap URLs, discovered and successful sample counts, request counts and bytes.
`crawlers` entries contain `name`, `allowed` and a one-based matching robots rule line, or null.
`allowed: null` means the policy could not be read.
Each entry also reports a live check: `liveStatus` is the HTTP status of one GET of the supplied page sent with that crawler's published user agent, and `liveBlocked` is true for any 4xx except 404 and 410 (such as 402 pay-per-crawl or 403), a 503 or a CDN challenge.
Both are null when nothing was sent: the vendor publishes no user agent (ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, Bytespider), robots.txt disallows the crawler or this audit, or the request failed.
`allowed: true` with `liveBlocked: true` adds a `crawler_edge_blocked` finding: robots.txt invites the crawler, but the CDN or firewall turns it away.

```json
{ "name": "GPTBot", "allowed": true, "rule": null, "liveStatus": 403, "liveBlocked": true }
```

A live check shows how the site answers that user agent from Apify's network, not how it treats the vendor's own IP ranges.
`llmsTxtValid` and `llmsFullValid` are null when their resources are unavailable.
Fields belonging to the other row type are absent.

Page `fetchedAt` records the original observation time, including on cache hits.
Site `fetchedAt` records summary production time. `reason` is null on success.

### Input

| Field | Default | Accepted input |
| --- | --- | --- |
| `urls` | `["https://llmstxt.org"]` | Up to five public HTTP(S) URL strings. An empty list returns one `no_results` site row. |
| `maxPagesPerSite` | `5` | Integer 1–25. Includes the supplied page; same-origin sitemap URLs supply the rest, or same-origin links on the supplied page when no sitemap yields any. |

Omitted fields use their defaults.
Duplicate origins share one audit within a run, using the first supplied URL as the starting page.
A supplied URL that redirects to another origin, such as `https://docs.bsky.app/` to `https://bsky.network/`, is audited where it lands unless its own origin's sitemap lists pages.
The site row's `source` is then the landed origin, and its robots.txt, sitemaps and llms.txt are the ones checked.
Crawler decisions describe that exact URL, not every path on the site.
Invalid URLs become individual site rows; malformed batch shapes fail input validation.

### Pricing

One Page event costs $0.009, including metadata checks and up to one Markdown-twin probe.
Five successfully audited pages cost $0.04505, including $0.00005 per run start at default 128 MB memory.
One thousand Page events cost $9, plus run starts.

Pages with findings count. Site summaries, failed pages and no-result rows incur no Page events.
Duplicate origins share work within the run. Cached successful pages still incur Page events.
The $0.00005 run start applies even with no billable pages at default memory.
Apify counts one start event per GB of memory, with a minimum of one event.

### Limits and data freshness

- Each run allows five sites, four concurrent upstream loads, 180 network attempts, 25 seconds of fetch work and 8 MiB total response bytes.
  Live crawler checks add at most 14 header-only requests per site, outside the 180.
- Resources cap at 512 KiB. Each site inspects up to two sitemaps and four index children at one level.
  When no sitemap yields a same-origin URL, links on the supplied page that robots.txt permits are discovered instead.
- Discovery retains up to 1,000 URLs; the generated draft includes up to 200, grouped by first path section. Sampled pages that answered 404 or 410 are left out.
- Draft labels use sampled metadata where available and paths otherwise. Confirmed Markdown links take precedence.
  A page title such as `Blog - AT Protocol` is labelled `Blog` when the draft's H1 is `AT Protocol`.
- The draft's H1 is the supplied page's title. If that page fails, it is a site name most sampled titles share, such as `Nature` in `Mice age slower | Nature`, else the hostname.
- No JavaScript rendering, compressed sitemap expansion or cross-origin sitemap crawling runs. HTML soft-404 pages may appear successful.

The llms.txt check lints a leading H1 and links in optional H2 sections against the [proposal](https://llmstxt.org/).
The llms-full.txt check only tests for available, nonempty non-HTML text.
An HTML page served at either path, such as a catch-all soft 404, is reported as `llms_txt_html` or `llms_full_html` rather than as malformed Markdown.
Crawler decisions report declared rules; they do not prove vendor enforcement or search-ranking effects.

Robots 404 means no published restriction. Other robots failures prevent page sampling.
The audit respects rules for `mia.cx-apify`; a positive crawl-delay skips sampling within this synchronous budget.

Successful resources can be reused for up to one hour from the named store `llms-txt-audit-cache`.
Shorter upstream freshness and cache restrictions apply.
Cache snapshots retain at most 32 resources and 1 MiB per site; records remain until overwritten or deleted.

### If a row fails

| Status | Next step |
| --- | --- |
| `invalid_url`, `private_host` | Supply a public HTTP(S) URL without credentials or custom ports. |
| `robots_denied`, `crawl_delay_required` | Inspect the published policy; sampling was skipped. |
| `http_403`, `http_404`, `network_error` | Check the resource URL and public availability. |
| `bot_challenge` | The site served a bot challenge (Cloudflare, AWS WAF or similar) instead of the resource; it was not audited or billed. |
| `unsupported_content`, `response_too_large` | Supply a supported HTML or Markdown page within the size cap. |
| `resource_limit`, `time_limit`, `budget_exhausted` | Reduce sites or pages, or increase the Page charge budget where applicable. |
| `robots_only` | Use `crawlers` as usual; check page rows for why no page could be sampled. |
| `no_results` | Check page rows and findings; no sampled page succeeded. |

For workflows exceeding the synchronous API's 300-second wait, use [asynchronous runs](https://docs.apify.com/api/v2/actor-runs-post).

# Actor input Schema

## `urls` (type: `array`):

Public HTTP(S) URLs, up to 5 strings. Omitted audits https://llmstxt.org; \[] returns no_results. Duplicate origins share one audit per run, using the first supplied URL as the starting page. Malformed or private URLs return individual error rows.

## `maxPagesPerSite` (type: `integer`):

Maximum HTML or Markdown pages sampled per site, 1-25; default 5 when omitted. Includes the supplied page, with remaining URLs from same-origin sitemaps, or from same-origin links on the supplied page when no sitemap yields any. Each successfully audited page incurs one Page event even when findings exist. Errors and site summaries incur no Page events.

## Actor input object example

```json
{
  "urls": [
    "https://llmstxt.org"
  ],
  "maxPagesPerSite": 5
}
```

# Actor output Schema

## `results` (type: `string`):

One row per item, each with a status field.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("miacx/llms-txt-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("miacx/llms-txt-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call miacx/llms-txt-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,miacx/llms-txt-generator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/zAApQOJctQnCOT25u/builds/TdxBhGfuqKyOhdGCy/openapi.json