# llms.txt Auditor: AI Search Readiness Checker (`master_invictus/llms-txt-auditor`) Actor

Check any website's llms.txt for AI crawlers and answer engines. Finds missing or broken files, invalid structure and dead links, gives each site a readiness score and suggests fixes. Bulk-friendly, from $0.003 per site.

- **URL**: https://apify.com/master_invictus/llms-txt-auditor.md
- **Developed by:** [Frank Nagy](https://apify.com/master_invictus) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## llms.txt Auditor: AI Search Readiness Checker

Check any website's llms.txt for AI crawlers and answer engines. Finds missing or broken files, invalid structure and dead links, gives each site a readiness score and suggests fixes. Bulk-friendly, from $0.003 per site.

Checks a list of websites for [llms.txt](https://llmstxt.org/) readiness. For each origin it fetches `/llms.txt` and `/llms-full.txt`, validates the index file against the llms.txt spec, optionally HEAD-checks the linked URLs, and reports whether `robots.txt` blocks common AI crawlers.

**Input:** `startUrls[]` (default `https://example.com`). Optional: `checkLinks` (default on), `maxLinksToCheck` (default 25), `draftStarterLlmsTxt` (default off), `maxConcurrency` (default 5), `requestTimeoutSecs` (default 20). The default site needs no credentials, so the store health run can finish on an empty or missing `/llms.txt`.

**Output: one dataset item per site**

- `hasLlmsTxt`, `hasLlmsFullTxt`, `specValid`
- `issues[]` (`code`, `severity`, `file`, `message`)
- `linkCount`, `brokenLinks` (each `{ url, status, error, file }`)
- `sizeBytes` (combined size of the accepted llms files)
- `robotsAiDirectives` for **GPTBot**, **ClaudeBot**, **PerplexityBot**, and **Google-Extended** (`blocked`, `matchedGroup`, `rule`)
- `readinessScore` (0-100) and `scoreLabel`: the **MasterInvictus AI readiness score (not an official standard)**
- `fixSuggestions`
- `starterLlmsTxt` when `draftStarterLlmsTxt` is on and `/llms.txt` is missing (homepage `<title>`, meta description, and the first `<nav>` / header / main links; no LLM call)

**Run summary** is stored in the key-value store as `SUMMARY` (sites checked and errored, average **MasterInvictus AI readiness score (not an official standard)**, spec-valid count, crawlers blocked). It is not a dataset row.

`specValid` is true only when `/llms.txt` exists and matches the spec: an H1 title first (the only required section), a blockquote summary directly under it is checked and recommended, every H2 is a markdown link list (`- [name](url)` plus optional `: notes`), and every link is an absolute `http(s)` URL or a relative path. `/llms-full.txt` is fetched and parsed the same way; structural mismatches are warnings, because the expanded file is often concatenated documentation.

**Score (100 before penalties):** llms.txt present 30, H1 15, blockquote summary 15, H2 link section 10, syntactically valid links 10, llms-full.txt present 10, AI crawlers allowed 10. Checked links that fail subtract up to 15. Unreachable domains get `readinessScore: null` and are not charged.

**Fails loudly:** the run exits non-zero when `startUrls` is empty or invalid, or when no domain could be checked.

**Starter draft:** with `draftStarterLlmsTxt: true`, a missing `/llms.txt` produces `starterLlmsTxt` from the homepage title, meta description, and main nav. Nothing is sent to an LLM.

### Pricing (pay-per-event)

| Event | Price |
|---|---|
| `run-started`: custom flat event, charged once per run and only after the first site is checked successfully. A run that checks no site is not billed (not Apify's memory-scaled `apify-actor-start`) | $0.01 |
| `site-audited`: once per domain checked successfully. Failed or unreachable domains are not charged | $0.003 |

When publishing, register `run-started` and `site-audited` as custom events at these prices. Do not enable the synthetic start event, which is priced per GB of memory.

***

Built and maintained by **MasterInvictus**.

# Actor input Schema

## `startUrls` (type: `array`):

Websites to audit. Each URL's origin is checked for /llms.txt, /llms-full.txt, and /robots.txt. Duplicate origins are checked once.

## `checkLinks` (type: `boolean`):

HEAD each markdown link in the llms.txt files (GET if HEAD is not allowed), up to the cap. Dead links lower the readiness score.

## `maxLinksToCheck` (type: `integer`):

Cap on link checks per site. Further links are counted but not requested.

## `draftStarterLlmsTxt` (type: `boolean`):

When /llms.txt is missing, build a starter file from the homepage title, meta description, and main navigation links. No LLM is used.

## `maxConcurrency` (type: `integer`):

How many sites to audit at once. Each site's own requests run one at a time, so this is also the cap on in-flight HTTP calls.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each HTTP request (the llms files, robots.txt, homepage, and link checks).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "checkLinks": true,
  "maxLinksToCheck": 25,
  "draftStarterLlmsTxt": false,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 20
}
```

# Actor output Schema

## `sites` (type: `string`):

One row per site: llms.txt and llms-full.txt presence, spec validity, link health, AI crawler robots rules, and the MasterInvictus AI readiness score (not an official standard).

## `issues` (type: `string`):

One row per issue, with a fix context for missing files, invalid structure, dead links, and blocked AI crawlers.

## `summary` (type: `string`):

Sites checked, average MasterInvictus AI readiness score (not an official standard), and how many AI crawlers were blocked.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("master_invictus/llms-txt-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("master_invictus/llms-txt-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ]
}' |
apify call master_invictus/llms-txt-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,master_invictus/llms-txt-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/6hrEiVvF42jmMPJFD/builds/NsDpphQcNvLBfK9cl/openapi.json