# robots.txt & llms.txt Monitor - Crawler & AI Access Rules (`cuantic_data/robots-llms-txt-monitor`) Actor

Fetches and parses a domain's robots.txt (RFC 9309) and llms.txt (the emerging AI-agent-friendly file), and flags what changed since the last run. Both files are public by design, meant to be read by automated clients - no login, no scraping of restricted content.

- **URL**: https://apify.com/cuantic\_data/robots-llms-txt-monitor.md
- **Developed by:** [Cuantic Data](https://apify.com/cuantic_data) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## robots.txt & llms.txt Monitor — Track Crawler & AI Agent Access Rules

Fetches and parses a domain's `robots.txt` (RFC 9309) and `llms.txt` (the
emerging file aimed at AI agents), and flags what changed since the last
run. No login, no API keys — both files are public by design.

### What it does

- Parses `robots.txt`: `User-agent` groups, `Allow`/`Disallow` rules and
  `Sitemap` entries.
- Parses `llms.txt`: title, summary and sections with their links,
  following the format proposed at [llmstxt.org](https://llmstxt.org/).
- **Compares against the previous run for the same domain** and returns a
  plain-text list of changes (sitemap added/removed, a new `Disallow` rule
  for `User-agent: *`, `llms.txt` appearing/disappearing, or its title/link
  count changing). The first run for a domain never reports changes (there's
  nothing to compare against yet).
- For the comparison to actually work, **schedule a recurring run on the
  same domain** (daily or weekly Schedule in the Apify console) — the Actor
  stores its own snapshot between runs.

### Who it's for

- SEO/growth teams that want to know if someone touched production
  `robots.txt` without telling anyone.
- Teams building their own `llms.txt` who want to confirm it published
  correctly and isn't changing on its own.
- Emerging category (2026): monitoring how a site declares its access rules
  for AI agents.

### Input

| Field | Type | Required | Description |
|---|---|---|---|
| `url` | string | Yes | One domain per run (pay-per-event = 1 event per domain). |

```json
{ "url": "https://example.com" }
```

### Output

```json
{
  "url": "https://apify.com",
  "hostname": "apify.com",
  "robotsTxt": {
    "found": true,
    "sitemaps": ["https://apify.com/sitemap.xml"],
    "groupCount": 1,
    "disallowForStar": []
  },
  "llmsTxt": {
    "found": true,
    "title": "Apify",
    "summary": "Apify is a marketplace of ready-to-run tools for AI...",
    "sectionCount": 9,
    "linkCount": 72
  },
  "changed": false,
  "firstRun": false,
  "changes": [],
  "checkedAt": "2026-09-20T03:37:16.202Z"
}
```

When there are changes, `changes` carries readable strings, e.g.:
`"Sitemaps: added https://example.com/sitemap-news.xml."`

### Pricing

Pay-per-event, provisional. Starting reference: USD 0.02 per domain
monitored per run (`03-plan.md` §2).

### How to call it

```bash
curl "https://api.apify.com/v2/acts/cuantic-data~robots-llms-txt-monitor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

### Terms and limits

See [`TERMS.md`](./TERMS.md): both files are public by design and
explicitly meant to be read by automated processes (RFC 9309 for
`robots.txt`, the llmstxt.org proposal for `llms.txt`).

### What it does NOT do

- Doesn't cover `Crawl-delay`, `Host` or wildcards (`*`, `$`) within paths
  in this v1.
- Doesn't crawl the site or use the rules to decide whether to access
  anything else: it only reads and reports both files.
- The between-run comparison depends on the user scheduling a recurring run
  on the same domain (a one-off run always returns `firstRun: true`).

### FAQ

**Why does my domain show `llms.txt found: false`?**
Most sites don't publish `llms.txt` yet — it's a category that only started
in 2026. It's not an Actor error.

**Found a bug?**
Email `cuanticwindows@gmail.com` — we reply within 72 hours.

***

See `build/README.md` for how to run tests and publish this Actor.

# Actor input Schema

## `url` (type: `string`):

http(s) URL of the site (the origin is used: protocol + host). Example: https://example.com

## Actor input object example

```json
{
  "url": "https://apify.com"
}
```

# Actor output Schema

## `results` (type: `string`):

robots.txt and llms.txt content for the monitored domain, and what changed since the previous run.

## `runSummary` (type: `string`):

Hostname monitored, whether it changed, and whether this was the first run for it.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://apify.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("cuantic_data/robots-llms-txt-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "url": "https://apify.com" }

# Run the Actor and wait for it to finish
run = client.actor("cuantic_data/robots-llms-txt-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://apify.com"
}' |
apify call cuantic_data/robots-llms-txt-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,cuantic_data/robots-llms-txt-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Er2FchnEa7nHgfDwJ/builds/cezILDlJReeDURyW5/openapi.json