robots.txt & llms.txt Monitor — Track Crawler & AI Agent Access Rules

Fetches and parses a domain's robots.txt (RFC 9309) and llms.txt (the emerging file aimed at AI agents), and flags what changed since the last run. No login, no API keys — both files are public by design.

What it does

• Parses robots.txt: User-agent groups, Allow/Disallow rules and Sitemap entries.
• Parses llms.txt: title, summary and sections with their links, following the format proposed at llmstxt.org.
• Compares against the previous run for the same domain and returns a plain-text list of changes (sitemap added/removed, a new Disallow rule for User-agent: *, llms.txt appearing/disappearing, or its title/link count changing). The first run for a domain never reports changes (there's nothing to compare against yet).
• For the comparison to actually work, schedule a recurring run on the same domain (daily or weekly Schedule in the Apify console) — the Actor stores its own snapshot between runs.

Who it's for

• SEO/growth teams that want to know if someone touched production robots.txt without telling anyone.
• Teams building their own llms.txt who want to confirm it published correctly and isn't changing on its own.
• Emerging category (2026): monitoring how a site declares its access rules for AI agents.

Input

Field	Type	Required	Description
url	string	Yes	One domain per run (pay-per-event = 1 event per domain).

{ "url": "https://example.com" }

Output

{
  "url": "https://apify.com",
  "hostname": "apify.com",
  "robotsTxt": {
    "found": true,
    "sitemaps": ["https://apify.com/sitemap.xml"],
    "groupCount": 1,
    "disallowForStar": []
  },
  "llmsTxt": {
    "found": true,
    "title": "Apify",
    "summary": "Apify is a marketplace of ready-to-run tools for AI...",
    "sectionCount": 9,
    "linkCount": 72
  },
  "changed": false,
  "firstRun": false,
  "changes": [],
  "checkedAt": "2026-09-20T03:37:16.202Z"
}

When there are changes, changes carries readable strings, e.g.: "Sitemaps: added https://example.com/sitemap-news.xml."

Pricing

Pay-per-event, provisional. Starting reference: USD 0.02 per domain monitored per run (03-plan.md §2).

How to call it

curl "https://api.apify.com/v2/acts/cuantic-data~robots-llms-txt-monitor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Terms and limits

See ./TERMS.md: both files are public by design and explicitly meant to be read by automated processes (RFC 9309 for robots.txt, the llmstxt.org proposal for llms.txt).

What it does NOT do

• Doesn't cover Crawl-delay, Host or wildcards (*, $) within paths in this v1.
• Doesn't crawl the site or use the rules to decide whether to access anything else: it only reads and reports both files.
• The between-run comparison depends on the user scheduling a recurring run on the same domain (a one-off run always returns firstRun: true).

FAQ

Why does my domain show llms.txt found: false? Most sites don't publish llms.txt yet — it's a category that only started in 2026. It's not an Actor error.

Found a bug? Email cuanticwindows@gmail.com — we reply within 72 hours.

---

See build/README.md for how to run tests and publish this Actor.