Sitemap Extractor + robots.txt and llms.txt Checker for AI Crawlers

Give it domains and get every URL from their sitemaps (indexes, gzip, nested), the robots.txt verdict for each AI and search crawler (GPTBot, ClaudeBot, Google-Extended, PerplexityBot...) with the rule that decided, and the site's llms.txt, as one JSON record per site. $0.002 per site.

For SEO and AI-visibility audits, crawler and scraper builders who need a URL list without crawling, and AI agents (Claude, ChatGPT, Cursor and others through the Apify MCP server) that must check whether they may read a site and where its docs are.

Use cases

• Sitemap URL extraction: a site's full URL inventory with lastmod, filtered by path prefix or regex.
• AI crawler access audit: which AI bots a site blocks or allows, and the exact robots.txt line that decides.
• llms.txt check: whether a site publishes llms.txt / llms-full.txt, parsed into sections and links.
• Crawler seeding and change monitoring: feed the URL list to a scraper; compare lastmod between runs.
• "May I fetch this URL?": check mode tests any URLs against robots.txt for the user agents you name.

Quick start

This input (the Actor's example) analyzes two sites in about 5 seconds for about $0.006:

{"sites": ["nytimes.com", "developers.cloudflare.com"], "maxUrls": 1000}

One dataset item per site (abridged, real output from 2026-10-24):

{
  "input": "nytimes.com",
  "site": "https://nytimes.com",
  "status": "ok",
  "crawlerAccess": {
    "GPTBot": {"allowed": false, "group": "GPTBot", "rule": "Disallow: /", "named": true, "reason": "longest matching rule is 'Disallow: /'"},
    "Googlebot": {"allowed": true, "group": "*, Googlebot", "rule": null, "named": false, "reason": "no rule matches"}
  },
  "sitemaps": {"source": "robots.txt", "count": 8, "complete": false, "incompleteReason": "maxUrls (1,000) reached", "returnedUrls": 1000},
  "urls": [{"loc": "https://www.nytimes.com/athletic/...", "lastmod": "2026-10-24T16:20:30Z"}, "..."],
  "llmsTxt": {"present": false, "httpStatus": 404, "note": "Not published (HTTP 404)."}
}

For developers.cloudflare.com the same record shows every crawler allowed and llmsTxt.present: true with the file's title, sections and links.

Pricing in plain words

You pay $0.002 per site analyzed plus $0.001 per 1,000 URLs returned. An AI-access and llms.txt audit of 100 sites with includeUrls: false costs $0.20; a site with 25,000 sitemap URLs costs $0.027. Sites that don't exist or never answer cost nothing, and maxUrls caps the cost per site. Full table under Pricing.

Why this one

• It does not give up. Sitemaps in the wild are messy. This Actor reads gzip files whatever their Content-Type says, double-gzipped files, BOMs, missing or wrong XML namespaces, unescaped & in URLs, files cut off mid-way, plain-text sitemaps, RSS and Atom feeds, redirects across hosts, and HTML "not found" pages served with status 200. Transient errors (429, 5xx, timeouts) are retried with backoff and Retry-After.
• Partial beats failed. One broken sitemap gets its own error entry with the reason and a fix; the rest of the site is still returned. Time and size limits return what was read so far, marked complete: false with the reason.
• Huge sites are fine. Sitemaps are parsed as a stream (no 50 MB file held in memory), indexes are expanded recursively without loops, and URLs are deduplicated across all sitemaps.
• robots.txt done properly. RFC 9309 matching (exact product-token groups, longest match wins, Allow wins ties, * and $ wildcards), lenient parsing of real-world mistakes (each reported as a warning), and the RFC rules for missing (4xx = allow all) and broken (5xx = disallow all) robots.txt.
• Agent-shaped output. Every verdict says which group applied and which rule decided, so an agent can explain it.
• Polite and honest. Only files that sites publish for machines are fetched: /robots.txt, sitemaps, /llms.txt, /llms-full.txt. No page is ever crawled. Requests identify as TinplateSiteDiscovery/1.0 (+https://tinplatelabs.github.io), are paced per host (Crawl-delay honoured up to 3 s), and sitemaps and llms files are skipped where robots.txt disallows them (sitemaps that robots.txt itself declares count as an invitation).

Input

{
  "sites": ["python.org", "developers.cloudflare.com"],
  "userAgents": ["GPTBot", "ClaudeBot", "Google-Extended", "*"],
  "maxUrls": 5000,
  "urlPrefixes": ["/workers/"]
}

Check mode:

{"mode": "check", "urlsToCheck": ["https://example.com/blog/post", "https://example.com/admin/"], "userAgents": ["GPTBot", "Googlebot"]}

Output

One dataset item per site (shortened):

{
  "input": "example.com",
  "site": "https://example.com",
  "status": "ok",
  "robotsTxt": {
    "url": "https://example.com/robots.txt", "httpStatus": 200, "status": "ok", "sizeBytes": 412,
    "groups": [
      {"userAgents": ["GPTBot", "CCBot"], "allow": [], "disallow": ["/"], "crawlDelay": null},
      {"userAgents": ["*"], "allow": ["/admin/public/"], "disallow": ["/admin/"], "crawlDelay": 2.0}
    ],
    "sitemaps": ["https://example.com/sitemap_index.xml"],
    "otherDirectives": {}, "warnings": []
  },
  "crawlerAccess": {
    "GPTBot": {"allowed": false, "group": "GPTBot, CCBot", "rule": "Disallow: /", "crawlDelay": null, "named": true, "reason": "longest matching rule is 'Disallow: /'"},
    "ClaudeBot": {"allowed": true, "group": "*", "rule": null, "crawlDelay": 2.0, "named": false, "reason": "no rule matches"}
  },
  "sitemaps": {
    "source": "robots.txt", "count": 3, "failed": 1, "complete": true, "incompleteReason": null,
    "urlsScanned": 1207, "uniqueUrls": 1180, "duplicateUrls": 27, "invalidEntries": 0,
    "matchedUrls": 1180, "returnedUrls": 1180,
    "items": [
      {"url": "https://example.com/sitemap_index.xml", "type": "sitemapindex", "httpStatus": 200, "gzip": false, "urls": 0, "sitemaps": 2, "depth": 0, "error": null},
      {"url": "https://example.com/post-sitemap.xml.gz", "type": "urlset", "httpStatus": 200, "gzip": true, "urls": 1207, "sitemaps": 0, "depth": 1, "error": null},
      {"url": "https://example.com/old-sitemap.xml", "type": null, "httpStatus": 404, "urls": 0, "sitemaps": 0, "depth": 1,
       "error": {"code": "HTTP_ERROR", "message": "HTTP 404. The sitemap URL listed for this site does not work; the site owner needs to fix or remove it."}}
    ]
  },
  "urls": [
    {"loc": "https://example.com/blog/hello", "lastmod": "2026-09-30", "changefreq": "weekly", "priority": 0.8}
  ],
  "llmsTxt": {
    "url": "https://example.com/llms.txt", "present": true, "httpStatus": 200, "sizeBytes": 1834,
    "title": "Example", "summary": "Docs for the Example API.", "details": null, "validFormat": true, "linkCount": 12,
    "sections": [{"title": "Docs", "optional": false, "notes": null,
                  "links": [{"title": "Quick start", "url": "https://example.com/start.md", "description": "install and first call"}]}],
    "content": "# Example\n\n> Docs for the Example API.\n..."
  },
  "llmsFullTxt": {"url": "https://example.com/llms-full.txt", "present": false, "httpStatus": 404, "note": "Not published (HTTP 404)."},
  "warnings": []
}

Check mode returns one item per URL:

{"input": "https://example.com/admin/", "url": "https://example.com/admin/", "status": "ok",
 "robotsTxtUrl": "https://example.com/robots.txt", "robotsTxtStatus": "ok",
 "results": {"GPTBot": {"allowed": false, "group": "GPTBot, CCBot", "rule": "Disallow: /", "named": true, "crawlDelay": null, "reason": "longest matching rule is 'Disallow: /'"}}}

A site that cannot be analyzed gives an error item, and the run carries on:

{"input": "no-such-domain.example", "status": "error",
 "error": {"code": "SITE_NOT_FOUND", "message": "The host name 'no-such-domain.example' does not resolve in DNS (...). Check the spelling of the domain; the site may not exist or may have expired."}}

• robotsTxt.status: ok, not_found (4xx: everything allowed), unreachable (5xx or no answer: everything disallowed, per RFC 9309), html (an HTML page was served: no rules).
• sitemaps.source: robots.txt (declared), common-locations (found at /sitemap.xml, /sitemap_index.xml or /wp-sitemap.xml) or none.
• Site error codes: INVALID_SITE, INVALID_URL, SITE_NOT_FOUND, SITE_UNREACHABLE, PRIVATE_ADDRESS, TIMEOUT, SPENDING_LIMIT_REACHED, INTERNAL_ERROR. Sitemap error codes: HTTP_ERROR, NOT_A_SITEMAP, PARSE_ERROR, TOO_LARGE, FETCH_FAILED, BLOCKED_BY_ROBOTS. Every message says what to do next.
• If a site's URL list is too big for one dataset item (~8 MB), it is saved in the run's key-value store and the item gives urlsKeyValueStoreKey and urlsUrl.

Pricing (pay per event)

Examples: an AI-access and llms.txt audit of 100 sites with includeUrls: false costs $0.20. A site with 25,000 URLs in its sitemaps costs $0.002 + $0.025 = $0.027. Sites that do not exist or never answer are not charged. When the run's maximum total charge is reached, URL lists are cut to what it covers and the remaining sites are marked SPENDING_LIMIT_REACHED.

Limits

• 1,000 sites (or 10,000 URLs to check) per run; up to 200,000 returned URLs per site; up to 1,000,000 URLs counted per site with includeUrls: false.
• Up to 2,000 sitemap files per site, nested up to 6 levels; 100 MB per uncompressed sitemap; 120 s per sitemap download.
• Only public addresses are fetched. Redirects: up to 5 per request.
• User-agent groups are matched by exact product token (as RFC 9309 says); Google's fallback from tokens such as Googlebot-Image to Googlebot is not applied.
• Image, video and news sitemap extensions are not returned (their page URLs are).
• llms.txt content is returned up to 100,000 characters; llms-full.txt is summarized (size, title, headings), not returned.

Use from an AI agent (MCP)

Add the Apify MCP server to your agent and enable this Actor. Typical calls:

{"sites": ["example.com"], "includeUrls": false}

{"sites": ["example.com"], "urlPrefixes": ["/docs/"]}

Open-source licenses

All dependencies are permissively licensed. CI fails on any copyleft dependency.

The robots.txt parser and matcher are our own (RFC 9309). Test-only: pytest (MIT), Protego (BSD-3-Clause) as a reference to compare robots.txt verdicts against.

Made by Tinplate Labs. This Actor is maintained by an AI; there is no support desk, so every error message is written to tell you exactly what to change.