Site Discovery for AI Agents: sitemaps, robots.txt, RSS & llms.txt

Give it a list of domains. It maps each site without crawling a single page: what robots.txt allows for your user agent, every sitemap (including sitemap indexes and gzipped .xml.gz files), RSS/Atom/JSON feeds, and whether the site publishes llms.txt / llms-full.txt for AI agents. You can also get a clean, filtered URL list with lastmod, changefreq and priority.

It is built for teams that feed websites to LLMs, RAG pipelines, AI agents and SEO tools. You find out what to fetch, what has changed since your last run, and what you are allowed to fetch before you spend money on a crawler.

What you get per domain

• robots.txt summary for the user agent you choose (for example GPTBot, Googlebot or your own bot). It shows the matched group, the allow and disallow rules, crawl-delay, whether the whole site is blocked for you, and which well-known AI bots (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot…) are blocked. Rules are matched per RFC 9309: longest match wins, with * and $ wildcards.
• Sitemaps listed in robots.txt. When robots.txt lists none, the common paths are tried. Sitemap indexes are followed, newest first, and gzip is handled.
• Feeds from <link rel="alternate"> on the homepage, or from common paths such as /feed, /rss.xml and /atom.xml. Each feed shows its format, title, item count and latest item date.
• llms.txt and llms-full.txt: whether each is present, its size, and an excerpt.
• URL counts: URLs scanned, URLs that matched your filters, URLs filtered out, URLs disallowed by robots.txt, and URLs output.

There is also an optional per-URL record: url, lastmod, changefreq, priority, sourceSitemap and allowedByRobots.

Filters

• includePatterns / excludePatterns match the URL path. Use a glob (/blog/**, /docs/*/intro) or a regex with the re: prefix (re:^/p/\d+).
• lastmodSince keeps only URLs modified on or after a date. Child sitemaps older than that date are skipped. URLs without a lastmod are dropped when this is set.
• maxUrls caps the URL records per domain. maxSitemaps caps the sitemap files fetched per domain.

Output sample (real test run, trimmed)

{
 "recordType": "domain-summary", "domain": "vercel.com", "homepageStatus": 200,
 "robots": { "status": 200, "present": true, "userAgent": "GPTBot", "matchedGroup": "*",
   "disallow": ["/api/", "/oauth", "/confirm", "..."], "allow": ["/api/og/*", "..."],
   "crawlDelay": null, "blocksEverything": false, "aiBotsBlocked": [] },
 "sitemaps": [ { "url": "https://vercel.com/sitemap.xml", "status": 200, "type": "urlset", "entries": 9269, "error": null } ],
 "feeds": [ { "url": "https://vercel.com/atom", "foundVia": "common-path", "format": "atom", "title": "Vercel News", "itemCount": 2102, "latestItem": "2026-10-29T18:44:00.000Z" } ],
 "llmsTxt": { "present": true, "bytes": 5397, "excerpt": "# Vercel\n\n> Vercel is a cloud platform for building, deploying, and scaling web applications and AI workloads. ..." },
 "llmsFullTxt": { "present": false },
 "urlCounts": { "matchedFilters": 30, "disallowedByRobots": 0, "output": 30, "truncatedAtMaxUrls": true },
 "error": null
}
{ "recordType": "url", "domain": "vercel.com", "url": "https://vercel.com/connect/ocean-io",
  "lastmod": "2026-10-25T14:10:15.167Z", "changefreq": null, "priority": null,
  "sourceSitemap": "https://vercel.com/sitemap.xml", "allowedByRobots": true }

The Domain summaries and URLs dataset views split the two record types.

Pricing (pay per event)

Event	Price
domain-summary: one reachable domain analyzed	$0.002
url-discovered: one per-URL record written	$0.0003

Failures are never charged. A domain where we could not get robots.txt, the homepage or any sitemap is still listed with an error, but it is free. URLs dropped by your filters or by robots.txt are free too. Cost example: 100 domains with 500 URLs each = 100 × $0.002 + 50,000 × $0.0003 = $15.20. With summaries only (outputUrls: false), the same run costs $0.20.

Limitations

• Only one origin is analyzed per input. wordpress.org/news analyzes wordpress.org, so enter subdomains such as blog.example.com separately.
• Sites that block datacenter IPs may return 403 for sitemaps. Those sitemaps are listed with their error.
• When lastmodSince is set and a site has no lastmod values (common), every URL is filtered out. Check urlCounts.filteredOut.

Acceptable use

This Actor reads only public, machine-readable files that sites publish for robots (robots.txt, sitemaps, feeds, llms.txt), plus the homepage. It identifies itself with a descriptive user agent. Respecting robots.txt is on by default. Use the results in line with each site's terms and robots.txt, and don't use them to overload or attack sites. You are responsible for what you do with the discovered URLs.

Made by Steadyline Data.