Site Discovery: sitemaps, robots.txt, RSS & llms.txt
Pricing
from $2.00 / 1,000 domain summaries
Map any website in seconds without crawling: robots.txt rules for your user agent (incl. AI-bot blocks), every sitemap and sitemap index (.gz too), RSS/Atom/JSON feeds, llms.txt and llms-full.txt, plus a filtered URL list with lastmod. By Steadyline Data.
Domains (example.com) or URLs. Each origin is analyzed once.
Write one dataset row per discovered URL (charged per URL). Off = domain summaries only.
Stop collecting URLs for a domain after this many (after filters).
Only keep URLs whose path matches one of these. Glob on path (/blog/**, /docs/*/intro) or regex with re: prefix (re:^/p/\d+).
[]Drop URLs whose path matches (same syntax).
[]Only keep URLs with a sitemap lastmod on/after this date. URLs without lastmod are dropped when set.
Recurse into sitemap index files (including gzipped .xml.gz).
Safety cap on sitemap files fetched per domain.
From on the homepage, else common paths (/feed, /rss.xml, /atom.xml...).
Check /llms.txt and /llms-full.txt and include an excerpt.
Characters of llms.txt content to include.
User-agent token whose robots.txt rules are evaluated (e.g. GPTBot, Googlebot, MyAgent).
Drop URLs disallowed for the user agent above (counted in urlCounts.disallowedByRobots). Off = output them with allowedByRobots=false.