Sitemap & Robots.txt Explorer with AI Crawlers
Pricing
from $0.50 / 1,000 urls
Sitemap & Robots.txt Explorer with AI Crawlers
List every URL in a website's sitemaps with last changed dates, and see which AI crawlers its robots.txt blocks. Works with many sites at once. No login needed.
Pricing
from $0.50 / 1,000 urls
Rating
0.0
(0)
Developer
Open Data Desk
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
2 days ago
Last modified
Categories
Features
Share
Sitemap & Robots.txt Explorer
List every page in a website's sitemaps, with last changed dates, and get a plain report on its robots.txt, including which AI crawlers are blocked. Works with several websites at once. No login and no proxy needed.
What does Sitemap & Robots.txt Explorer do?
For each website you give it, the Actor:
- Reads the robots.txt file and finds the sitemaps the site declares.
- Reads those sitemaps, including sitemap index files that point to other sitemaps, and compressed
.xml.gzfiles. - Returns one row per page URL, with the date it last changed when the sitemap has one.
- Builds a report about the robots.txt, so you can see at a glance how the site treats crawlers.
It is useful for:
- SEO checks and site audits.
- Making a clean list of URLs before you run a bigger crawl.
- Finding pages that changed recently.
- Checking whether a site blocks AI crawlers such as GPTBot or ClaudeBot.
- Comparing the language versions of pages, because hreflang links from the sitemap are included.
What data do you get?
URLs (dataset)
| Field | What it is |
|---|---|
site | The website |
url | The page URL |
lastmod | The date the sitemap says the page changed, or null if the sitemap has none |
changefreq, priority | As stated in the sitemap, or null |
sitemapUrl | The sitemap file the URL came from |
alternates | Other language or region versions of the page (hreflang), if listed |
imageCount | How many images the sitemap lists for the page |
Example row
{"site": "nytimes.com","url": "https://www.nytimes.com/2026/11/16/world/europe/pope-leo-peru-health.html","lastmod": "2026-11-16T12:43:12.000Z","changefreq": null,"priority": null,"sitemapUrl": "https://www.nytimes.com/sitemaps/new/news.xml.gz","alternates": [],"imageCount": 1}
Robots.txt and AI crawler report (key-value store, key ROBOTS_REPORT)
One entry per website with:
- Whether a robots.txt exists, and the sitemaps it declares.
- Which sitemap files were read, and any problems found, such as a file that could not be loaded.
- The crawl delay, if the site sets one.
- Whether the site has an
llms.txtfile. - For 15 well known AI crawlers (for example GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot):
blocked,partly blockedorallowed, and whether the rule is written for that crawler or comes from the general rules for everyone.
The list of crawlers is based on the names that the companies publish, and names do change over time. The report reflects what robots.txt says. It does not prove that a crawler follows it.
How to use it
- Open the Input tab and add websites. A domain such as
example.comis enough. You can also paste a direct link to a sitemap file. - Optional: limit the number of URLs, keep only URLs that match a pattern, or only URLs changed after a date.
- Click Start and download the URLs as JSON, CSV or Excel. Open the key-value store to see the report.
Input options
- Websites: domains or links, one per line.
- Find sitemaps automatically: reads robots.txt, then tries
/sitemap.xml,/sitemap_index.xmland a few other usual places. - Include robots.txt and AI crawler report: on by default. It is billed once per website.
- Max URLs per website and Max URLs in total: set a ceiling for large sites.
- Only URLs matching and Skip URLs matching: regular expressions, for example
/blog/or\.pdf$. - Only URLs changed after and Keep URLs without a date: Many sitemaps do not list dates, so choose what should happen to those.
- Proxy: optional. Most sites do not need it for sitemaps.
How much does it cost?
You pay for each URL that is returned, and once per website for the report. URLs that are filtered out are free. The prices are shown at the top of this page, and you can set a maximum cost for a run.
Tips
- Run it with a small limit first, to see what a site's sitemap looks like.
- Very large sites can have hundreds of sitemap files. Use the patterns or the date filter to keep the result focused.
- If a site has no sitemap, the report says so in the problems list.
- This Actor reads sitemap and robots.txt files only. It does not visit the pages, so it does not tell you whether a page still works.
Is it legal to use?
Sitemaps and robots.txt files are public files that websites publish so that machines can read them. This Actor reads only those files, one request at a time per website. You are responsible for how you use the URLs you collect, including the terms of the websites you plan to crawl next.
Issues and requests
If a site's sitemap is not read correctly, please open an issue on the Issues tab with the website address.