Sitemap & Robots.txt Explorer with AI Crawlers avatar

Sitemap & Robots.txt Explorer with AI Crawlers

Pricing

from $0.50 / 1,000 urls

Go to Apify Store
Sitemap & Robots.txt Explorer with AI Crawlers

Sitemap & Robots.txt Explorer with AI Crawlers

List every URL in a website's sitemaps with last changed dates, and see which AI crawlers its robots.txt blocks. Works with many sites at once. No login needed.

Pricing

from $0.50 / 1,000 urls

Rating

0.0

(0)

Developer

Open Data Desk

Open Data Desk

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

2 days ago

Last modified

Features

Limited permissions

Share

Sitemap & Robots.txt Explorer

List every page in a website's sitemaps, with last changed dates, and get a plain report on its robots.txt, including which AI crawlers are blocked. Works with several websites at once. No login and no proxy needed.

What does Sitemap & Robots.txt Explorer do?

For each website you give it, the Actor:

  1. Reads the robots.txt file and finds the sitemaps the site declares.
  2. Reads those sitemaps, including sitemap index files that point to other sitemaps, and compressed .xml.gz files.
  3. Returns one row per page URL, with the date it last changed when the sitemap has one.
  4. Builds a report about the robots.txt, so you can see at a glance how the site treats crawlers.

It is useful for:

  • SEO checks and site audits.
  • Making a clean list of URLs before you run a bigger crawl.
  • Finding pages that changed recently.
  • Checking whether a site blocks AI crawlers such as GPTBot or ClaudeBot.
  • Comparing the language versions of pages, because hreflang links from the sitemap are included.

What data do you get?

URLs (dataset)

FieldWhat it is
siteThe website
urlThe page URL
lastmodThe date the sitemap says the page changed, or null if the sitemap has none
changefreq, priorityAs stated in the sitemap, or null
sitemapUrlThe sitemap file the URL came from
alternatesOther language or region versions of the page (hreflang), if listed
imageCountHow many images the sitemap lists for the page

Example row

{
"site": "nytimes.com",
"url": "https://www.nytimes.com/2026/11/16/world/europe/pope-leo-peru-health.html",
"lastmod": "2026-11-16T12:43:12.000Z",
"changefreq": null,
"priority": null,
"sitemapUrl": "https://www.nytimes.com/sitemaps/new/news.xml.gz",
"alternates": [],
"imageCount": 1
}

Robots.txt and AI crawler report (key-value store, key ROBOTS_REPORT)

One entry per website with:

  • Whether a robots.txt exists, and the sitemaps it declares.
  • Which sitemap files were read, and any problems found, such as a file that could not be loaded.
  • The crawl delay, if the site sets one.
  • Whether the site has an llms.txt file.
  • For 15 well known AI crawlers (for example GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot): blocked, partly blocked or allowed, and whether the rule is written for that crawler or comes from the general rules for everyone.

The list of crawlers is based on the names that the companies publish, and names do change over time. The report reflects what robots.txt says. It does not prove that a crawler follows it.

How to use it

  1. Open the Input tab and add websites. A domain such as example.com is enough. You can also paste a direct link to a sitemap file.
  2. Optional: limit the number of URLs, keep only URLs that match a pattern, or only URLs changed after a date.
  3. Click Start and download the URLs as JSON, CSV or Excel. Open the key-value store to see the report.

Input options

  • Websites: domains or links, one per line.
  • Find sitemaps automatically: reads robots.txt, then tries /sitemap.xml, /sitemap_index.xml and a few other usual places.
  • Include robots.txt and AI crawler report: on by default. It is billed once per website.
  • Max URLs per website and Max URLs in total: set a ceiling for large sites.
  • Only URLs matching and Skip URLs matching: regular expressions, for example /blog/ or \.pdf$.
  • Only URLs changed after and Keep URLs without a date: Many sitemaps do not list dates, so choose what should happen to those.
  • Proxy: optional. Most sites do not need it for sitemaps.

How much does it cost?

You pay for each URL that is returned, and once per website for the report. URLs that are filtered out are free. The prices are shown at the top of this page, and you can set a maximum cost for a run.

Tips

  • Run it with a small limit first, to see what a site's sitemap looks like.
  • Very large sites can have hundreds of sitemap files. Use the patterns or the date filter to keep the result focused.
  • If a site has no sitemap, the report says so in the problems list.
  • This Actor reads sitemap and robots.txt files only. It does not visit the pages, so it does not tell you whether a page still works.

Sitemaps and robots.txt files are public files that websites publish so that machines can read them. This Actor reads only those files, one request at a time per website. You are responsible for how you use the URLs you collect, including the terms of the websites you plan to crawl next.

Issues and requests

If a site's sitemap is not read correctly, please open an issue on the Issues tab with the website address.