Sitemap & Robots.txt Analyzer

Audit any website's robots.txt and sitemaps in seconds: which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked, how many URLs are listed, how fresh they are, and which technical SEO issues to fix.

What does Sitemap & Robots.txt Analyzer do?

• AI crawler access report for 16 bots
• Recursive sitemap index parsing with gzip support
• Freshness stats from lastmod and top site sections
• Detects sitemap URLs blocked by robots.txt and missing sitemap declarations
• Checks for llms.txt

How to use it

1. Click Try for free and open the Input tab.
2. Fill in the fields below, or keep the defaults for a quick test.
3. Press Start. Results appear in the Output tab within seconds to a few minutes.
4. Download as JSON, CSV or Excel, or call the actor from the API, a schedule, Make, Zapier or an AI agent through the Apify MCP server.

Input

Field	Type	Description
websites	array	Domains or URLs, one per line.
maxSitemapFiles	integer	How many sitemap files to fetch per site.
maxUrlsToCheckAgainstRobots	integer	Sample size for robots conflicts.

Example input

{
  "websites": [
    "apify.com",
    "nytimes.com"
  ],
  "maxSitemapFiles": 20,
  "maxUrlsToCheckAgainstRobots": 5000
}

Output

Each result is a flat JSON object that you can open in the Output tab, export to CSV or Excel, or read through the API. See the dataset views for the main columns.

How much does it cost to use Sitemap & Robots.txt Analyzer?

Pricing is pay per event: $0.01 per website. You pay only for results saved to the dataset, plus nothing extra for platform usage on the free plan limits. Apify's free monthly credit lets you test it before paying.

FAQ

Does it crawl the pages? No. It only reads robots.txt, llms.txt and sitemap files.

What counts as blocked? A bot is blocked if robots.txt disallows the site root for that user agent.

Can I run it for many domains? Yes. Add as many as you like; each domain is one billable result.

Is it legal?

The actor reads public, open-licence or government data only and does not collect private information. You are responsible for using the results in line with the source's terms and applicable laws.

Use it with AI agents (MCP)

This actor is agent-ready: it is a plain tool with a documented JSON input and a flat JSON output, so Claude, ChatGPT, Cursor and other agents can call it through the Apify MCP server.

Add it to an MCP client with this URL (Apify account or agentic payment required):

https://mcp.apify.com/?tools=flyingnimbus/sitemap-robots-analyzer

Example prompt: "Use the Sitemap & Robots.txt Analyzer tool with the example input below and summarise the results."

Calls are pay per event ($0.01 per website), so an agent can estimate the cost before it runs. Empty or failed lookups return a status record instead of failing the run.

More actors by flyingnimbus

• Property Details by Address API
• Schools Near Address Finder API
• School Rating Converter & Equivalence API
• Building Permits Multi-City API
• Company Registry Lookup (LEI + SEC)
• Restaurant Inspections API - NYC, Chicago, Toronto
• CVE Risk Enricher - NVD, CISA KEV & EPSS
• SEC Insider Trades (Form 4) - Buys & Sells
• SEC 13F Holdings Tracker
• EU Company Registry Search
• Canada Recalls & Safety Alerts Monitor
• Email Validator - MX, SPF & DMARC
• Website Tech Stack Detector
• PDF Text & Table Extractor
• URL to Markdown + Metadata
• SSL Certificate Expiry Checker
• VIN Decoder + Recalls API
• ClinicalTrials.gov Search API
• US Federal Contracts & Grants (USAspending)
• Federal Register Search API
• FDA Recalls & Drug Labels API
• Eurostat Data API
• UK Government Tenders (Contracts Finder)

Feedback

Found a bug or need another field or city? Open an issue on the actor's Issues tab and it will be looked at quickly.