Sitemap Indexability & AI Crawler Audit API

Audit sitemaps, robots.txt, HTTP status, canonical tags, noindex directives, and AI crawler access for domains or URL lists. The Actor returns structured JSON rows in the default dataset, ready for SEO workflows, site migrations, content operations, and AI-agent checks.

What it checks

• Discovers robots.txt and sitemap URLs from a domain.
• Parses sitemap indexes and nested sitemaps.
• Audits sitemap URLs for HTTP status, redirects, final URL, and content type.
• Parses HTML for meta robots, noindex, canonical URL, and page title.
• Evaluates robots.txt access for *, Googlebot, and configurable AI crawler user agents such as GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, and CCBot.
• Returns one siteSummary row per site and one url row per audited URL.

Common use cases

• Technical SEO audits before site launches and migrations.
• Detecting sitemap URLs that redirect, 404, or are noindexed.
• Checking whether important pages are blocked by robots.txt.
• Monitoring whether AI crawlers can access public documentation or content hubs.
• Feeding clean crawlability data into AI agents and MCP workflows.

Input example

{
  "domains": ["example.com"],
  "maxUrlsPerSite": 100,
  "checkHttpStatus": true,
  "checkIndexability": true,
  "checkRobotsTxt": true,
  "checkAiCrawlerAccess": true,
  "requestsPerSecond": 2
}

Output example

{
  "recordType": "url",
  "domain": "example.com",
  "siteUrl": "https://example.com",
  "url": "https://example.com/product/widget",
  "sourceType": "sitemap",
  "sitemapUrl": "https://example.com/sitemap.xml",
  "httpStatus": 200,
  "finalUrl": "https://example.com/product/widget",
  "robotsAllowed": true,
  "googlebotAllowed": true,
  "aiCrawlerAllowed": false,
  "aiCrawlerAccess": {
    "GPTBot": false,
    "ClaudeBot": true
  },
  "metaRobots": "index,follow",
  "noindex": false,
  "canonicalUrl": "https://example.com/product/widget",
  "canonicalMatches": true,
  "indexable": true
}

Pricing

Launch pricing is prepared for pay-per-event monetization:

• USD 0.01 per Actor start.
• USD 0.003 per audited URL or site summary dataset item.

This keeps small checks inexpensive while matching the recurring value of technical SEO and AI crawler monitoring.

Responsible usage

Use this Actor for websites you own, manage, or are allowed to audit. It reads public robots.txt, sitemap, and page metadata, and includes rate limiting so runs can be kept conservative.