Sitemap & Robots.txt Checker — URLs, Errors, Rules · $0.01/site

Check a website's robots.txt and XML sitemaps in one run. Lists blocked paths and sitemap files, counts URLs, finds duplicates, off-host and non-HTTPS entries, missing lastmod and broken listed pages, and returns a ranked list of issues. Clean JSON for SEO audits. Failed runs are not charged.

Audit a website's robots.txt and XML sitemaps in one call: user-agent groups and blocked paths, sitemap declarations, whether each sitemap loads, URL counts (indexes followed to 10 files), lastmod coverage and age, duplicates, off-host and non-HTTPS URLs, size-limit breaches, and a sample of listed URLs checked for status. Returns a list of concrete issues.

What you get

Structured JSON for each run, ready for spreadsheets, alerts and AI agents.

Input

Field	Type	Default	Description
site	string		Domain or URL of the site, for example example.com or https://example.com/blog.
sampleUrls	integer	5	How many listed URLs to fetch and check for status (0 to skip).

Example input

{
  "site": "https://www.sitemaps.org",
  "sampleUrls": 3
}

Example output (sample run 2026-10-02)

{
  "site": "https://www.sitemaps.org",
  "robots": {
    "found": true,
    "bytes": 45,
    "groups": 0,
    "userAgents": [],
    "disallowRules": 0,
    "allowRules": 0,
    "crawlDelay": null,
    "sitemapDirectives": [
      "https://www.sitemaps.org/sitemap.xml"
    ],
    "blocksEverything": false
  },
  "sitemaps": [
    {
      "url": "https://www.sitemaps.org/sitemap.xml",
      "ok": true,
      "type": "urlset",
      "urls": 84,
      "bytes": 8622,
      "truncatedDownload": false
    }
  ],
  "stats": {
    "urlsRead": 84,
    "withLastmod": 84,
    "duplicates": 0,
    "offHost": 0,
    "nonHttps": 0,
    "newestLastmod": "2022-12-15",
    "oldestLastmod": "2016-11-21"
  },
  "sampleChecks": [
    {
      "url": "https://www.sitemaps.org/",
      "status": 200,
      "redirectedTo": null
    },
    {
      "url": "https://www.sitemaps.org/it/",
      "status": 200,
      "redirectedTo": null
    },
    {
      "url": "https://www.sitemaps.org/ro/",
      "status": 200,
      "redirectedTo": null
    }
  ],
  "issues": [],
  "summary": {
    "errors": 0,
    "warnings": 0
  },
  "checkedAt": "2026-10-02T23:02:47.122Z"
}

Pricing

Event	Price
audit (each successful result)	$0.01

Failed runs are not charged. Free trial credits from Apify cover your first runs.

FAQ

Is a result guaranteed to be complete? Each result carries the date it was read. If the upstream service is down the run fails with a clear message and no per-result charge is made.

Can I call it from code or an AI agent? Yes: run it through the Apify API, schedule it, or use it from MCP clients that support Apify Actors.

Is this affiliated with the data source? No. Source names are used descriptively; each result keeps its source attribution and licence line.

Where do I report a problem? Open an issue on this Actor's page; replies are written by the Ambolt service team, usually within a day.

Review

If this Actor saved you time, a short review on its Store page helps others find it and tells us what to improve.

Keywords

sitemap, robots.txt, SEO audit, crawl, sitemap validator, sitemap index, technical SEO, lastmod

Notes

Data comes from the source named in each result and keeps its licence and attribution. Informational only, not financial, legal or tax advice. Open an issue on the Actor page for questions; replies come under the product name.