Sitemap URL Auditor - Free Beta

Discover and audit public XML sitemaps without crawling the pages listed inside them. Give the Actor 1-10 HTTPS websites or direct sitemap URLs; it checks robots.txt, follows bounded sitemap indexes, safely decompresses GZIP, and writes one structured dataset row per unique HTTPS URL.

The Actor is designed for technical SEO, migration checks, publishing QA, and first-party/authorized website audits. It never logs in, bypasses access controls, submits a sitemap, or requests a listed page.

Input

{
  "startUrls": [
    { "url": "https://www.djangoproject.com/" },
    { "url": "https://nodejs.org/sitemap.xml" }
  ],
  "maxDepth": 2,
  "maxSitemaps": 20,
  "maxUrlsPerTarget": 2000,
  "requestTimeoutSecs": 15,
  "maxRetries": 2,
  "maxConcurrency": 2
}

Website inputs first read https://host/robots.txt. Every Sitemap: directive is considered; when none is present, the Actor tries https://host/sitemap.xml. Direct sitemap inputs still require the origin's robots policy to allow their path.

Field	Hard boundary
startUrls	1-10 HTTPS websites or sitemap URLs; no credentials, IP literals, custom ports, or reserved/internal names
maxDepth	0-5 nested sitemap-index levels
maxSitemaps	1-50 fetched sitemap documents per target
maxUrlsPerTarget	1-10,000 unique output URLs per target
requestTimeoutSecs	5-30 seconds per request
maxRetries	0-2 additional attempts for network failures, HTTP 408/425/429, and HTTP 5xx
maxConcurrency	1-4 separate targets; files inside one target remain sequential

Limits are per target. Reaching a depth, sitemap, or URL boundary is visible as partial; the Actor never presents truncated work as complete.

Dataset rows

{
  "type": "sitemap_url_audit",
  "urlKey": "https://example.org/|https://example.org/products/widget",
  "url": "https://example.org/products/widget",
  "lastmod": "2026-07-18",
  "lastmodValid": true,
  "changefreq": "weekly",
  "changefreqValid": true,
  "priority": 0.8,
  "priorityRaw": "0.8",
  "priorityValid": true,
  "sourceSitemap": "https://example.org/sitemap-products.xml.gz",
  "sitemapDepth": 1,
  "target": "https://example.org/",
  "targetHost": "example.org",
  "crossDomain": false,
  "scrapedAt": "2026-07-18T00:00:00.000Z"
}

Malformed optional metadata remains auditable: the raw lastmod, changefreq, or priorityRaw is retained while its validity flag is false. Invalid or unsafe loc values cannot become dataset rows; bounded samples and counts appear in OUTPUT.

Deduplication occurs within each input target using the normalized URL. Fragments are removed because they are not sent in HTTP requests; query strings remain significant.

Audit summary

The key-value store record OUTPUT contains totals and per-target details for:

• duplicate URLs and repeated sitemap references;
• invalid lastmod, changefreq, and priority values;
• invalid or non-HTTPS loc values;
• URLs whose hostname differs from the input target;
• cross-host child sitemap references, which are recorded and not fetched;
• circular sitemap-index references;
• robots denials and unsupported crawl delays;
• fetch, decompression, XML, depth, count, and size failures.

Status meanings:

• success: all discovered sitemap work completed inside the configured boundaries;
• partial: useful rows were written, but at least one target/file was denied, failed, malformed, circular, cross-host, or truncated;
• failed: every target failed or was denied before producing a usable sitemap result. OUTPUT is saved, then the Actor run fails.

Empty, valid sitemaps are successful. Dataset storage failures are always fatal, never downgraded to an upstream partial result.

Protocol behavior

The implementation follows the Sitemaps XML protocol:

• XML must be UTF-8 and rooted at <urlset> or <sitemapindex>;
• <loc> is required for usable entries;
• lastmod is checked against W3C date/datetime forms;
• changefreq accepts always, hourly, daily, weekly, monthly, yearly, or never;
• priority must be between 0 and 1;
• sitemap indexes should reference sitemap files on the same host;
• the protocol allows GZIP and sets an outer limit of 50,000 entries and 50 MB uncompressed.

This free beta intentionally uses stricter resource limits: 2 MB compressed wire size, 10 MB after decompression, 50 sitemap documents, 10,000 output URLs per target, and five index levels. XML DOCTYPE and ENTITY declarations are rejected to prevent entity expansion. RSS, Atom, and plain-text sitemap formats are out of scope for this version.

Cross-submitted root sitemaps explicitly declared by a site's robots file can be fetched after their own host passes HTTPS, robots, and DNS safety checks. Cross-host children inside a sitemap index are skipped because the protocol requires them to stay on the index host. Cross-domain page URLs are output with crossDomain: true but are never fetched.

Robots and SSRF safety

The Actor follows RFC 9309 User-agent, Allow, and Disallow matching, including longest-match and */$ patterns. It identifies itself as RexSitemapAuditor. Sitemap: is treated as an independent directive. A 404 or 410 robots response means no policy was published; other robots retrieval failures fail closed. Explicit crawl delays up to 30 seconds are honored; larger delays cause the sitemap to be skipped rather than ignored.

Every fetched URL must use HTTPS without credentials or a custom port. Before each connection and redirect, all DNS answers are checked; if any answer is private, loopback, link-local, reserved, documentation-only, or otherwise non-global, the request is rejected. One validated address is then pinned into the actual TLS connection while preserving hostname certificate verification. This closes the usual gap where a hostname passes a preflight lookup and changes to a private address during connection. Same-host redirects are limited to three and re-run the checks; cross-host redirects are rejected so they cannot bypass the destination host's robots policy.

The Actor fetches only robots.txt and sitemap documents. It never resolves or requests the page URLs listed by a sitemap.

Robots rules are crawler instructions, not legal permission or proof of ownership. Only audit sites you own or are authorized to assess, and follow the site's terms, applicable law, privacy requirements, and reasonable request rates.

Local verification

Node.js 22 or newer is required.

npm ci
npm test
npm run check
npm run smoke
docker build -t sitemap-url-auditor-free-beta .

The real-network smoke uses two public sites that explicitly expose crawlable sitemap information: www.sitemaps.org and www.sphinx-doc.org. It requires no credential and validates at least 50 URL rows across both targets.

To run the complete Actor locally, put input in storage/key_value_stores/default/INPUT.json and run:

$APIFY_LOCAL_STORAGE_DIR=./storage npm start

Fail-closed FREE release

ops/release.mjs validates local artifacts, creates or updates the Actor while private, enforces LIMITED_PERMISSIONS, rejects non-FREE pricing, builds version 0.1, runs the two-site cloud smoke, re-checks all gates, publishes, and verifies the final state. It never sends a pricingInfos field. A failed gate leaves the Actor private, and an already-public Actor is never overwritten.

Safe local validation performs no API call:

$npm run release:validate

The exact state-changing command is deliberately guarded:

$CONFIRM_PUBLISH_FREE_BETA=sitemap-url-auditor-free-beta npm run release

Do not run it without explicit publication approval. Building and testing this directory does not call the Apify API or publish anything.