# Sitemap URL Auditor - Free Beta (`rex-law/sitemap-url-auditor-free-beta`) Actor Discover and audit public XML and GZIP sitemaps with robots.txt, recursion, data-quality, and SSRF safeguards. - **URL**: https://apify.com/rex-law/sitemap-url-auditor-free-beta.md - **Developed by:** [Rex Law](https://apify.com/rex-law) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Sitemap URL Auditor - Free Beta Discover and audit public XML sitemaps without crawling the pages listed inside them. Give the Actor 1-10 HTTPS websites or direct sitemap URLs; it checks `robots.txt`, follows bounded sitemap indexes, safely decompresses GZIP, and writes one structured dataset row per unique HTTPS URL. The Actor is designed for technical SEO, migration checks, publishing QA, and first-party/authorized website audits. It never logs in, bypasses access controls, submits a sitemap, or requests a listed page. ### Input ```json { "startUrls": [ { "url": "https://www.djangoproject.com/" }, { "url": "https://nodejs.org/sitemap.xml" } ], "maxDepth": 2, "maxSitemaps": 20, "maxUrlsPerTarget": 2000, "requestTimeoutSecs": 15, "maxRetries": 2, "maxConcurrency": 2 } ```` Website inputs first read `https://host/robots.txt`. Every `Sitemap:` directive is considered; when none is present, the Actor tries `https://host/sitemap.xml`. Direct sitemap inputs still require the origin's robots policy to allow their path. | Field | Hard boundary | | --- | --- | | `startUrls` | 1-10 HTTPS websites or sitemap URLs; no credentials, IP literals, custom ports, or reserved/internal names | | `maxDepth` | 0-5 nested sitemap-index levels | | `maxSitemaps` | 1-50 fetched sitemap documents per target | | `maxUrlsPerTarget` | 1-10,000 unique output URLs per target | | `requestTimeoutSecs` | 5-30 seconds per request | | `maxRetries` | 0-2 additional attempts for network failures, HTTP 408/425/429, and HTTP 5xx | | `maxConcurrency` | 1-4 separate targets; files inside one target remain sequential | Limits are per target. Reaching a depth, sitemap, or URL boundary is visible as `partial`; the Actor never presents truncated work as complete. ### Dataset rows ```json { "type": "sitemap_url_audit", "urlKey": "https://example.org/|https://example.org/products/widget", "url": "https://example.org/products/widget", "lastmod": "2026-07-18", "lastmodValid": true, "changefreq": "weekly", "changefreqValid": true, "priority": 0.8, "priorityRaw": "0.8", "priorityValid": true, "sourceSitemap": "https://example.org/sitemap-products.xml.gz", "sitemapDepth": 1, "target": "https://example.org/", "targetHost": "example.org", "crossDomain": false, "scrapedAt": "2026-07-18T00:00:00.000Z" } ``` Malformed optional metadata remains auditable: the raw `lastmod`, `changefreq`, or `priorityRaw` is retained while its validity flag is `false`. Invalid or unsafe `loc` values cannot become dataset rows; bounded samples and counts appear in `OUTPUT`. Deduplication occurs within each input target using the normalized URL. Fragments are removed because they are not sent in HTTP requests; query strings remain significant. ### Audit summary The key-value store record `OUTPUT` contains totals and per-target details for: - duplicate URLs and repeated sitemap references; - invalid `lastmod`, `changefreq`, and priority values; - invalid or non-HTTPS `loc` values; - URLs whose hostname differs from the input target; - cross-host child sitemap references, which are recorded and not fetched; - circular sitemap-index references; - robots denials and unsupported crawl delays; - fetch, decompression, XML, depth, count, and size failures. Status meanings: - `success`: all discovered sitemap work completed inside the configured boundaries; - `partial`: useful rows were written, but at least one target/file was denied, failed, malformed, circular, cross-host, or truncated; - `failed`: every target failed or was denied before producing a usable sitemap result. `OUTPUT` is saved, then the Actor run fails. Empty, valid sitemaps are successful. Dataset storage failures are always fatal, never downgraded to an upstream partial result. ### Protocol behavior The implementation follows the [Sitemaps XML protocol](https://www.sitemaps.org/protocol.html): - XML must be UTF-8 and rooted at `` or ``; - `` is required for usable entries; - `lastmod` is checked against W3C date/datetime forms; - `changefreq` accepts `always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly`, or `never`; - priority must be between 0 and 1; - sitemap indexes should reference sitemap files on the same host; - the protocol allows GZIP and sets an outer limit of 50,000 entries and 50 MB uncompressed. This free beta intentionally uses stricter resource limits: 2 MB compressed wire size, 10 MB after decompression, 50 sitemap documents, 10,000 output URLs per target, and five index levels. XML `DOCTYPE` and `ENTITY` declarations are rejected to prevent entity expansion. RSS, Atom, and plain-text sitemap formats are out of scope for this version. Cross-submitted root sitemaps explicitly declared by a site's robots file can be fetched after their own host passes HTTPS, robots, and DNS safety checks. Cross-host children inside a sitemap index are skipped because the protocol requires them to stay on the index host. Cross-domain page URLs are output with `crossDomain: true` but are never fetched. ### Robots and SSRF safety The Actor follows [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) `User-agent`, `Allow`, and `Disallow` matching, including longest-match and `*`/`$` patterns. It identifies itself as `RexSitemapAuditor`. `Sitemap:` is treated as an independent directive. A 404 or 410 robots response means no policy was published; other robots retrieval failures fail closed. Explicit crawl delays up to 30 seconds are honored; larger delays cause the sitemap to be skipped rather than ignored. Every fetched URL must use HTTPS without credentials or a custom port. Before each connection and redirect, all DNS answers are checked; if any answer is private, loopback, link-local, reserved, documentation-only, or otherwise non-global, the request is rejected. One validated address is then pinned into the actual TLS connection while preserving hostname certificate verification. This closes the usual gap where a hostname passes a preflight lookup and changes to a private address during connection. Same-host redirects are limited to three and re-run the checks; cross-host redirects are rejected so they cannot bypass the destination host's robots policy. The Actor fetches only `robots.txt` and sitemap documents. It never resolves or requests the page URLs listed by a sitemap. Robots rules are crawler instructions, not legal permission or proof of ownership. Only audit sites you own or are authorized to assess, and follow the site's terms, applicable law, privacy requirements, and reasonable request rates. ### Local verification Node.js 22 or newer is required. ```bash npm ci npm test npm run check npm run smoke docker build -t sitemap-url-auditor-free-beta . ``` The real-network smoke uses two public sites that explicitly expose crawlable sitemap information: `www.sitemaps.org` and `www.sphinx-doc.org`. It requires no credential and validates at least 50 URL rows across both targets. To run the complete Actor locally, put input in `storage/key_value_stores/default/INPUT.json` and run: ```bash APIFY_LOCAL_STORAGE_DIR=./storage npm start ``` ### Fail-closed FREE release `ops/release.mjs` validates local artifacts, creates or updates the Actor while private, enforces `LIMITED_PERMISSIONS`, rejects non-FREE pricing, builds version `0.1`, runs the two-site cloud smoke, re-checks all gates, publishes, and verifies the final state. It never sends a `pricingInfos` field. A failed gate leaves the Actor private, and an already-public Actor is never overwritten. Safe local validation performs no API call: ```bash npm run release:validate ``` The exact state-changing command is deliberately guarded: ```bash CONFIRM_PUBLISH_FREE_BETA=sitemap-url-auditor-free-beta npm run release ``` Do not run it without explicit publication approval. Building and testing this directory does not call the Apify API or publish anything. # Actor input Schema ## `startUrls` (type: `array`): Public HTTPS website roots or direct sitemap URLs that you own or are authorized to audit. ## `maxDepth` (type: `integer`): Maximum number of nested sitemap-index levels to follow for each target. ## `maxSitemaps` (type: `integer`): Maximum number of sitemap documents to fetch for each input target. ## `maxUrlsPerTarget` (type: `integer`): Maximum number of unique sitemap URL rows to store for each input target. ## `requestTimeoutSecs` (type: `integer`): Per-request timeout in seconds for robots.txt and sitemap documents. ## `maxRetries` (type: `integer`): Number of bounded retries after transient network, rate-limit, or server failures. ## `maxConcurrency` (type: `integer`): Sitemaps within one target remain sequential; only separate targets run in parallel. ## Actor input object example ```json { "startUrls": [ { "url": "https://www.djangoproject.com/" }, { "url": "https://nodejs.org/sitemap.xml" } ], "maxDepth": 2, "maxSitemaps": 20, "maxUrlsPerTarget": 2000, "requestTimeoutSecs": 15, "maxRetries": 2, "maxConcurrency": 2 } ``` # Actor output Schema ## `urls` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("rex-law/sitemap-url-auditor-free-beta").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("rex-law/sitemap-url-auditor-free-beta").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call rex-law/sitemap-url-auditor-free-beta --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=rex-law/sitemap-url-auditor-free-beta", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Sitemap URL Auditor - Free Beta", "description": "Discover and audit public XML and GZIP sitemaps with robots.txt, recursion, data-quality, and SSRF safeguards.", "version": "0.1", "x-build-id": "4tfq28AINjSuhUDmK" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/rex-law~sitemap-url-auditor-free-beta/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-rex-law-sitemap-url-auditor-free-beta", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/rex-law~sitemap-url-auditor-free-beta/runs": { "post": { "operationId": "runs-sync-rex-law-sitemap-url-auditor-free-beta", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/rex-law~sitemap-url-auditor-free-beta/run-sync": { "post": { "operationId": "run-sync-rex-law-sitemap-url-auditor-free-beta", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Websites or sitemap URLs", "minItems": 1, "maxItems": 10, "type": "array", "description": "Public HTTPS website roots or direct sitemap URLs that you own or are authorized to audit.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "title": "HTTPS URL", "description": "A public HTTPS website root or direct sitemap URL that you own or are authorized to audit.", "type": "string", "maxLength": 2048 } } }, "default": [ { "url": "https://www.djangoproject.com/" }, { "url": "https://nodejs.org/sitemap.xml" } ] }, "maxDepth": { "title": "Maximum sitemap-index depth", "minimum": 0, "maximum": 5, "type": "integer", "description": "Maximum number of nested sitemap-index levels to follow for each target.", "default": 2 }, "maxSitemaps": { "title": "Maximum sitemap files per target", "minimum": 1, "maximum": 50, "type": "integer", "description": "Maximum number of sitemap documents to fetch for each input target.", "default": 20 }, "maxUrlsPerTarget": { "title": "Maximum URLs per target", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum number of unique sitemap URL rows to store for each input target.", "default": 2000 }, "requestTimeoutSecs": { "title": "Request timeout", "minimum": 5, "maximum": 30, "type": "integer", "description": "Per-request timeout in seconds for robots.txt and sitemap documents.", "default": 15 }, "maxRetries": { "title": "Transient retries", "minimum": 0, "maximum": 2, "type": "integer", "description": "Number of bounded retries after transient network, rate-limit, or server failures.", "default": 2 }, "maxConcurrency": { "title": "Parallel targets", "minimum": 1, "maximum": 4, "type": "integer", "description": "Sitemaps within one target remain sequential; only separate targets run in parallel.", "default": 2 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```