Sitemap URLs — unique listed URLs, dates and robots rules

Give it domains or sitemap URLs; it finds the sitemaps (robots.txt, /sitemap.xml, /sitemap_index.xml), follows sitemap indexes, reads gzip, text and RSS/Atom sitemaps, and returns every listed URL once with lastmod, changefreq and priority, plus a free summary of the site's robots.txt rules.

Development validation produced actual successful public-input records using the production Dockerfile and SDK entrypoint on GitHub-hosted Linux. The smoke input, all three tasks and every defined benchmark passed their exact expectations. The independent review completed on 2026-10-19; its findings are tracked in the committed fix report. Platform costs, settlement, publication and daily tests remain unverified.

Call it from an agent

Through Apify's MCP server (https://mcp.apify.com) use call-actor:

{
  "actor": "hermeskd/sitemap-urls",
  "input": {
    "urls": [
      "https://docs.apify.com/"
    ],
    "maxUrls": 50
  },
  "callOptions": {
    "maxTotalChargeUsd": 0.05,
    "timeout": 1800
  },
  "waitSecs": 30
}

Then call get-dataset-items with the returned defaultDatasetId. Dedicated tool URL: https://mcp.apify.com?tools=hermeskd/sitemap-urls

Input

Field	Type	Meaning
urls	array	Domains or absolute sitemap URLs; bare domains use HTTPS.
maxUrls	integer	Maximum unique matching URL records across the run.
filter	string	Optional regex; at most 512 characters and a 20 ms match timeout.
includeRobots	boolean	One free all-group robots summary per host; independent of HermesBot policy enforcement.

Output

Field	Meaning
type	Listed URL or free robots summary.
url	Listed URL, robots URL, or original failed input.
ok	True for returned results; errors never charged.
lastmod	Publisher-provided modification/publication value.
changefreq	Publisher change frequency.
priority	Publisher priority between zero and one.
sitemap	Source sitemap final URL.
robots	Free all-group summary, separate from the HermesBot access decision.
error	Failure, always uncharged.

Actual production-container record from 2026-10-18T20:27:13.734929+00:00 (code f063187693578dd7551fa2bec552281e2f0f3e33, smoke input, SDK local storage). The saved record is unchanged; local event logs do not prove Apify settlement.

{
  "type": "url",
  "url": "https://docs.apify.com/api",
  "ok": true,
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemap": "https://docs.apify.com/sitemap_base.xml"
}

Events and prices

Event	Title	Charged when	FREE / BRONZE	SILVER	GOLD / PLATINUM / DIAMOND
url-listed	URL listed	One unique matching URL written from a sitemap.	$0.0003	$0.00027	$0.00024

Platform usage is not passed through. Apify's automatic actor-start event remains $0.00005 per started GB (one GB or less counts once). The synthetic dataset-item event is absent. Charge limits stop further paying work; unprocessed inputs receive uncharged errors. Completed readable pending events are settled on abort.

Worked cost example: ten successful primary events cost $0.003 at list, plus actor start. Failures are not included in that count.

Limits and run time

1–20 domains or sitemap URLs; maxUrls 1–200,000 across the run, after filtering and deduplication. XML/index, gzip, UTF-8 text, RSS and Atom are supported. Every sitemap is bounded to 50 MiB expanded and 50,000 entries; traversal to 10,000 documents and depth 50, with cycles deduplicated. Regex filters are at most 512 characters with a 20 ms match timeout. Robots summaries are free and do not mask failed sitemap outcomes. Protocol extensions do not replace protocol namespace fields.

Default run timeout is 1800 seconds. Typical cold platform time, costs and the smallest passing memory have not been measured for this new implementation. Memory probes are evidence to collect later; configured memories are specification defaults, not measured claims.

Errors

Code	Meaning
no_sitemap	No readable sitemap URL result was found for the input.
download_failed	The request or connection failed within its bounded deadline.
http_error	The source answered an unsuccessful HTTP status.
robots_disallowed	robots.txt forbids fetching the URL; no request was made to it.
blocked_host	The host is on the deny-list or resolves to a private or local address; no request was made.
too_large	A documented response, expansion or traversal cap was exceeded.
source_schema_changed	A source or tool response no longer has the verified protocol shape.
timeout	The input did not finish within the run's time budget.
invalid_input	The run input passed the input schema but cannot be used; the message names the field.
internal_error	Unexpected failure; the message carries the error class.
charge_limit_reached	The run reached its maximum total charge; this input was not processed.

No failure record is charged. A run fails only when all input outcomes are Actor-side failures; caller errors and a mixture of caller/Actor failures do not fail the run.

Sources and terms

Only process content you have the right to process. URL fetches obey HermesBot robots-group selection, the shared deny-list, public-address checks on connections and redirects, honest identity, deadlines and rate limits. No caller credentials, cookies, CAPTCHA solving, residential proxies or evasion. Public robots files and sitemap documents are fetched; listed pages are not downloaded. The bounded parser supports XML urlsets/indexes, gzip, UTF-8 text, RSS and Atom. DTDs and entities are refused. The public documentation sitemap source was checked directly; its Apache-2.0 source repository and observed document counts are recorded with the manifests. The 2026-10-18 GitHub-hosted run completed actual production-container traversal, with raw records and provenance shown above. Cloud resolver access remains unavailable; this does not negate the separate hosted evidence.

Not affiliated

This Actor is not affiliated with Google, Apify, IANA, Verisign or other named protocol/source providers. Third-party names describe compatibility and source material.

Related Hermes Actors

website-screenshot, lighthouse-audit, seo-page-audit, domain-availability, dns-and-email-auth, broken-links and sitemap-urls.

Changelog

2026-10-17 (historical): initial implementation and fixture contracts; packaging and public production validation were incomplete on that date.

2026-10-18: all six production images, public smoke inputs, three tasks per Actor and defined benchmarks passed on GitHub-hosted Linux; exact code, records and image provenance are retained with the examples. These local SDK runs do not prove platform costs, settlement, publication, minimum memory, daily tests or payouts.

2026-10-19: independent-review repairs cover DNS privacy, malformed publisher URLs, empty sitemap failures, responsive assets and schema prefill, as applicable to this Actor. See the committed unit 4 fix report for measured verification and remaining platform NO-COVERAGE.