Sitemap Extractor — Unique URLs From XML, Gzip & Index Sitemaps

Sitemap extractor discovers website sitemaps through robots and conventional paths, follows indexes and reads XML, gzip, text and RSS or Atom maps, returning deduplicated URLs with lastmod, changefreq and priority. AI agents can call and pay through Apify MCP. List event rates are $0.0003 per URL listed; start is extra, usage is included and failed results are uncharged.

Example result

Example record for the minimal input under How to use it, as this Actor returned it on 2026-10-18 in a run of its production image; the record is kept unchanged with the release.

{
  "type": "url",
  "url": "https://docs.apify.com/api",
  "ok": true,
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemap": "https://docs.apify.com/sitemap_base.xml"
}

How to use it

Start a run in Apify Console, by API or from an agent with an input like this one — it is the input Apify runs every day to test this Actor:

{"urls": ["https://docs.apify.com/"], "maxUrls": 50}

Start with the minimal JSON example above, then adjust the fields in the table below. Read the Example result and Output fields sections for the returned record, Errors for failure codes, and Events and prices for result events and the separate platform start.

Field	Type	Meaning
urls	array	Domains or absolute sitemap URLs; bare domains use HTTPS.
maxUrls	integer	Maximum unique matching URL records across the run.
filter	string	Optional regex; at most 512 characters and a 20 ms match timeout.
includeRobots	boolean	One free all-group robots summary per host; independent of HermesBot policy enforcement.

Events and prices

Event	Title	Charged when	FREE / BRONZE	SILVER	GOLD / PLATINUM / DIAMOND
url-listed	URL listed	One unique matching URL written from a sitemap.	$0.0003	$0.00027	$0.00024

Platform usage is not passed through. Apify's automatic actor-start event remains $0.00005 per started GB (one GB or less counts once). The synthetic dataset-item event is absent. Charge limits stop further paying work; unprocessed inputs receive uncharged errors. Completed readable pending events are settled on abort.

Worked cost example: ten successful primary events cost $0.003 at list, plus actor start. Failures are not included in that count.

Call it from an agent

Through Apify's MCP server (https://mcp.apify.com) use call-actor:

{
  "actor": "hermeskd/sitemap-urls",
  "input": {
    "urls": [
      "https://docs.apify.com/"
    ],
    "maxUrls": 50
  },
  "callOptions": {
    "maxTotalChargeUsd": 0.05,
    "timeout": 1800
  },
  "waitSecs": 30
}

Then call get-dataset-items with the returned defaultDatasetId. Dedicated tool URL: https://mcp.apify.com?tools=hermeskd/sitemap-urls

Output fields

Field	Meaning
type	Listed URL or free robots summary.
url	Listed URL, robots URL, or original failed input.
ok	True for returned results; errors never charged.
lastmod	Publisher-provided modification/publication value.
changefreq	Publisher change frequency.
priority	Publisher priority between zero and one.
sitemap	Source sitemap final URL.
robots	Free all-group summary, separate from the HermesBot access decision.
error	Failure, always uncharged.

Limits and run time

1–20 domains or sitemap URLs; maxUrls 1–200,000 across the run, after filtering and deduplication. XML/index, gzip, UTF-8 text, RSS and Atom are supported. Every sitemap is bounded to 50 MiB expanded and 50,000 entries; traversal to 10,000 documents and depth 50, with cycles deduplicated. Regex filters are at most 512 characters with a 20 ms match timeout. Robots summaries are free and do not mask failed sitemap outcomes. Protocol extensions do not replace protocol namespace fields.

The default run timeout is 1800 seconds. Typical run time on Apify, measured by its deploy of 2026-10-20: the minimal input under How to use it took 7–8 seconds a run at the default 1,024 MB (three runs, peak memory 61 MiB). The same input also ran and passed the pricing floor at 512 MB and 4,096 MB (the deploy's memory probes).

Errors

Code	Meaning
no_sitemap	No readable sitemap URL result was found for the input.
download_failed	The request or connection failed within its bounded deadline.
http_error	The source answered an unsuccessful HTTP status.
robots_disallowed	robots.txt forbids fetching the URL; no request was made to it.
blocked_host	The host is on the deny-list or resolves to a private or local address; no request was made.
too_large	A documented response, expansion or traversal cap was exceeded.
source_schema_changed	A source or tool response no longer has the verified protocol shape.
timeout	The input did not finish within the run's time budget.
invalid_input	The run input passed the input schema but cannot be used; the message names the field.
internal_error	Unexpected failure; the message carries the error class.
charge_limit_reached	The run reached its maximum total charge; this input was not processed.

No failure record is charged. A run fails only when all input outcomes are Actor-side failures; caller errors and a mixture of caller/Actor failures do not fail the run.

Sources and terms

Only process content you have the right to process. URL fetches obey HermesBot robots-group selection, the shared deny-list, public-address checks on connections and redirects, honest identity, deadlines and rate limits. No caller credentials, cookies, CAPTCHA solving, residential proxies or evasion. Public robots files and sitemap documents are fetched; listed pages are not downloaded. The bounded parser supports XML urlsets/indexes, gzip, UTF-8 text, RSS and Atom. DTDs and entities are refused. The public documentation sitemap source was checked directly; its Apache-2.0 source repository and observed document counts are recorded with the manifests. The 2026-10-18 GitHub-hosted run completed actual production-container traversal, with raw records and provenance shown above.

Not affiliated

This Actor is not affiliated with Google, Apify, IANA, Verisign or other named protocol/source providers. Third-party names describe compatibility and source material.

Related Hermes Actors

• AI Crawler Checker: Which AI Bots robots.txt Allows per URL — check declared robots.txt permissions for six AI crawlers at URL paths.
• Article Extractor — News & Blog Text, Date and Markdown by URL — an article's clean text, title and publish date, without the rest of the page.
• ATS Job Search — Jobs Confirmed Open on Company Career Boards — search open jobs in the Hermes directory of Greenhouse and Lever boards, each confirmed live.
• Speech to Text — Transcribe Audio & Video URLs, Speaker Labels — speech to text with SRT and VTT subtitles.
• Video Converter — MP4 to MP3, Trim, GIF and Audio Extraction — audio and video conversion: MP4 to MP3, trim, GIF.
• Broken Links Checker — Links, Assets, Redirects and Their Pages — broken links, 404s and redirects on a site.
• ATS Detector — Company Hiring System & Job Board Lookup — which hiring system and job board a company uses.
• SEC EDGAR Filings — Recent Company Forms, Dates and XBRL Flags — Read recent company financial filing metadata from the official SEC JSON API
• DOI Lookup — Crossref Titles, Venues, Dates and Citation Counts — crossref doi metadata.
• CVE Vulnerability Lookup — NVD CVSS Scores and Exploited Flags — look up NVD CVE dates, status, numeric CVSS scores and severity.
• Merge JSON Datasets — Deduplicate Rows, Export JSON and CSV — json dataset cleaner.
• DNS Lookup — Records, SPF Lookup Counts, DKIM & DMARC by Domain — DNS records and SPF, DKIM and DMARC checks.
• DOCX to Markdown — Word, Excel, PowerPoint and EPUB Converter — Word, Excel, PowerPoint and other office files to Markdown.
• Domain Availability Checker — Bulk RDAP Status and Registrar — whether domain names are available, from registry RDAP.
• USGS Earthquake Lookup — Event Magnitude, Depth, Coordinates — Read numeric USGS earthquake metadata for known event identifiers.
• Currency Exchange Rates — ECB Daily Euro Rates and Cross Pairs — Read latest ECB daily currency reference rates and calculated cross pairs.
• Bulk File Downloader — Save URL Lists to Storage With SHA-256 — files and images from URLs, saved with a public link each.
• GitHub Repo Stats — Stars, Forks, Subscribers and Open Issues — Read numeric metrics for known public GitHub repositories.
• Greenhouse Jobs — Live Openings, Offices and Apply Links — every open job on a company's Greenhouse board.
• Hacker News Story Metrics — Scores, Comment Counts, Timestamps — read numeric Hacker News story scores, timestamps and comment counts by ID.
• HTTP GET Request — Bulk Response Bodies, Status and SHA-256 — public http get.
• Security Headers Checker — HSTS, CSP, Framing, Nosniff by URL — Check seven declared HTTP security header families without returning raw values.
• Image Background Remover — U2NetP Cutouts for JPEG, PNG, WebP — Remove image backgrounds with the small U2NetP model and return transparent PNG pixels as base64
• Image Converter — JPEG, PNG and WebP With Resize and Quality — image format converter.
• Image Metadata Remover — EXIF and GPS Scrub, Pixels Verified — GPS location, camera serial numbers and other private metadata removed from photos, losslessly.
• Image to Text OCR — Text With Word Positions and Confidence — OCR: the text in images.
• Image Upscaler — Real-ESRGAN 2x/4x for Photos and Graphics — image upscaling 2x or 4x, and photo restoration.
• Lever Jobs Scraper — Job Board Postings With Salary, US & EU — every published job on a company's Lever site, global or EU.
• Lighthouse Audit — Lab Web Vitals and Scores, Mobile & Desktop — Lighthouse scores and Core Web Vitals.
• npm Package Lookup — Latest Version, License, Dependency Count — read latest npm package versions, license tokens and dependency counts.
• Website Change Monitor — Page Text Diffs Against Your Snapshots — what changed on a page since the last check.
• PDF to Markdown — PDF Parser & Text Extractor With OCR — PDF to Markdown and text, with OCR for scanned pages.
• Public Holidays Calendar — Holiday Dates for Five Countries — build offline public-holiday calendars for five countries and bounded years.
• Remote Jobs Search — Remote OK Jobs by Keyword, Tag and Salary — current remote jobs from Remote OK, with salary and listing links.
• Research Paper Search — OpenAlex Papers, DOIs, Citation Counts — Search OpenAlex for work-level paper metadata without authors, affiliations, abstracts or full text
• RSS Feed Reader — Atom and JSON Feed Parser With Enclosures — every item of an RSS, Atom or JSON feed.
• SEO Audit — On-Page Score, Meta Tags, Schema & Fix Guidance — an on-page SEO audit: meta tags, schema, headings and links.
• Text to Speech — Offline English, German, Spanish WAV Voices — Convert short nonpersonal texts to offline formant WAV speech for agents
• Time Zone Converter API — Convert one offset-qualified instant to named IANA time zones using pinned data.
• URL to Markdown — Clean Text for LLM & RAG, With Selectors — a public job page as clean Markdown where the site allows it (deny-listed job sites are refused).
• Extract Video Frames — Timed Stills by Interval or Scene Cut — frames, thumbnails and scenes from a video.
• Website Color Palette — CSS Colors With Alpha and Usage Counts — Extract declared website CSS color literals and counts from public HTML and bounded linked stylesheets
• Favicon Extractor — Icons, Touch Icons, Manifests, Theme Color — declared website icons, theme colors and manifest links.
• Website Screenshot — Full Pages, Elements, Dark Mode & PDF — screenshots and PDF prints of web pages.
• Tech Stack Detector — Website Technologies With Match Evidence — the technologies a website runs, detected live.
• Website Crawler to Markdown — Same-Site Pages for LLM and RAG — bounded whole-site Markdown with robots and crawl-delay enforced

Changelog

• 1.0 (2026-10-26) — store listing rewritten for the people and agents searching for this job: title, description, search-engine title and description, categories, and the README in the order a searcher reads it (what it does, a real result, how to use it, prices, the agent call).
• 1.0 (2026-10-20) — first public release on Apify Store, after the deploy's typical calls, memory probes, benchmarks and pricing floor passed on the platform.
• 1.0 (2026-10-19) — repairs from an independent review: DNS privacy, malformed publisher URLs, empty sitemap failures, responsive assets and schema prefill, as applicable to this Actor.
• 1.0 (2026-10-18) — the production image, smoke input, three example tasks and benchmarks passed on GitHub-hosted Linux.
• 1.0 (2026-10-17) — first implementation.