# Changelog of Sitemap URL Extractor: All URLs from XML Sitemaps & Changes (`humble-echidna/sitemap-urls`) Actor - **URL**: https://apify.com/humble-echidna/sitemap-urls/changelog.md - **Full Actor documentation**: https://apify.com/humble-echidna/sitemap-urls.md ## Changelog Versions follow MAJOR.MINOR.PATCH (`src/version.py`); Apify shows MAJOR.MINOR from `.actor/actor.json`. Every run logs its version and records it in the `RUN_STATS` key-value record. ### 1.3.3 (2026-10-16) - A run that returns nothing (every input bad, private, blocked or unreachable, or an empty input) now ends SUCCEEDED with an empty dataset and the reason as its status message and in the log, instead of FAILED. Only an unexpected error still fails the run. `RUN_STATS` keeps the reason as before. - No change to output fields or price. ### 1.3.2 (2026-10-03) - A run no longer fails when the store it remembers earlier runs in can't be opened (403: the account already has a store of that name that this actor didn't create). It falls back to `humble-echidna-`, and if that is refused too it runs without memory for that run. Users' existing memory is untouched: the usual name is tried first. ### 1.3.1 (2026-10-01) - **Waits out longer `Retry-After` answers:** when a site or API asks it to wait before retrying, it now waits up to 90 seconds (was 30), once per site, with at most 3 minutes of such long waits per run; a longer wait, or a site that asks again after it was waited out, is reported with the reason instead of stalling the run. - No change to input, output or price. ### 1.3.0 (2026-10-01) - Accepts `startUrls` (a list of strings or `{"url": ...}` objects) and `maxUrls` (0 = no limit), the names other sitemap actors use, so an API call or integration built for one of them works by changing only the actor id. `startUrls` adds to `sites`; when `sites` holds only the default Apify fills in (`sitemaps.org`), it's replaced, so that default is never fetched or charged. `maxResults` wins when both caps are set. - Another actor's regular-expression filter (`urlFilterInclude`, `urlFilterExclude`) stops the run before anything is fetched, with how to write it as a wildcard pattern, instead of being ignored (which would return and charge URLs the user meant to filter out). - README: "Switching from another sitemap actor?" (input mapping) and an FAQ on HTTP status codes per sitemap URL. - Store title now says "XML Sitemaps" (the search term "xml sitemap"). ### 1.2.1 (2026-09-29) - `sites` is no longer marked required, so AI agents (Apify's MCP server) see that a dataset alone (`datasetId`) is enough. Nothing else changes: a run with neither still stops with a clear message. ### 1.2.0 (2026-09-29) List the sitemap URLs of the websites in another actor's results, or chain it after any actor with an integration. - A dataset run where none of the websites has a sitemap succeeds with no rows (and says so) instead of failing: small businesses from a leads scraper often have none. Typed-in sites without a sitemap still fail the run. - New inputs **Or: websites from a dataset** (`datasetId`, a dataset picker with read access) and **Field with the website** (`datasetUrlField`). With a dataset, each item's website becomes a line of `sites` and `sites` is ignored (Apify fills in its default on API and integration runs; it isn't fetched or charged). In an integration, `"datasetId": "{{resource.defaultDatasetId}}"` passes the finished run's results. The field is found automatically when left empty: the first of `website`, `url`, `domain`, `site`, `homepage` with a website in the first 100 items, then the same names one level down (`contact.website`); a dotted path names a nested one. A Google Maps link (a Maps scraper's `url`) is never used. - One line per site: a sitemap belongs to the whole site, so a website is read from its home page whatever page the item links to (`https://bakery.com/locations/denver` is the line `https://bakery.com`), and several items on one website (`www.` or not) read it once, with the first item's source fields. A sitemap address in an item is kept and read directly. Every other option applies as to a site typed in. - **Only new and removed URLs** keeps one list per site and settings, as before; the dataset isn't part of it, so a chained run (a new dataset every time) compares with the last run, and a site from a dataset shares the list of the same site typed in. Inputs without a dataset keep their lists from earlier runs (the key is unchanged). - Items without a website, values that aren't a public web address and Google Maps links are skipped, never fetched or charged, and counted in the new `RUN_STATS.dataset` record. A dataset with no website at all, or one that can't be read, fails the run with the reason. - New nullable output fields `sourceTitle`, `sourcePlaceId` and `sourceIndex` (the source item's title or name, its `placeId`, its position in the dataset) on every URL from that item's site, removed URLs included; `null` for sites typed in. - Read with the run's own token (your access, read-only), only the fields needed; at most the first 20,000 items and 10,000 distinct websites per run. - Pricing unchanged: no new events; a URL costs the same whichever way its site came in. ### 1.1.2 (2026-09-28) - 2 ready-to-run examples published on Apify Store and linked from the README. ### 1.1.1 (2026-09-28) - Cheaper on paid Apify plans: $0.25 per 1,000 URLs on Starter, $0.18 on Scale and $0.10 on Business ($0.30 on the Free plan). ### 1.1.0 (2026-09-26) - **New and removed URLs since the last run** (`onlyNewAndRemovedUrls`, off by default): each run compares a line's sitemaps with its previous run and returns only URLs not listed before (`changeType: "new"`) and URLs no longer listed (`changeType: "removed"`, with the sitemap that last listed them; the other fields empty). URLs listed both times are counted (`RUN_STATS` per site `monitor.unchanged`), not returned or charged, so a quiet run costs the start fee. The first run returns every URL (`changeType` null) as the baseline. Charged like any URL; no new events. - The list is kept in the user's `sitemap-urls-memory` key-value store, one record per input line and its settings (URL patterns, max sitemap depth and files, hashed; the key uses only Apify's allowed characters), as gzipped JSON (21.8 KB for crawlee.dev's 4,852 URLs). A URL is remembered or forgotten only once its row is in the dataset, so rows cut by the max results or max cost come back next run. A quiet run writes nothing. - Guards: nothing is reported removed unless every sitemap found for the line was read to the end (no failed download, robots.txt block, depth or file limit, file over 50,000 entries, or stop at the run's limit; `monitor.removedNotReported` says why). A clean read that lists nothing, or loses more than half of the tracked URLs (at least 20) at once, is held until the next run sees the same (`monitor.removedHeld`). Up to 200,000 URLs are tracked per line. The lastmod date filters can't be combined with it (the run fails with the reason). - Dataset: new nullable field `changeType` (`null` with the option off) and a "New and removed URLs" view. `RUN_STATS` gains `monitor`, `changes` and, per site, `monitor`. - Shared `mms_sitemap.crawl`: `crawl_site(..., is_new=)` lets a caller skip URLs it already knows, and `incomplete_reasons(stats)` says why a line's list may be missing URLs. Other actors are unaffected. ### 1.0.1 (2026-09-24) - Internal: the URL-pattern and lastmod filters moved into the shared `mms_sitemap` package (next to the discovery, parsing and crawl code already there), so article-extractor uses the same ones. No change in behaviour or output. ### 1.0.0 (2026-09-24) First release. - Returns every URL in the XML sitemaps of the sites you enter, one row per URL: `lastmod`, `changefreq`, `priority`, the sitemap it came from, and image, news, video and hreflang alternate entries. - Discovery: robots.txt `Sitemap:` lines, then `/sitemap.xml` and `/sitemap_index.xml`; a sitemap URL is read directly. Sitemap indexes are followed breadth-first with depth and per-site file caps. Gzipped (`.xml.gz` or gzip transfer encoding) and plain-text sitemaps are read; HTML soft-404 pages are skipped. - Filters: include/exclude URL patterns, lastmod on-or-after / before (absolute or relative dates), keep or drop URLs without a lastmod. Filtered-out and duplicate URLs are not charged. - Politeness: honours robots.txt Disallow and Crawl-delay on every request, retry and redirect hop (robots.txt read once per origin, matched on the product token `HumbleEchidnaApify`, RFC 9309 incl. byte-order marks and percent-encoding), and `Retry-After` (seconds or an HTTP date); a Crawl-delay or Retry-After above 30 s fails that site or sitemap with the reason instead of being cut short silently. Honest User-Agent (`HumbleEchidnaApify/1.0 (+https://apify.com/humble-echidna)`). - Safety: refuses private, loopback, link-local and other non-public addresses, including the IPv6 forms that carry them (IPv4-mapped, IPv4-compatible `::a.b.c.d`, NAT64 `64:ff9b::/96`; 6to4, Teredo and site-local are refused outright), checked on the resolved IP at connect time on every hop; only ports 80 and 443; a time limit per download attempt and one per sitemap (retries, redirects and robots.txt included); 50 MB cap per sitemap before and after decompression, counted while streaming (gzip, zlib and raw deflate); a 3xx without a Location is an error; XML parsed with DTDs forbidden. - Memory bounded per sitemap: each file goes to a temporary file (on disk past 1 MB), is parsed as a stream and emitted in batches; at most 50,000 entries per file (the sitemaps.org limit), the rest reported. - Accurate stats: `sitemapsRead` counts files read (guessed locations that don't exist aren't failures, and robots.txt-disallowed ones are counted as skipped, not failed); `emitted` counts URLs actually written; `RUN_STATS` has the same keys on every exit path. - The same site typed as `example.com`, `http://example.com` or `https://example.com/` is read and charged once. - URL patterns are matched without regular expressions (runs of `*` collapse, at most 500 characters), so no pattern can make a run slow. - The safety code is the shared `mms_common` implementation every actor in this portfolio uses. - Charged per URL returned, through Apify's standard `apify-default-dataset-item` event. - "Max results per run" input, and the maximum cost per run is honoured: the budget is checked before every sitemap fetch, and a download in progress is abandoned as soon as every result the run can pay for is reserved, so a capped run stops downloading; each site's log line and `stoppedAtLimit` say when it stopped there, and sites not started are listed as `queriesSkipped` in `RUN_STATS`. - Memory: decompression runs in 64 KiB steps, so memory is bounded per step, not by the 50 MB cap. A whole run on a 48.7 MiB, 50,000-URL sitemap (plain and `.xml.gz`) in a 512 MB container peaked at 83 MiB RSS. - Failure isolation: a failing site only affects itself.