Versions follow MAJOR.MINOR.PATCH (src/version.py); Apify shows MAJOR.MINOR from .actor/actor.json.
Every run logs its version and records it in the RUN_STATS key-value record.
- A run that returns nothing (every input bad, private, blocked or unreachable, or an empty input) now ends
SUCCEEDED with an empty dataset and the reason as its status message and in the log, instead of FAILED. Only an
unexpected error still fails the run.
RUN_STATS keeps the reason as before.
- No change to output fields or price.
- A run no longer fails when the store it remembers earlier runs in can't be opened (403: the account already has
a store of that name that this actor didn't create). It falls back to
humble-echidna-<name>, and if that is
refused too it runs without memory for that run. Users' existing memory is untouched: the usual name is tried first.
- Waits out longer
Retry-After answers: when a site or API asks it to wait before retrying, it now waits up
to 90 seconds (was 30), once per site, with at most 3 minutes of such long waits per run; a longer wait, or a
site that asks again after it was waited out, is reported with the reason instead of stalling the run.
- No change to input, output or price.
- Accepts
startUrls (a list of strings or {"url": ...} objects) and maxUrls (0 = no limit), the names other
sitemap actors use, so an API call or integration built for one of them works by changing only the actor id.
startUrls adds to sites; when sites holds only the default Apify fills in (sitemaps.org), it's replaced, so
that default is never fetched or charged. maxResults wins when both caps are set.
- Another actor's regular-expression filter (
urlFilterInclude, urlFilterExclude) stops the run before anything is
fetched, with how to write it as a wildcard pattern, instead of being ignored (which would return and charge URLs
the user meant to filter out).
- README: "Switching from another sitemap actor?" (input mapping) and an FAQ on HTTP status codes per sitemap URL.
- Store title now says "XML Sitemaps" (the search term "xml sitemap").
sites is no longer marked required, so AI agents (Apify's MCP server) see that a dataset alone (datasetId) is enough. Nothing else changes: a run with neither still stops with a clear message.
List the sitemap URLs of the websites in another actor's results, or chain it after any actor with an integration.
- A dataset run where none of the websites has a sitemap succeeds with no rows (and says so) instead of failing:
small businesses from a leads scraper often have none. Typed-in sites without a sitemap still fail the run.
- New inputs Or: websites from a dataset (
datasetId, a dataset picker with read access) and Field with the
website (datasetUrlField). With a dataset, each item's website becomes a line of sites and sites is ignored
(Apify fills in its default on API and integration runs; it isn't fetched or charged). In an integration,
"datasetId": "{{resource.defaultDatasetId}}" passes the finished run's results. The field is found automatically
when left empty: the first of website, url, domain, site, homepage with a website in the first 100 items,
then the same names one level down (contact.website); a dotted path names a nested one. A Google Maps link (a
Maps scraper's url) is never used.
- One line per site: a sitemap belongs to the whole site, so a website is read from its home page whatever page the
item links to (
https://bakery.com/locations/denver is the line https://bakery.com), and several items on one
website (www. or not) read it once, with the first item's source fields. A sitemap address in an item is kept
and read directly. Every other option applies as to a site typed in.
- Only new and removed URLs keeps one list per site and settings, as before; the dataset isn't part of it, so a
chained run (a new dataset every time) compares with the last run, and a site from a dataset shares the list of
the same site typed in. Inputs without a dataset keep their lists from earlier runs (the key is unchanged).
- Items without a website, values that aren't a public web address and Google Maps links are skipped, never fetched
or charged, and counted in the new
RUN_STATS.dataset record. A dataset with no website at all, or one that can't
be read, fails the run with the reason.
- New nullable output fields
sourceTitle, sourcePlaceId and sourceIndex (the source item's title or name, its
placeId, its position in the dataset) on every URL from that item's site, removed URLs included; null for sites
typed in.
- Read with the run's own token (your access, read-only), only the fields needed; at most the first 20,000 items and
10,000 distinct websites per run.
- Pricing unchanged: no new events; a URL costs the same whichever way its site came in.
- 2 ready-to-run examples published on Apify Store and linked from the README.
- Cheaper on paid Apify plans: $0.25 per 1,000 URLs on Starter, $0.18 on Scale and $0.10 on Business ($0.30 on the Free plan).
- New and removed URLs since the last run (
onlyNewAndRemovedUrls, off by default): each run compares a line's
sitemaps with its previous run and returns only URLs not listed before (changeType: "new") and URLs no longer
listed (changeType: "removed", with the sitemap that last listed them; the other fields empty). URLs listed both
times are counted (RUN_STATS per site monitor.unchanged), not returned or charged, so a quiet run costs the
start fee. The first run returns every URL (changeType null) as the baseline. Charged like any URL; no new events.
- The list is kept in the user's
sitemap-urls-memory key-value store, one record per input line and its settings
(URL patterns, max sitemap depth and files, hashed; the key uses only Apify's allowed characters), as gzipped JSON
(21.8 KB for crawlee.dev's 4,852 URLs). A URL is remembered or forgotten only once its row is in the
dataset, so rows cut by the max results or max cost come back next run. A quiet run writes nothing.
- Guards: nothing is reported removed unless every sitemap found for the line was read to the end (no failed
download, robots.txt block, depth or file limit, file over 50,000 entries, or stop at the run's limit;
monitor.removedNotReported says why). A clean read that lists nothing, or loses more than half of the tracked
URLs (at least 20) at once, is held until the next run sees the same (monitor.removedHeld). Up to 200,000 URLs
are tracked per line. The lastmod date filters can't be combined with it (the run fails with the reason).
- Dataset: new nullable field
changeType (null with the option off) and a "New and removed URLs" view.
RUN_STATS gains monitor, changes and, per site, monitor.
- Shared
mms_sitemap.crawl: crawl_site(..., is_new=) lets a caller skip URLs it already knows, and
incomplete_reasons(stats) says why a line's list may be missing URLs. Other actors are unaffected.
- Internal: the URL-pattern and lastmod filters moved into the shared
mms_sitemap package (next to the discovery,
parsing and crawl code already there), so article-extractor uses the same ones. No change in behaviour or output.
First release.
- Returns every URL in the XML sitemaps of the sites you enter, one row per URL:
lastmod, changefreq,
priority, the sitemap it came from, and image, news, video and hreflang alternate entries.
- Discovery: robots.txt
Sitemap: lines, then /sitemap.xml and /sitemap_index.xml; a sitemap URL is read
directly. Sitemap indexes are followed breadth-first with depth and per-site file caps. Gzipped (.xml.gz or
gzip transfer encoding) and plain-text sitemaps are read; HTML soft-404 pages are skipped.
- Filters: include/exclude URL patterns, lastmod on-or-after / before (absolute or relative dates), keep or drop
URLs without a lastmod. Filtered-out and duplicate URLs are not charged.
- Politeness: honours robots.txt Disallow and Crawl-delay on every request, retry and redirect hop (robots.txt
read once per origin, matched on the product token
HumbleEchidnaApify, RFC 9309 incl. byte-order marks and
percent-encoding), and Retry-After (seconds or an HTTP date); a Crawl-delay or Retry-After above 30 s fails that
site or sitemap with the reason instead of being cut short silently. Honest User-Agent
(HumbleEchidnaApify/1.0 (+https://apify.com/humble-echidna)).
- Safety: refuses private, loopback, link-local and other non-public addresses, including the IPv6 forms that
carry them (IPv4-mapped, IPv4-compatible
::a.b.c.d, NAT64 64:ff9b::/96; 6to4, Teredo and site-local are
refused outright), checked on the resolved IP at connect time on every hop; only ports 80 and 443; a time limit
per download attempt and one per sitemap (retries, redirects and robots.txt included); 50 MB cap per sitemap
before and after decompression, counted while streaming (gzip, zlib and raw deflate); a 3xx without a Location
is an error; XML parsed with DTDs forbidden.
- Memory bounded per sitemap: each file goes to a temporary file (on disk past 1 MB), is parsed as a stream and
emitted in batches; at most 50,000 entries per file (the sitemaps.org limit), the rest reported.
- Accurate stats:
sitemapsRead counts files read (guessed locations that don't exist aren't failures, and
robots.txt-disallowed ones are counted as skipped, not failed); emitted counts URLs actually written;
RUN_STATS has the same keys on every exit path.
- The same site typed as
example.com, http://example.com or https://example.com/ is read and charged once.
- URL patterns are matched without regular expressions (runs of
* collapse, at most 500 characters), so no
pattern can make a run slow.
- The safety code is the shared
mms_common implementation every actor in this portfolio uses.
- Charged per URL returned, through Apify's standard
apify-default-dataset-item event.
- "Max results per run" input, and the maximum cost per run is honoured: the budget is checked before every
sitemap fetch, and a download in progress is abandoned as soon as every result the run can pay for is reserved,
so a capped run stops downloading; each site's log line and
stoppedAtLimit say when it stopped there, and sites
not started are listed as queriesSkipped in RUN_STATS.
- Memory: decompression runs in 64 KiB steps, so memory is bounded per step, not by the 50 MB cap. A whole run on a
48.7 MiB, 50,000-URL sitemap (plain and
.xml.gz) in a 512 MB container peaked at 83 MiB RSS.
- Failure isolation: a failing site only affects itself.
