# Changelog of Sitemap URL Extractor: robots.txt, gzip, Changes (`succinct_glider/sitemap-urls`) Actor

- **URL**: https://apify.com/succinct\_glider/sitemap-urls/changelog.md
- **Full Actor documentation**: https://apify.com/succinct\_glider/sitemap-urls.md

## Changelog

All notable changes to this Actor are documented here. The output fields are a contract: no field is
renamed, removed or retyped outside a major version.

### \[0.2] — unreleased

- One website receives at most 4 requests at a time and 10 a second, sitemap downloads and status checks
  together, whatever the run does elsewhere.

- A status check that returned nothing (timeout, refused address) is not charged.

- New option **Check the HTTP status of each URL** (`checkUrlStatus`, off by default): a HEAD request per saved
  URL, redirects reported and not followed, GET only when HEAD is not supported, internal addresses never
  requested. New output fields `statusCode`, `redirectUrl` and `statusError` (`null` when the option is off),
  and a **URL status** view. Charged as the `url-status-check` event, $0.70 per 1,000 checked URLs on top of
  the URL itself; the budget counts both events of a checked URL.

- RSS 2.0, RSS 1.0 and Atom feeds are read like sitemaps (link and publication or update date). Feed URLs
  (`/feed`, `/rss`, `.rss`, `.atom`) are read directly.

- Memory limit raised from about 450 to 1,000 URL entries per MB: the old figure came from a measurement
  distorted by the memory leak fixed in 0.1.5. Measured since on the platform: about 580 bytes per entry
  (885 MB of 1,024 at 1.5 million entries).

- No existing output field changed.

### \[0.1] — 2026-09-17, published

#### README (build 0.1.6)

- New sections: why this extractor, run summary example, known limits. No code or output change.

#### First version

First version.

- Lists the URLs of websites from their XML sitemaps. Input: domains, pages or sitemap URLs. Sitemaps are
  found in robots.txt, then at `/sitemap.xml`, `/sitemap_index.xml` and `/sitemap-index.xml`.
- Follows sitemap indexes (5 levels), reads `.xml.gz` files whatever their content type, plain-text
  sitemaps, and the image, video, news and `hreflang` extensions.
- Output: `url`, `site`, `lastmod`, `changefreq`, `priority`, `sitemapUrl`, `discoveredFrom`, `imageCount`,
  `videoCount`, `newsTitle`, `newsPublishedAt`, `alternates`, change-tracking fields and `scrapedAt`.
- Filters applied before saving: URL patterns, `modifiedSince` (absolute or relative), `requireLastmod`, and
  `sitemapUrlPatterns` for the children of an index.
- Change tracking (`trackChanges`, `onlyChanges`): new, changed, unchanged and removed URLs, removals only
  after complete reads. The state is stored gzipped, in chunks, in a named key-value store.
- Limits: `maxItems` (default 500), `maxItemsPerSite`, `maxSitemapsPerSite`, and the user's maximum cost per
  run (pay-per-event), all claimed before each push. A website that reaches its limit stops being read.
- Only public http(s) addresses are requested, redirects included (each hop is checked).
- Files are downloaded as streams (at most 100 MB each) and only kept while they are read: with Crawlee's
  `HttpCrawler`, every sitemap body stayed in memory until the end of the run.
- Connection attempts get 2.5 s per address family, so dual-stack websites no longer time out on the
  platform, which has no IPv6 route.
- `SUMMARY` record with the status of each website (`ok`, `partial`, `no_sitemap`, `not_reached`,
  `blocked`, `failed`), with the HTTP status, content type and size of unexpected answers. A website that
  answers with a bot challenge (AWS WAF, Cloudflare) is reported as `blocked`, not failed. The run fails
  only when no website could be read for technical reasons.
- The run keeps at most about 450 URLs per MB of memory (deduplication and tracked state) and stops cleanly
  past that, with a message asking for more memory.