# Changelog of Sitemap URL Extractor: Every Page with Last-Modified Dates (`rowcraft/sitemap-url-extractor`) Actor

- **URL**: https://apify.com/rowcraft/sitemap-url-extractor/changelog.md
- **Full Actor documentation**: https://apify.com/rowcraft/sitemap-url-extractor.md

## Changelog

### 1.0.4 (2026-10-17)

- Fix: a robots.txt served as an HTML page (catch-all page or a redirect to the home page, for example thumb.wikimedia.org) counted as an anti-bot block, so every page on that host was refused; it is now treated as a missing robots.txt, as the standard says. Real anti-bot pages are still recognised.

### 1.0.3 (2026-10-17)

- README: the Use cases section links the two public examples kept on the Store (Apify allows 100 public tasks per account; the third example stays available as a private task spec).

### 1.0.2 (2026-10-17)

- README: the list of other Rowcraft actors now includes the five website and extension tools published on 16.10.2026.
- Shared library: URL-like fields named link or guid (feed items) are no longer flagged as phone numbers; retry log line now reads "retry n/m".
- Shared library: an HTML page where JSON was expected (maintenance or block page, seen on 17.10.2026 at the Polish KRS API and at Workday) is retried and then reported with the page text instead of crashing the run on a parse error.

### 1.0.1 (2026-10-14)

- README: real screenshots of the input form and of a results table from Apify Console.

### 1.0.0 (2026-10-14)

- First release: every page address that a website lists in its sitemaps, with last-modified date and sitemap hints. Give a domain, a robots.txt address or a sitemap address.
- Sitemaps are found through the `Sitemap:` lines of robots.txt; when a site names none, `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml` are tried. Sitemap indexes are followed (up to 5 levels, 500 files per site by default), `.xml.gz` files are unpacked as they download, and text sitemaps and RSS or Atom feeds are read too. Files are parsed as a stream (up to 60 MB each), so a sitemap of 300,000 addresses runs in 256 MB.
- One row per page address, delivered once across all sitemaps: `url`, `lastmod`, `changefreq`, `priority`, `sitemapUrl`, `sitemapType`, `imageCount`, `videoCount`, `newsTitle`, `newsPublicationDate`, `alternates` (hreflang versions), `site`, `firstSeenAt`, `extractedAt`. No person fields; author and user sitemaps are not read.
- Filters before anything is charged: `urlIncludes` and `urlExcludes` (text, or `re:` and a regular expression), `lastmodAfter` (a day or `7 days`; index files whose own date is older are not opened), `maxUrlsPerSite`, `maxSitemapFiles`, `maxItems`.
- `onlyNew` and `stateName`: deliver only the addresses that are new since the last run with the same state name (a compact memory of up to 1,000,000 addresses), the way to watch a competitor's new pages. `countOnly` counts matching addresses and estimates the cost without delivering anything.
- `COVERAGE` per entry (ok, no_sitemap, blocked, robots_disallowed, not_found, error, not_checked, with the files read and the reason); robots.txt is obeyed on the host of every sitemap, redirects are checked host by host, anti-bot pages are never charged, and the run stops after 10 blocked entries in a row.
- Standard Rowcraft automation: end-of-run notification (`notifyUrl`), charge cap respected, stop with the summary saved 90 seconds before the run timeout, one request per second per host by default (minimum 250 ms).