# Changelog of Sitemap Extractor: All URLs from XML Sitemaps for SEO (`fecklesspleb/sitemap-extractor`) Actor

- **URL**: https://apify.com/fecklesspleb/sitemap-extractor/changelog.md
- **Full Actor documentation**: https://apify.com/fecklesspleb/sitemap-extractor.md

## Changelog

### Unreleased: default mode for an empty input (GAP8)

- **BREAKING (default change):** the default `maxUrlsPerSite` is now 1,000 (was 10,000), in the input schema and in code when the field is left out. A run that relied on the old default now stops after 1,000 URLs per site, at most $0.20 per site. Set `maxUrlsPerSite` to 10000, or 0 for no limit, to get the old behaviour.
- An empty input (`{}`) no longer fails with HTTP 400 or a missing-field error. `startUrls` is no longer required and now has a schema `default` (and prefill) of `["https://www.nasa.gov/wp-sitemap.xml"]`. When `startUrls` is empty or is exactly that sample, the run is in default mode: it reads the NASA sitemap, returns at most 40 URLs (a smaller `maxUrlsPerSite` wins), logs `Default mode: ...`, adds `defaultMode` and `defaultStartUrl` to `RUN_SUMMARY` and prefixes the status message with `Default mode (sample sitemap): `.
- Input schema: the prefill changed from nasa.gov and the Cloudflare blog sitemap to the NASA sample sitemap, so the Store test runs default mode. Runs that list any other sites behave as before, apart from the new 1,000 default cap.
- Lint only: removed unused imports in `common.py` (pyflakes clean). No behaviour change.

### Unreleased: unreachable sitemaps are an error, robots.txt on every redirect

- A site where not one sitemap could be read (every sitemap returned an error or could not be reached) now has `status: "error"` in `RUN_SUMMARY`, with the last error, instead of `ok`. If every sitemap was blocked by robots.txt, the status is `skipped`. A site where some sitemaps load and others fail stays `ok`, with the failures counted in `sitemapErrors`.
- Redirects are now followed one hop at a time, and robots.txt is checked for every hop. A sitemap that redirects to a disallowed path or to another host whose robots.txt disallows it is not fetched. Up to 5 redirects are followed.
- If a site's robots.txt can't be read (a server error after one retry, a network failure, or a 200 that is an HTML page or not `text/plain`), the site is skipped for the run with a warning and `status: "skipped"`. Before, it was read as if it had no rules. A missing robots.txt (any 4xx) still means no rules.

### Unreleased: URLs without a lastmod

- With `lastmodAfter` or `lastmodWithinDays` set, URLs without a `lastmod` are dropped by default and counted (`undatedDropped`). Set `keepUndatedUrls` to `true` to keep them (`undatedKept`).