# Changelog of Sitemap Extractor & URL Discovery (`madpai/sitemap-url-discovery`) Actor

- **URL**: https://apify.com/madpai/sitemap-url-discovery/changelog.md
- **Full Actor documentation**: https://apify.com/madpai/sitemap-url-discovery.md

## Changelog

### 2026-11-02 (security update)

#### Security

- Connections are pinned to the addresses that passed the private-address check. Each host name is checked again when the connection opens, including redirect hops and robots.txt, so a name that switches to a private address after the first lookup (DNS rebinding) is never connected to. Private IP addresses written directly in a URL are refused too. TLS still verifies the host name.

#### Changed

- If a site's robots.txt host resolves to a private, loopback, link-local or reserved address, the item is now a free `blocked-private`, not a robots.txt result.

### 0.1.3 (2026-11-01) — hotfix

#### Fixed

- `allowedByRobots` is now checked against the robots.txt of each URL's own host. URLs on a different host than the input site (e.g. a sitemap on www.google.com declared in mail.google.com/robots.txt, or apex → www) used to be reported as allowed from the wrong robots.txt. Other hosts' robots.txt files are fetched once each at the normal 1 request/s; a 5xx or unreachable robots.txt gives `false` (RFC 9309); past 10 extra hosts the value is `null` with a `robotsNote`. A host without a robots.txt (404) now gives `true` instead of `null`.
- When no sitemap is found, the run summary now explains why (`noSitemapReason`, also added to `sitemapErrors` and the log), e.g. "No sitemap found: robots.txt not found (HTTP 404); tried /sitemap.xml (HTTP 404), … fallbackCrawl is off". Before, the result was empty with no reason.
- `maxUrls` now stops sitemap fetching on time. Before, an apex-to-www redirect could fetch 5 sitemaps when 3 were enough.
- The fallback crawl follows an apex-to-www redirect (or the reverse), so sites like that return their pages instead of 1 URL. Each URL's `domain` field is now its own host.
- One shared polite layer for every request: about 1 request per second per host, Crawl-delay, Retry-After (including HTTP dates), a pause after three 429 responses in a row, and robots.txt checked per host, including redirect hops. A robots.txt that returns 5xx or can't be reached now counts as disallow-all (RFC 9309).

#### Added

- Dataset field `robotsNote` (string or null).
- RSS 2.0, RSS 1.0 and Atom feeds are read as sitemaps. Any input with a path is tried directly before looking for the site's sitemaps.
- Inputs, sitemap entries and redirects pointing at private, loopback, link-local or reserved addresses are never requested; they show up as `blocked-private` errors.
- Unit tests (`npm test`).

### 0.1.2 (2026-11-01)

#### Added

- Output schema (`.actor/output_schema.json`): the run's Output tab links the URL dataset (overview view) and the `OUTPUT_SUMMARY` run summary.
- Store README.

#### Fixed

- The first sitemap request after `robots.txt` now also waits for the site's `Crawl-delay`, not just the default 1 s delay.

#### Pricing

- Pay per event: `url` at $0.40 per 1,000 URLs saved ($0.0004 per URL). Sites or sitemaps that fail save no URLs, so they add no `url` events.

### 0.1.1 (2026-11-01)

- Defaults fixed for cloud runs (256 MB memory, 300 s timeout). Private test builds only.

### 0.1.0 (2026-11-01)

#### Added

- Initial release. URL discovery from robots.txt, sitemap indexes, gzipped and plain-text sitemaps, with lastmod, hreflang alternates, include/exclude and lastmod filters, and a shallow fallback link crawl for sites without a sitemap.
- Respects robots.txt and waits between requests by default; `ownSiteConfirmed` is required to relax this for your own sites.