Sitemap URL extractor: enter a website or a sitemap.xml and get all its URLs with lastmod, changefreq, priority, image count and optional hreflang. Finds sitemaps via robots.txt, follows sitemap indexes, reads .gz. Regex and date filters, only-new-URLs mode. Usable via Apify MCP. $0.20 per 1K URLs.
Unreleased: number list items no longer crash the run (1104d)
A number as a list item in startUrls, includePatterns, excludePatterns or sitemapPatterns (e.g. startUrls: [5]), or a non-text url in a {"url": ...} item, used to crash the run with a traceback (exit 91, FAILED, empty dataset). Now the run ends SUCCEEDED (exit 0) with the existing free invalid-input status row (status: "error", message Invalid input, nothing was fetched or charged: list items must be text or {"url": "..."} objects, not 5.). Nothing is fetched, no RUN_SUMMARY is written, no result event is charged.
An input that is not a JSON object (a list, string or number) now gets the same status row. Before, it was silently run as {} (the NASA demo). No input at all is still run as {}.
1104d changed these files: CHANGELOG.md, my_actor/main.py, tests/test_1104d_invalid_input.py. No change to prices, events, .actor/ files, README or requirements (apify >= 4.1, < 4.2 was already pinned).
Unreleased: private-address guard (GAP17)
On the Apify platform (APIFY_IS_AT_HOME set), every request now checks the host first: if any address it resolves to is private, loopback, link-local, reserved, multicast, unspecified or otherwise not global (also IPv4-mapped IPv6), the request is not sent. This covers start URLs, every redirect hop (an httpx request hook on the client), robots.txt, sitemaps listed in robots.txt, the fallback sitemap paths and child sitemaps of an index. URLs listed in a sitemap are only reported, never fetched, as before. Local runs are unchanged.
A blocked site shows status: error, error: "not fetched: private, loopback and link-local addresses are not allowed" in RUN_SUMMARY and costs nothing; a blocked sitemap counts in sitemapErrors (its text is the last error when no sitemap of the site could be read). When no site could be read, the existing free status row is written. A host whose DNS lookup fails reads DNS lookup failed for <host>.
No change to prices, events, input, output fields or .actor/ files. Files: CHANGELOG.md, README.md (one FAQ entry), my_actor/common.py, my_actor/main.py, tests/test_ssrf_guard_1102t.py.
Unreleased: an invalid regex no longer crashes the run (GAP16)
An invalid regular expression in includePatterns, excludePatterns or sitemapPatterns (for example () used to crash the run with a traceback (exit 91) and an empty dataset. Now the run fails with one clear status message that names the field and the pattern, pushes one free status item (status: "error", no charged event), and fetches nothing.
GAP16 changed these files: CHANGELOG.md, my_actor/main.py, tests/test_gap16_bad_pattern.py. No change to .actor/input_schema.json or .actor/actor.json.
Unreleased: changelog file lists (GAP13)
GAP13 changed these files: CHANGELOG.md. No change to .actor/input_schema.json or .actor/actor.json.
GAP12 changed these files: .actor/actor.json, CHANGELOG.md, README.md, my_actor/common.py, my_actor/main.py, my_actor/runguard.py, tests/_e2e.py, tests/test_gap12_body_cap.py, tests/test_gap12_example_input.py, tests/test_gap12_fail_soft_e2e.py, tests/test_gap12_parser.py, tests/test_gap12_runguard.py, tests/test_site_status.py. In .actor/actor.json GAP12 only added exampleRunInput (shown in the GAP12 entry below); it did not change the description, the version or any other field. GAP12 made no change to .actor/input_schema.json (defaults, prefill and field texts are as before).
Unreleased: fail soft and a time budget for the Store test (GAP12)
Fail soft: a site that is down, slow or blocked no longer gives an empty dataset. When no URL was read (in the demo, or when no site could be read), the run pushes one free status item (no charged event) with the reason, logs a warning, and ends SUCCEEDED.
Lower memory for large sitemaps: XML is parsed incrementally and each entry is freed once read (a 47 MB sitemap with 50,000 URLs and hreflang alternates peaked at 531 MB before, 312 MB now, and 152 MB with includeAlternates off, the default). A sitemap body is streamed and dropped above 52 MB (10 MB in the demo), and a .gz sitemap is decompressed with the same cap, so a small gzip file cannot inflate past it.
The sitemap-extractor-state key-value store is opened only when onlyNewUrls is on, and store errors are warnings.
Time budget: the demo run ({}, the prefill or the exampleRunInput) starts no new request after about 40 s, uses a 10 s timeout per request with at most 2 attempts and waits at most 2 s between them (a long Retry-After is capped). Every run, demo or not, also stops starting requests 20 s before the platform timeout (ACTOR_TIMEOUT_AT) and gets cut off at most 10 s after that, so a slow or hung source can no longer make the run time out. When the platform aborts the run, no new request is started.
robots.txt is now fetched with one lock per host instead of one lock for all hosts, so one slow host no longer holds up the others.
New exampleRunInput in actor.json: {"startUrls": ["https://www.nasa.gov/wp-sitemap.xml"]}. It matches the prefill and runs the same small demo.
Tests added for a source that is down, slow or hung, a partial failure, the time budget, the free status item and an unreadable state store.
Unreleased: default mode for an empty input (GAP8)
BREAKING (default change): the default maxUrlsPerSite is now 1,000 (was 10,000), in the input schema and in code when the field is left out. A run that relied on the old default now stops after 1,000 URLs per site, at most $0.20 per site. Set maxUrlsPerSite to 10000, or 0 for no limit, to get the old behaviour.
An empty input ({}) no longer fails with HTTP 400 or a missing-field error. startUrls is no longer required and now has a schema default (and prefill) of ["https://www.nasa.gov/wp-sitemap.xml"]. When startUrls is empty or is exactly that sample, the run is in default mode: it reads the NASA sitemap, returns at most 40 URLs (a smaller maxUrlsPerSite wins), logs Default mode: ..., adds defaultMode and defaultStartUrl to RUN_SUMMARY and prefixes the status message with Default mode (sample sitemap): .
Input schema: the prefill changed from nasa.gov and the Cloudflare blog sitemap to the NASA sample sitemap, so the Store test runs default mode. Runs that list any other sites behave as before, apart from the new 1,000 default cap.
Lint only: removed unused imports in common.py (pyflakes clean). No behaviour change.
Unreleased: unreachable sitemaps are an error, robots.txt on every redirect
A site where not one sitemap could be read (every sitemap returned an error or could not be reached) now has status: "error" in RUN_SUMMARY, with the last error, instead of ok. If every sitemap was blocked by robots.txt, the status is skipped. A site where some sitemaps load and others fail stays ok, with the failures counted in sitemapErrors.
Redirects are now followed one hop at a time, and robots.txt is checked for every hop. A sitemap that redirects to a disallowed path or to another host whose robots.txt disallows it is not fetched. Up to 5 redirects are followed.
If a site's robots.txt can't be read (a server error after one retry, a network failure, or a 200 that is an HTML page or not text/plain), the site is skipped for the run with a warning and status: "skipped". Before, it was read as if it had no rules. A missing robots.txt (any 4xx) still means no rules.
Unreleased: URLs without a lastmod
With lastmodAfter or lastmodWithinDays set, URLs without a lastmod are dropped by default and counted (undatedDropped). Set keepUndatedUrls to true to keep them (undatedKept).