Sitemap URL Extractor with lastmod Filter

What it does

This Actor lists the page URLs in a website's sitemaps. For each site you give, it reads robots.txt first, takes the sitemaps named there, follows sitemap indexes, and reads .xml.gz and plain text sitemaps. If robots.txt names no sitemap it can read, it can try four common sitemap paths, each only where robots.txt allows. Each page URL becomes one dataset row with its lastmod value, and an optional date filter keeps only pages changed on or after a day you pick. It requests only robots.txt files and files named as sitemaps, the files sites publish for machines. It never requests the pages themselves: each page URL is only checked against its host's robots.txt and, if allowed, written to your dataset.

How to use it

1. Open the Actor in Apify Console.
2. The form opens with https://docs.apify.com in Websites and 20 in Maximum URLs to save, as a sample. Replace the sample site with your own sites, one per line, and set how many URLs you want.
3. If you know a sitemap's address, add it to Sitemap URLs (optional). To get only recently changed pages, pick a date in Only URLs changed since.
4. Click Start.
5. When the run ends, open the Output tab for the URL list. REPORT and SKIPPED are in the run's key-value store, on the Storage tab.

Input

Field	What it does	Default	Limits
Websites (sites)	Domains or site addresses, one per line. https:// is added when no scheme is given. Any path is ignored, so the whole site's sitemaps are read. An address ending in .xml, .xml.gz or .txt is read as a sitemap file instead	None. Prefilled with https://docs.apify.com	http and https only, no user name or password in the address, no control or invisible characters (an entry holding one is listed as invalid_character, even in a path that would be ignored). Up to 1,000 entries
Sitemap URLs (optional) (sitemapUrls)	Direct links to sitemap files or sitemap indexes, one per line. robots.txt is still read and obeyed first. For a host given only here, robots.txt Sitemap lines and common paths are not used	None	As for Websites. At least one website or sitemap URL is needed
Only URLs changed since (changedSince)	Saves only URLs whose lastmod is on or after this date. Empty saves every URL	Empty	YYYY-MM-DD or a full ISO date and time, read as UTC when it has no time zone. A date the Actor cannot read stops the run before any request, so no URL is charged
Keep URLs that have no lastmod (includeUrlsWithoutLastmod)	Used only with the date filter. On: URLs with no lastmod, or one that cannot be read, are saved too. Off: they are left out and counted in REPORT	Off
Maximum URLs to save (maxUrls)	The run stops after saving this many URLs. Each saved URL is one charge, so this caps what a run can charge for URLs	1,000. Prefilled with 20	1 to 500,000
Maximum sitemap files to fetch (maxSitemaps)	Upper limit on sitemap files fetched in one run, index files and their children included	200	1 to 5,000
Maximum sitemap index depth (maxSitemapDepth)	Levels of sitemap indexes to follow. 0 reads only the top sitemap files, 1 also reads their children, and so on	3	0 to 10
Try common sitemap paths (tryCommonPaths)	When robots.txt names no sitemap that can be read, tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /sitemap.xml.gz, one at a time, each only if robots.txt allows it, and stops at the first that works	On	Sites given in Websites only
Request timeout (requestTimeoutSecs)	Seconds to wait at each step of a request, such as connecting or the next part of the answer. A file that takes more than four times this in total is abandoned and listed as timeout	30 seconds	5 to 120 seconds

Output

Each saved URL is one dataset item. This real item is from a local test run of the prefilled input:

{"url": "https://docs.apify.com/api", "lastmod": null, "sourceSitemap": "https://docs.apify.com/sitemap_base.xml", "site": "https://docs.apify.com"}

Field	What it holds
url	The page URL as the sitemap lists it
lastmod	The lastmod text exactly as the sitemap gives it, or null when there is none. Text sitemaps have none
sourceSitemap	The sitemap file the URL was listed in, as the Actor requested it, before any redirect
site	The site from your input that the URL belongs to, as scheme and host

That run requested /robots.txt, then /sitemap.xml (a sitemap index), then /sitemap_base.xml, and saved 20 URLs, none with a lastmod. With the date filter on, URLs like these are left out unless Keep URLs that have no lastmod is on.

Pricing

This Actor is priced per event, with two charges:

• Actor start: Apify's automatic start event, $0.00005 per start (one event per GB of run memory, minimum one).
• URL saved: one charge for each URL written to the dataset.

Price per URL: $0.20 per 1,000 URLs, which is $0.0002 each.

Never charged: anything listed in SKIPPED (URLs robots.txt disallows, skipped sites and sitemap files, invalid URLs, including addresses with control or invisible characters), duplicates removed within the run, URLs left out by the date filter, anything past Maximum URLs to save or the sitemap limits, and the REPORT and SKIPPED records themselves.

If you set a maximum cost for the run and it is reached, the Actor saves no more URLs and requests no more files, writes REPORT saying the spending limit was reached and how many URLs were ready but not saved, and ends the run without an error. Nothing after the limit is saved or charged. If Apify restarts the run on another server, URLs already in the run's dataset are not saved or charged again. A new run starts from the beginning, so URLs saved by an earlier run are charged again.

robots.txt and site rules

The robots.txt check is always on, and no input turns it off. Before any other request to a host (scheme, host and port), the Actor reads that host's robots.txt and follows the group for SitemapUrlExtractor, otherwise the * group. A group for apify also applies, and a path must pass both. Every skip below is listed in SKIPPED with the reason and is not charged.

Case	What the Actor does
robots.txt disallows a sitemap file, a common sitemap path or a redirect target	Never requests it
robots.txt disallows a page URL listed in a sitemap	Does not save or charge it. Page URLs are never requested either way
A page URL or sitemap is on another host	Reads that host's robots.txt first, and only robots.txt for a page URL. Up to 25 hosts beyond your sites; past that, the URL or file is skipped
Disallow: / for this Actor, even with Allow lines	Skips the whole site
robots.txt answers 401, 403, 429 or a server error, cannot be reached, shows a login or CAPTCHA page, or redirects to another host somewhere other than its robots.txt	Skips the whole site. No other file is requested from it
robots.txt answers another 4xx, such as 404	Treats the site as having no rules
A sitemap answers 401, 403 or 429, shows a login or CAPTCHA page, or redirects to a login page	Stops at the door: no more requests go to that host in this run, and it never tries to get past
Requests to one host	One at a time, at least 0.25 seconds apart, counted from the end of the last one. Up to 5 of your sites are read at once
robots.txt sets a Crawl-delay	Waits that long between requests to that host
The Crawl-delay is over 60 seconds	Skips the whole site rather than crawl it faster

Only a 500, 502, 503 or 504 answer, a timeout or a network error is retried, once, after 2 seconds. The user agent is SitemapUrlExtractor/0.1 (Apify Actor sitemap-url-extractor; reads robots.txt first), and it never borrows another crawler's name. No cookies are stored or sent and no proxy is used. There is no cookie, header, login, proxy or user agent input. On Apify, private and local network addresses are refused before any request, listed in SKIPPED as private_address and not charged.

Your responsibility

You are responsible for submitting only sites you are allowed to access and for following each website's terms of use. The Actor checks robots.txt for you, but it cannot read or judge a site's terms.

Run records

Two records in the run's key-value store, never charged:

Record	What it holds
REPORT	The result, finished, stopped early or input problem, and the stop reason: maxUrls was reached, the spending limit, or the run was aborted. Start and finish times and duration. Input problems and adjustments, and the settings used, including the user agent and the caps. Counts: URLs saved, URLs listed in sitemaps, duplicates removed, URLs left out by the date filter (older, or without a lastmod), URLs disallowed by robots.txt, invalid URLs, URLs not saved at the spending limit, sitemap files fetched, requests made and robots.txt files read. Skips by reason. For each site: the sitemaps from robots.txt and from your input, the common paths tried, files read, URLs saved and the outcome. For each host: the robots.txt status and result, any skip or stop, the Crawl-delay used and the number of requests
SKIPPED	Every skipped site, sitemap file or URL: the address, the site, the stage (input, robots.txt, sitemap, url or site), a reason code such as robots_txt_disallowed, site_disallows_all, robots_txt_server_error, crawl_delay_too_long, access_denied, rate_limited, login_page, captcha, timeout, too_large, unsafe_xml, private_address, invalid_character, max_depth or max_sitemaps, a detail, and where it was found. Split into SKIPPED, SKIPPED-2 and so on every 10,000 entries. Past 50,000 entries the rest are only counted in REPORT. Duplicates and date-filtered URLs are counted in REPORT, not listed here

Limits

Limit	Value
URLs	Maximum URLs to save, 1 to 500,000, 1,000 by default
Sitemap files	Maximum sitemap files to fetch, 1 to 5,000, 200 by default. Files past it are listed as max_sitemaps and never requested
Index depth	Maximum sitemap index depth, 0 to 10, 3 by default. Deeper files are listed as max_depth and not requested
File size	50 MiB per sitemap file, before and after gzip. A larger file is skipped and listed. Gzip is unpacked 1 MiB at a time. Only the first 500 KiB of a robots.txt is read
Formats	XML sitemaps and sitemap indexes, .xml.gz files, answers sent with gzip encoding, and text sitemaps with one URL per line. A broken XML file keeps the URLs read before the error, and the error is listed
XML safety	Sitemaps are read with a safe XML parser that refuses DTDs, entity declarations and external references, so no sitemap can make the Actor expand or fetch anything, and such files are skipped as unsafe_xml
Date filter	A lastmod without a time zone is read as UTC, and 2026 or 2026-10 as the start of that year or month. With the filter on, a URL with no lastmod, or one that cannot be read, is left out unless Keep URLs that have no lastmod is on
Duplicates	Removed across the whole run. Two URLs count as the same when they match after lower-casing the scheme and host and dropping a default port and any # part
Hosts	At most 25 hosts beyond the sites you give
Input lists	Up to 1,000 entries in each list. Only the first 1,000 are used
Redirects	At most 5 per file. Each sitemap redirect is checked against robots.txt again
Memory	256 MB by default, up to 1,024 MB