Sitemap URL Extractor: Every Page with Last-Modified Dates
Pricing
from $0.30 / 1,000 sitemap urls
Get every page a site lists in its sitemaps, with last-modified dates, from robots.txt through sitemap indexes and .gz files. Track new pages of competitors with Only new rows. robots.txt obeyed, no personal data, pay only per URL delivered.
One per line. A domain (example.com) finds the sitemaps through robots.txt; a robots.txt address (https://example.com/robots.txt) reads the sitemaps it lists; a sitemap address (https://example.com/sitemap.xml or .xml.gz) is read directly. Duplicates and invalid entries are skipped and never charged.
For a domain or a robots.txt address: read the Sitemap lines of robots.txt, and when there are none try /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml. Turn off to try only /sitemap.xml.
Stop after this many URLs are delivered. The default is a safe start; raise it, or use 0, to get everything. Set a Maximum total charge in the run options for a hard cost cap.
Text that must appear in the path or query of the URL, for example /blog/ or /products/. Start a line with re: for a regular expression, for example re:/20(25|26)/. A URL is kept when it matches any line. Leave empty to keep all.
Text, or re: and a regular expression, matched in the path or query. A URL that matches any line is dropped and not charged, for example /tag/ or ?replytocom.
Keep only URLs whose last-modified date is on or after this day: YYYY-MM-DD, or a relative value such as 7 days or 2 weeks. URLs without a date are dropped, and sitemap files of an index whose own date is older are not opened.
Stop reading a site after this many URLs (one entry of the list). 0 = no limit.
Stop after this many sitemap files of one site (an index can name thousands). Files inside indexes are followed up to 5 levels deep.
Only count how many rows match your filters and estimate the cost at your price tier. Nothing is delivered and nothing is charged apart from the start event. Turn it off to get the data.
Remember the rows already delivered to you (in a key-value store in your Apify account) and deliver only rows that appeared since the last run with the same state key. The first run delivers everything that matches; later runs cost only the new rows. Ideal for schedules.
Name of the memory used by "Only new rows". Use a different name for each schedule or search (for example stockholm-python) so they do not share one memory.
A webhook address that receives one JSON POST with the number of new rows and links to the dataset: a Slack or Discord incoming webhook, a Make, n8n or Zapier catch hook, or your own endpoint. The message is also in the text (Slack) and content (Discord) fields.
By default the notification is sent only when at least one row was delivered.
