Sitemap Extractor: Every Page URL, with Change Alerts

See every page a website publishes, straight from its sitemaps, or schedule it and get only the pages that were added or removed (and, if you want, updated) since the last run.

• Every page URL in one table. Give it a website and it finds the sitemaps itself: robots.txt first, then /sitemap.xml and /sitemap_index.xml. Sitemap indexes, gzip files (.xml.gz) and plain-text sitemaps all work.
• Monitoring built in. Turn on Only new and removed pages, schedule it, and you pay only for changes: new pages and gone pages. Turn on Also report updated pages to get changed pages too (their last-modified date moved, with the previous date); sitemap files that re-date most of their pages at once are ignored, so you do not pay for noise.
• Safe by design. A page is reported removed only when the sitemap it was listed in was read in full. A failed sitemap, a robots.txt hiccup or your URL cap never produces a flood of false "gone" rows.
• Alerts out of the box. Every row has a one-line summary, and the run's status message lists the first changes, so the built-in Slack integration posts "New page on competitor.com: …" with no extra tools.

It reads only sitemaps, the files websites publish so that their pages can be found. No browser, no proxies, no page downloads, so runs are fast, cheap and light on the sites you watch.

Sample output

{
  "status": "new",
  "site": "docs.apify.com",
  "url": "https://docs.apify.com/api",
  "lastModified": null,
  "changeFrequency": "weekly",
  "priority": 0.5,
  "sitemapUrl": "https://docs.apify.com/sitemap_base.xml",
  "firstSeenAt": "2026-10-29T12:00:00Z",
  "scrapedAt": "2026-10-29T12:00:00Z"
}

lastModified is filled when the site's sitemap has <lastmod> dates (many do; this one does not), normalised to UTC ISO 8601.

Who uses it

• Marketers and SEO teams: see the moment a competitor launches a landing page, a pricing page, a product or a blog post, and which pages they quietly remove.
• Content and SEO audits: a complete list of a site's indexed pages, with last-modified dates, in one click.
• Ecommerce and market research: new product URLs from a retailer's product sitemap, every day.
• Developers: feed fresh URLs into another crawler, a search index or an LLM pipeline, without crawling the whole site.
• AI agents: a narrow, predictable tool for "what pages does this site have, and what changed?"; works through the Apify MCP server.

How to use it

1. Paste websites into Websites or sitemap URLs, one per line, for example https://docs.apify.com. You can also paste a single sitemap, for example https://example.com/sitemap-products.xml.
2. Optionally keep only part of a site with URL must contain, for example /blog/.
3. Click Start. Download the results as JSON, CSV or Excel, or read them through the API.

To monitor: turn on Only new and removed pages since the last run, save the input as a task and add a daily schedule. The first run returns every page marked new (or, with Start quietly, just records them for free); after that you get only changes. Connect the task to Slack, email, Google Sheets, Make, Zapier or n8n to get alerts.

Input examples

All pages of a website:

{ "websites": ["https://docs.apify.com"] }

Daily monitor of three competitors' blogs and landing pages, starting quietly:

{
  "websites": ["competitor-one.com", "competitor-two.com", "https://competitor-three.com/sitemap.xml"],
  "onlyNew": true,
  "skipExistingOnFirstRun": true,
  "urlContains": ["/blog/", "/solutions/", "/pricing"]
}

Which inputs work

You enter	What happens
https://docs.apify.com, docs.apify.com, or any page of the site	The site's sitemaps are found through robots.txt, then /sitemap.xml and /sitemap_index.xml
A sitemap URL ending in .xml, .xml.gz or .txt, or with sitemap in its path	That sitemap and any sitemaps it lists

Inputs that cannot be used produce a free row with status: invalid_input, not_found or error and a plain-English error. The same site entered twice is read and charged once, and a URL listed in several sitemaps is returned once.

Output

One row per page: status, summary (one readable line, e.g. New page on docs.apify.com: https://docs.apify.com/api), site, url, lastModified, changeFrequency, priority, sitemapUrl, firstSeenAt (monitoring) and scrapedAt. status is current in a normal run, or new, gone and (with Also report updated pages) changed when monitoring. changed rows also have previousLastModified. Inputs that produced no data get a free row with status invalid_input, not_found, error or skipped (not checked because your spending limit was reached).

A RUN_SUMMARY record in the run's key-value store lists every site with how many sitemaps and URLs were read, the site checks charged, any sitemap that failed, whether the read was complete or cut by your URL cap, and any sitemap files whose dates were ignored as a re-stamp.

Get alerts in Slack, Google Drive, Make, Zapier or n8n

1. Turn on monitoring, click Save as a new task, and add a daily schedule to the task.
2. On the task's Integrations tab, pick Slack, connect your workspace, choose the Run succeeded event and a channel, and paste this as the message:

*{{resource.statusMessage}}*
<https://console.apify.com/storage/datasets/{{resource.defaultDatasetId}}|Open all rows>

The status message reads like Checked 3 site(s), saved 4 URL row(s) (3 new, 0 changed, 1 gone). New page on competitor.com: https://competitor.com/pricing; … so the alert itself tells you what changed.

• Google Drive: the Upload results to GDrive integration saves each run's rows as CSV or JSON.
• Make, Zapier, n8n: use the Apify app's "Actor run finished" trigger and map the summary field into a message, a sheet row or a CRM note.
• Webhooks: send the run to your own endpoint and read the dataset through the API.

Pricing

Pay per event, no platform usage charges on top:

Event	Price
Page URL returned	$0.50 per 1,000 URLs
Website or sitemap checked	$1.00 per 1,000 site checks: one per site, plus one more per started 50,000 URLs read after the first 50,000

Rows for inputs that could not be used, sites with no sitemap and sites that failed are free. A site check is charged after its rows are delivered, so if your spending limit stops the run before any of a site's rows go out, that check is free.

• One-off export: a documentation site with about 4,200 pages costs about $2.10 ($2.10 for the URLs plus $0.001 for the check).
• Daily monitoring: 20 competitor sites checked every day with about 50 new or removed pages a day in total costs about $0.045 a day, about $1.35 a month (20 checks for $0.02 plus 50 rows for $0.025), because you pay only for changes. With Start quietly, the first run costs $0.02 instead of paying for every existing page.
• A big shop: a daily monitor of a 100,000-page site (with Max URLs per site at 100,000) pays 2 site checks a run, $0.002 a day when nothing changed, plus $0.0005 per new or removed page. A URL filter on that site scans at most 10 times the cap, so a filtered 1,000-URL cap pays at most one check for its 10,000 URLs read.

Set a maximum cost per run in the run options and the Actor stops cleanly when it reaches it: sites it could not check get a free skipped row, and when monitoring, changes it could not deliver come on the next run. Rows it delivered are never charged twice.

FAQ

Is this allowed? Sitemaps are published so that search engines and other tools can find a site's pages; the Actor reads only those files. It follows Disallow rules in robots.txt for the sitemap locations it guesses, reads one sitemap at a time per site, and never downloads the pages themselves. Page URLs are not personal data. A site's own terms can still forbid automated monitoring, so watch only sites you have a lawful reason to follow, do not use it on sites that ask robots to stay out, and follow the terms of any service you send results to.

Why do I get no changed rows? Updated pages are opt-in: turn on Also report updated pages. They are off by default because many sites re-date pages without changing them: in a test on three Shopify stores, a run one minute after another saw 2,338 pages with a new <lastmod> (1,072 of 1,318 on one store), and you would have paid for every one. Turning it on starts a separate monitoring memory, so its first run is a first run again (use Start quietly). Many sitemaps also have no <lastmod> dates at all, so only new and removed pages can be seen on those.

What if a site has a huge sitemap? Max URLs per site (default 10,000) caps cost, and sitemap files over 60 MB are skipped. With URL must contain or URL must not contain, a site scans at most 10 times the cap. When a site is cut short, that run reports no removed pages. A capped monitor sees a different slice of a big site when its sitemap is reordered, so a page it has not seen before is reported new only when its <lastmod> is after the previous run; the others are recorded for free (the status message counts them), and pages it has not seen for 30 days are forgotten, so its memory stays small. For complete monitoring, set the cap above the site's size, or narrow it with URL must contain to the part you care about.

Why did a run stop early? To keep runs fair and cheap, a run stops checking new sites when your spending limit is reached (a site is read only when the limit still covers one site check and one URL), when 50 inputs in a row fail or have no sitemap (a dead list or an outage), or when it has worked a long time without producing any paid results (a safety limit for runs that only meet errors and timeouts). The sites it did not check share one free skipped row and are checked on the next run; when monitoring, pages it found but did not deliver come back on the next run, and nothing is charged twice. A run reads at most 1,000 websites; split longer lists across tasks. One request may take at most 60 seconds, a host that fails 10 times in a row is left alone for the rest of the run, and the Actor sends at most 2 requests at a time to one host. If one Apify account's runs keep producing nothing to charge (for example a list of dead sites run again and again), new runs from that account stop early for up to 24 hours, with a free row that says so; normal use never gets near this.

Why was my URL refused? The Actor reads only public websites. Addresses on private, local or reserved networks (localhost, 127.0.0.1, 10.x, 192.168.x, 169.254.169.254, .internal names and other spellings of them) are refused with a free invalid_input row, and so are sitemaps and redirects that lead there.

What if a site re-dates every page on each deploy? With updated pages on, a sitemap file in which half or more of its known pages (at least 50) get a new <lastmod> at once is treated as a re-stamp: none of its pages are reported, the new dates are remembered, and RUN_SUMMARY lists the file under restampedSitemaps (the status message says how many dates were ignored). The same test is applied to the whole site, for sites with many small sitemap files. Real edits in other sitemap files are still reported. You are not charged for thousands of fake changes.

Why is the first monitoring run bigger? Nothing has been seen yet, so every page counts as new. Turn on Start quietly to skip that. Each saved task, and each combination of filters, keeps its own memory, and normal (non-monitoring) runs never touch it.

What if a sitemap is briefly empty? If a site that had pages suddenly lists none, the Actor waits for a second empty check before reporting them all removed.

Can I use it from code or an AI agent? Yes: through the Apify API, the Python and JavaScript clients, or the Apify MCP server. Inputs and outputs are stable and documented above.

Support

Open an issue on the Issues tab with the website you tried; I aim to answer within a day.