Sitemap Extractor: every URL of a website with lastmod

$0.20 per 1,000 URLs. Pay only for the URLs you get. Errors, missing sitemaps, duplicates and filtered URLs are free. No monthly fee, no API key, no login.

Give it a domain (example.com), any page of a site, a robots.txt or a sitemap URL. The Sitemap Extractor finds the sitemaps, follows nested sitemap indexes and returns one row per page URL with:

• lastmod, changefreq, priority
• alternates (hreflang language versions), imageUrls, videoTitles
• newsTitle and newsPublicationDate (Google News sitemaps), title (RSS/Atom feeds)
• sitemapUrl (the file where the URL was found), domain, path, inputUrl

What it handles that simple sitemap parsers miss:

• Discovery: Sitemap: lines of robots.txt, then /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.txt, with and without www.
• Nested sitemap indexes (up to 10 levels), .xml.gz files (even when the server gzips them twice), plain-text sitemaps, RSS and Atom feeds used as sitemaps
• Huge sitemaps: files are read as a stream, so a 50,000-URL / 50 MB sitemap uses little memory, and the download stops as soon as your limit is reached
• Clear errors per input in the run summary: "no sitemap found", "domain does not resolve", "HTTP 404", "anti-bot page", so you know exactly why a site gave nothing

How much does it cost?

One event: url = $0.0002 (one unique URL delivered).

URLs delivered	Price
100	$0.02
1,000	$0.20
50,000 (a full large sitemap)	$10
10 sites × 5,000 URLs, every week	$10 per week, or ~$1 with onlyNew once the first run is done

Not billed: duplicates across sitemaps, URLs removed by your filters, URLs already delivered in a previous run (onlyNew), sites with no sitemap, unreachable or blocked sites. Set Max URLs or a maximum cost per run in Apify: the actor stops cleanly at your limit.

Input

{
  "urls": ["https://www.gov.uk/sitemap.xml", "crawlee.dev"],
  "maxUrls": 100
}

Field	What it does
urls	Domains, page URLs, robots.txt URLs or sitemap URLs (XML, .gz, index, text, RSS/Atom)
maxUrls	Total limit (default 10,000; 0 = no limit)
maxUrlsPerInput	Limit per domain or sitemap
includePatterns / excludePatterns	Regular expressions or plain text matched on the URL (e.g. /blog/, /tag/)
modifiedAfter	Keep URLs with lastmod on or after a date: 2026-01-01 or 7 days
keepUrlsWithoutLastmod	With modifiedAfter, also keep URLs without lastmod
deduplicate	One row per URL even if it is listed in several sitemaps (default on)
onlyNew, onlyNewMode, stateKey	For scheduled runs: only URLs not delivered before (new), or also those whose lastmod changed (newOrUpdated)
useRobotsTxt, tryCommonPaths, maxDepth, maxSitemapsPerInput	Discovery settings
saveUrlList	Also save a plain-text list of URLs (record URL_LIST)
proxyFallback, residentialFallback, maxResidentialMb	What to do if a site refuses the connection

Output

One dataset item per URL:

{
  "url": "https://www.gov.uk/government/news/miniature-surveillance-helicopters-help-protect-front-line-troops",
  "title": null,
  "lastmod": "2026-08-14T16:41:45Z",
  "changefreq": null,
  "priority": 0.29,
  "domain": "www.gov.uk",
  "path": "/government/news/miniature-surveillance-helicopters-help-protect-front-line-troops",
  "sitemapUrl": "https://www.gov.uk/sitemaps/sitemap_1.xml",
  "sitemapType": "urlset",
  "inputUrl": "https://www.gov.uk/sitemap.xml",
  "alternates": null,
  "imageUrls": null,
  "newsTitle": null,
  "newsPublicationDate": null,
  "videoTitles": null,
  "scrapedAt": "2026-10-17T09:12:03Z"
}

A multilingual page gives "alternates": [{"hreflang": "fr", "url": "https://example.com/fr/page"}, ...].

Dates: lastmod is kept as YYYY-MM-DD when the sitemap gives only a date, and converted to ISO 8601 UTC when it gives a time.

Also in the key-value store:

• URL_LIST: every delivered URL, one per line (up to 200,000), ready for wget -i, a crawler or an LLM pipeline
• SUMMARY: per input, the sitemaps found, the number of sitemap files read, URLs delivered and the exact errors; plus duplicates, filtered and already-seen counts and network traffic

Download the dataset as JSON, CSV, Excel, XML or HTML from the run page or the API.

Use cases

1. SEO audit: find stale and orphan pages

Who: an SEO consultant auditing a client site before a migration. Input:

{ "urls": ["client-site.com"], "maxUrls": 0, "excludePatterns": ["/tag/", "/author/"] }

Output (extract):

[
  { "url": "https://client-site.com/services/audit", "lastmod": "2026-09-30", "priority": 0.8, "sitemapUrl": "https://client-site.com/page-sitemap.xml" },
  { "url": "https://client-site.com/blog/old-guide", "lastmod": "2019-03-02", "priority": 0.3, "sitemapUrl": "https://client-site.com/post-sitemap.xml" }
]

Next: export to CSV, sort by lastmod to list pages not updated for 2+ years, and compare with the crawl or Google Search Console to find pages in the sitemap but not linked (orphans) or linked but missing from the sitemap. 12,000 URLs = $2.40.

2. Competitor content monitoring every morning

Who: a content marketing manager watching what 5 competitors publish. Input (scheduled daily):

{
  "urls": ["competitor-a.com", "competitor-b.com", "competitor-c.io", "competitor-d.com", "competitor-e.co.uk"],
  "includePatterns": ["/blog/", "/guides/"],
  "onlyNew": true
}

Output (extract):

[
  { "url": "https://competitor-b.com/blog/pricing-page-teardown", "lastmod": "2026-10-16", "inputUrl": "competitor-b.com" }
]

Next: an n8n or Make scenario sends the new URLs to Slack and to a Google Sheet. The first run returns the whole back catalogue; after that, a typical day returns 5 to 20 URLs, i.e. less than $0.01 per day.

3. Feed a RAG index or a crawler with the right pages only

Who: an AI engineer building a support chatbot on a product's documentation. Input:

{ "urls": ["https://docs.apify.com/sitemap.xml"], "includePatterns": ["/platform/"], "modifiedAfter": "30 days", "maxUrls": 2000 }

Output (extract):

[
  { "url": "https://docs.apify.com/platform/actors/running", "lastmod": "2026-10-02", "sitemapUrl": "https://docs.apify.com/sitemap_base.xml" }
]

Next: pass the URL_LIST file to a content crawler (or a Markdown converter) and re-embed only the pages changed in the last 30 days instead of re-crawling the whole site. 2,000 URLs = $0.40.

4. E-commerce catalogue tracking

Who: a pricing analyst tracking a retailer's product range. Input: { "urls": ["shop.example.com"], "includePatterns": ["/products/"], "onlyNew": true, "onlyNewMode": "newOrUpdated" } Next: new product URLs (new references) and updated ones (price or stock change pages) go to the price scraper; nothing else is re-scraped.

Use with the API

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("YOUR_USERNAME/sitemap-extractor").call(run_input={
    "urls": ["crawlee.dev"],
    "maxUrls": 500,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item["lastmod"])

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('YOUR_USERNAME/sitemap-extractor').call({
    urls: ['crawlee.dev'],
    maxUrls: 500,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((i) => `${i.url} ${i.lastmod}`).join('\n'));

One HTTP call (sync, returns the items)

curl -X POST "https://api.apify.com/v2/acts/YOUR_USERNAME~sitemap-extractor/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["crawlee.dev"], "maxUrls": 100}'

Integrations: Make, Zapier, n8n, Google Sheets

Example automation (n8n or Make):

1. Schedule trigger, every day at 7:00.
2. Apify: Run actor sitemap-extractor with {"urls": ["competitor-a.com", "competitor-b.com"], "onlyNew": true}.
3. Apify: Get dataset items of the run.
4. Google Sheets: Append row with url, lastmod, inputUrl, then Slack: Send message "N new pages on competitor sites".

On Zapier, use the Apify app ("Run Actor" then "Find dataset items"). You can also connect the run to Google Sheets, Airtable, webhooks or email directly in the Apify Console (Integrations tab).

Use with AI agents (MCP)

The actor is available through the Apify MCP server, so Claude, Cursor, ChatGPT and other AI agents can call it as a tool: "list every blog URL of example.com updated this month" becomes {"urls": ["example.com"], "includePatterns": ["/blog/"], "modifiedAfter": "30 days"}. Point your MCP client to https://mcp.apify.com and add this actor. Output is flat JSON with self-explanatory field names, and the URL_LIST text file is easy to pass to a model.

FAQ

Do I need to know where the sitemap is? No. Give the domain; the actor reads robots.txt and tries the usual paths. If you already know the sitemap URL, give it directly (it is faster).

Does it crawl the website? No. It reads only the sitemaps (and robots.txt), never the pages themselves. That is why it is fast and cheap. A site without a sitemap returns nothing (and costs nothing); use a crawler for those.

Are sitemap index files billed? No. Only the page URLs delivered are billed, not the sitemap files that list other sitemaps.

Some URLs have no lastmod. Why? Many sites do not publish it. The field is then null. With modifiedAfter, these URLs are dropped unless you enable keepUrlsWithoutLastmod.

How do I get only new pages every day? Enable onlyNew and schedule the actor (or a task). The history is stored in your account in the key-value store sitemap-extractor-state, one history per input configuration (or per stateKey).

A site returned nothing. What happened? Open the SUMMARY record: each input lists its errors ("no sitemap found", "HTTP 404", "domain does not resolve", "anti-bot challenge page"). Nothing is charged for it.

Is it legal? Sitemaps are public files that site owners publish precisely so that robots can find their pages. The actor reads only those public files, without login.

Limits

• Sites that protect their sitemap behind a JavaScript challenge or a captcha are reported as blocked; the actor does not try to get around them.
• Requests go out directly from Apify. If a site refuses them (HTTP 403/429), the actor retries through Apify datacenter proxies, then through residential IPs, capped by maxResidentialMb (20 MB by default).
• The plain-text URL_LIST holds up to 200,000 URLs; the dataset has no limit.
• onlyNew remembers up to 500,000 URLs per history.
• lastmod, priority and changefreq are what the site declares; some CMSs set every lastmod to today's date.
• Sitemap files larger than 80 MB compressed or 250 MB decompressed are truncated (the sitemap protocol limit is 50 MB).