Sitemap URL Extractor & Sitemap Scraper - All Website URLs

What does Sitemap URL Extractor do?

Sitemap URL extractor and sitemap scraper: get all URLs of a website (lastmod, images, videos, hreflang) from robots.txt, sitemap indexes and .gz files, with optional HTTP status checks and change detection. $0.50 per 1,000 URLs on the Free plan, from $0.15 on Business and higher.

Get all URLs of a website

It returns every URL a website publishes in its sitemaps, together with the metadata the site provides: last modified date, change frequency, priority, images, videos and language alternates (hreflang). Optionally it checks each URL's HTTP status and tells you what changed since the last run. Need only the list? Turn on Output URLs only for one url column.

• 🔎 Finds sitemaps for you: give it https://example.com and it reads robots.txt, then tries the usual locations (/sitemap.xml, /sitemap_index.xml, WordPress /wp-sitemap.xml, /sitemap.txt), also when the sitemaps listed in robots.txt cannot be read. XML, .xml.gz and plain-text sitemaps (one URL per line) are supported
• 🗂️ Follows sitemap indexes and nested sitemaps automatically
• 🗜️ Handles compressed .xml.gz sitemaps
• 🧹 Removes duplicates across sitemaps
• 🌐 Many websites in one run: paste a plain list, one per line (bare domains like example.com are fine). Every URL row has startUrl and input, so you can group results by website, and Max URLs per website gives each site its share
• 🧾 A clear row for every website without a sitemap (not charged): it lists every location tried with its HTTP status, and says blocked when the site refused our requests (HTTP 401/403/429) instead of not_found
• 🎯 Filters: include/exclude URL patterns and "modified after" date, e.g. only blog posts updated this month
• 🚦 Status check (optional): httpStatus, statusClass (ok, redirect, broken, restricted, server-error, error), finalUrl and redirected for every URL, to find sitemap entries that are broken or redirect
• 🔔 Change detection (optional): each URL is marked new, updated or unchanged compared with the previous run, and URLs that disappeared are listed as removed
• 🖼️ Image and video sitemaps: image title and caption (imageDetails) and video entries (videos: title, description, thumbnail, content/player URL, duration)
• 📰 Google News sitemaps: reads newsTitle and newsPublicationDate from each news entry. When lastmod is missing and the publication date is valid, it uses that date with lastmodSource: "news" for filtering and change detection.
• 🕒 Normalised dates: lastmodIso in ISO 8601 UTC next to the original lastmod
• 📅 Missing dates filled in: a valid Google News publication date fills a missing sitemap lastmod first. With the status check on, URLs still missing a date can use the page's Last-Modified header (lastmodSource: "header"). Change detection uses the chosen date
• ⚡ Fast and cheap: thousands of URLs in seconds, $0.50 per 1,000 URLs, down to $0.15 on Apify's Business plan and higher

For a news URL without sitemap lastmod, the publication date can change which URLs pass modifiedAfter and which URLs count as changed in a named monitor. Filtered and unchanged URLs are not output or charged; an updated URL returned by Only output changes is charged at the existing URL rate.

Who is it for?

• SEO: list all indexable pages, check what changed recently, compare sitemaps with what is actually crawlable
• Scraping & RAG pipelines: get the full URL list first, then feed it to a crawler or to our Website to Markdown Crawler
• Content audits and migrations: inventory of every page on a site
• Monitoring: run it on a schedule with Compare with the previous run to get only new, updated and removed pages

How much does it cost?

Price per 1,000 by Apify plan:

Event	Free	Starter	Scale	Business and higher
URL	$0.50	$0.40	$0.25	$0.15
URL status check (optional)	$0.30	$0.30	$0.27	$0.24

Example: a website with 50,000 sitemap URLs costs 50,000 × $0.0005 = $25.00 on the Free plan ($7.50 on Business); status checks for all of them add $15.00 on the Free plan.

Google News metadata and its date fallback have no extra charge or event.

No start fee. Websites without a readable sitemap and invalid input lines are not charged. Status checks that cannot reach the URL at all (DNS or connection errors) are not charged. With Only output changes, unchanged URLs are not output and not charged (their status checks are not charged either).

For comparison (checked September 2026): Apify's own Sitemap Extractor charges per 1,000 URLs $0.50 on the free plan, $0.30 on Starter, $0.20 on Scale and $0.10 on Business and higher, with a status check included. Without status checks this Actor costs the same on the Free plan and more on paid plans ($0.40 vs $0.30 on Starter, $0.25 vs $0.20 on Scale, $0.15 vs $0.10 on Business); with status checks on it costs more on every plan (e.g. $0.39 vs $0.10 on Business). What you get in addition: image and video details, hreflang alternates, modifiedAfter filtering and change detection between runs.

Control your cost

• What is charged: each URL returned, and each status check when Check each URL's HTTP status is on.
• What is free: websites without a readable sitemap, invalid or empty input, timed-out websites (the URLs found before the limit are charged, the error row is not), URLs dropped by your filters, unchanged URLs with Only output changes, status checks that could not reach the URL, and the "no changes" row.
• Max URLs caps the whole run (default 10,000 when you call the API without it: at most $5 on the Free plan, $1.50 on Business and higher). With several websites, Max URLs per website stops one big site from using it all.
• At the start, the run logs its worst case (e.g. Plan: 10000 URLs × $0.0005 = at most $5) and warns when that is more than your maximum charge per run.
• When the maximum charge per run is reached, the run stops, keeps everything found so far, and says so in the status message. The SUMMARY record then has status: "LIMIT_REACHED" and notProcessed (how many websites were not started, and up to 100 of them). The same notProcessed appears when Max URLs runs out before the last website.
• Time limit per website (Advanced settings, default 300 seconds): reading one website stops after this long. The URLs found before the limit stay in the results (charged); the website gets a row with errorType: "timeout" (not charged), and the next website starts.
• Parallel websites (Advanced settings, default 5): several websites are read at the same time, so one slow website does not hold up the others. Each website keeps its own time limit; status checks share the Parallel status checks slots. With several websites in parallel, Max URLs goes to whichever websites list URLs first: set Max URLs per website to share it evenly.

How to use it

1. Enter one or more websites (or direct sitemap URLs), one per line.
2. Optional: set Max URLs, include/exclude patterns, Only URLs modified after, or Output URLs only (just the url column, same price).
3. Optional: turn on Check each URL's HTTP status and/or Compare with the previous run.
4. Click Start and export the list as CSV, Excel or JSON.

Input example

{
    "urls": ["docs.apify.com", "https://yoast.com/post-sitemap.xml"],
    "maxUrls": 10000,
    "maxUrlsPerSite": 5000,
    "excludeUrlPatterns": ["**/changelog/**"],
    "modifiedAfter": "2026-09-01"
}

Output example

A real result from https://docs.apify.com (this site gives no lastmod, so it is null):

{
    "url": "https://docs.apify.com/api",
    "lastmod": null,
    "lastmodIso": null,
    "changefreq": "weekly",
    "priority": 0.5,
    "images": [],
    "imageDetails": [],
    "videos": [],
    "alternates": [],
    "sitemap": "https://docs.apify.com/sitemap_base.xml",
    "site": "https://docs.apify.com",
    "startUrl": "https://docs.apify.com",
    "input": "docs.apify.com",
    "lastmodSource": null,
    "success": true,
    "charged": true
}

startUrl is the website from your input (with https:// added), input the line exactly as you entered it.

A website without a sitemap (example.com, real output, shortened). One row per such website, not charged:

{
    "input": "example.com",
    "startUrl": "https://example.com",
    "url": null,
    "success": false,
    "errorType": "not_found",
    "error": "No sitemap found for https://example.com. Tried: https://example.com/robots.txt (HTTP 404); https://example.com/sitemap.xml (HTTP 404); ... (not charged)",
    "sitemapsTried": [
        { "url": "https://example.com/robots.txt", "httpStatus": 404, "result": "robots" },
        { "url": "https://example.com/sitemap.xml", "httpStatus": 404, "result": "failed" }
    ],
    "charged": false
}

errorType is blocked when the site answered HTTP 401, 403, 418, 429 or a bot-check page (HTTP 202); try a proxy under Advanced settings. It is network or timeout when the site could not be reached, invalid_input for input lines that are not a website, and no_input when the list is empty. When some websites fail, SUMMARY.status is PARTIAL_RESULTS. Input mistakes never fail the run: you get these free rows and the run ends normally.

A sitemap without dates, with Check each URL's HTTP status on (https://pages.github.com/sitemap.xml, real output, shortened). The date comes from the page's Last-Modified header:

{
    "url": "https://pages.github.com/themes/",
    "lastmod": "Wed, 13 Aug 2025 12:02:54 GMT",
    "lastmodIso": "2025-08-13T12:02:54.000Z",
    "lastmodSource": "header",
    "httpStatus": 200,
    "statusClass": "ok"
}

lastmodSource is "sitemap" when the sitemap gave the date, "news" when a valid news:publication_date filled a missing date, "header" when it came from Last-Modified, and null when none did. News entries also have newsTitle and newsPublicationDate; ordinary entries do not gain these fields. A news date never replaces an existing sitemap lastmod. Headers that just say "now" (common on dynamic pages) are ignored, because they would mark every URL as updated on every run.

A Google News entry without a sitemap lastmod (illustrative row):

{
    "url": "https://news.example/article",
    "lastmod": "2026-10-24T08:00:00Z",
    "lastmodIso": "2026-10-24T08:00:00.000Z",
    "lastmodSource": "news",
    "newsTitle": "Markets & Science",
    "newsPublicationDate": "2026-10-24T08:00:00Z"
}

A site with dates and images (https://yoast.com/post-sitemap.xml):

{
    "url": "https://yoast.com/how-to-remove-www-from-your-url/",
    "images": ["https://yoast.com/app/uploads/2007/04/Remove_www_FI.png"],
    "imageDetails": [{ "loc": "https://yoast.com/app/uploads/2007/04/Remove_www_FI.png", "title": null, "caption": null }]
}

lastmod keeps the site's own format (e.g. 2026-09-21T14:36:11+00:00); lastmodIso is the same moment in UTC (2026-09-21T14:36:11.000Z). A video entry in videos looks like { "title", "description", "thumbnailLoc", "contentLoc", "playerLoc", "durationSecs", "publicationDate" }.

Monitoring: status codes and changes since the last run

{
    "urls": ["https://docs.apify.com"],
    "includeUrlPatterns": ["https://docs.apify.com/academy/**"],
    "checkStatus": true,
    "compareWithPrevious": true,
    "onlyChanges": true,
    "stateKey": "apify-academy"
}

Schedule this input (for example daily). Each run compares with the previous run of the same Monitor name (stateKey) and outputs only what changed. Real output from our test (we edited the saved state to simulate an update):

{
    "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description",
    "lastmod": null,
    "changefreq": "weekly",
    "priority": 0.5,
    "sitemap": "https://docs.apify.com/sitemap_base.xml",
    "site": "https://docs.apify.com",
    "changeType": "updated",
    "previousLastmod": "2020-01-01T00:00:00.000Z",
    "httpStatus": 200,
    "statusClass": "ok",
    "finalUrl": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description",
    "redirected": false
}

A URL that disappeared from the sitemap:

{ "url": "https://docs.apify.com/academy/this-page-was-removed", "changeType": "removed", "previousLastmod": null }

• The first run marks every URL as new.
• A valid Google News publication date fills a missing lastmod before the status check and is used for change detection. With status checks on, URLs still without a date are compared by their Last-Modified header. Differences of up to 2 minutes count as unchanged (CDN servers can report the same file a second apart); a real edit shows up as updated.
• removed URLs are reported only when every sitemap was read and no limit (Max URLs, spending limit) cut the run short; otherwise the previous list is kept for next time.
• If nothing changed, the dataset has one row saying so (not charged).
• The SUMMARY record in the key-value store has status (SUCCESS, PARTIAL_RESULTS, FAILED, NO_RESULTS or LIMIT_REACHED), sites (one entry per website: status, urls, sitemapsRead, and the error if any), counts per change type and per status class. In our test: "changes": {"new": 1, "updated": 1, "unchanged": 194, "removed": 1}.
• Feed the new and updated URLs to Website to Markdown Crawler to keep a knowledge base up to date.

Tips

• If a site has no sitemap at the usual places, pass the sitemap URL directly (you can often find it in robots.txt).
• To get all URLs of big sites, raise Max URLs (10,000 by default for API calls, 1,000 in the Console form); the run stops at your maximum charge per run if you set one.
• Calling the API or n8n? Use the plain urls list. The older startUrls field is still accepted, but Apify rejects the whole run (HTTP 400) when it holds a bare domain or a blank line.
• removed URLs are reported only when every website was read in full: a failed website or Max URLs per website cutting a site short keeps the previous list for next time.
• Combine with Website to Markdown Crawler or SEO Audit Crawler (both by TidyTools) to process the URLs.

Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/sitemap-url-extractor) to Claude, Cursor or any MCP client, then ask e.g. "List all blog URLs in yoast.com's sitemap that were updated this month and check which ones are broken."

Minimal input:

{ "urls": ["docs.apify.com"], "maxUrls": 1000 }

Failed items are not charged and carry an errorType (e.g. not_found when no sitemap exists, blocked when the site refuses our requests).

Use it from code and integrations

Run it from your own code with the Apify API. This call waits for the run and returns the results as JSON (replace YOUR_TOKEN with your Apify API token):

curl -X POST "https://api.apify.com/v2/acts/tidytools~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://docs.apify.com"],"maxUrls":1000}'

The synchronous endpoint waits up to 5 minutes. For bigger runs, start the run with POST https://api.apify.com/v2/acts/tidytools~sitemap-url-extractor/runs and read the dataset when it finishes, or use the apify-client package for JavaScript or Python.

Schedules and integrations: run it daily or weekly with Apify Schedules, get a webhook when a run finishes, or send the results to Zapier, Make, n8n, Google Sheets, Slack and other apps with Apify integrations. Results can be exported as JSON, CSV, Excel or XML.

Limitations

• Only URLs that the site lists in its sitemaps are returned. Pages missing from the sitemap are not discovered (use a crawler for that).
• Some sites block automated access to their sitemaps. Under Advanced settings you can add an Apify proxy (billed to your Apify account).
• Last-Modified is used only when Check each URL's HTTP status is on (the header comes from that request) and neither sitemap lastmod nor a valid Google News publication date is available. Many dynamic sites send no usable header; their lastmod stays null.

FAQ

Does the Actor collect usage data? The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

Can it get all URLs of a website? It returns every URL the site lists in its sitemaps: give it a website and it reads robots.txt, tries the usual locations, follows sitemap indexes and handles .xml.gz files. Pages missing from the sitemap are not discovered (use a crawler for that). For big sites, raise Max URLs (10,000 by default for API calls, 1,000 in the Console form).

What happens if a website has no sitemap? You get one row for that website, not charged, listing every location tried with its HTTP status; errorType is blocked when the site answered HTTP 401, 403 or 429 instead of not_found. If the sitemap is somewhere unusual, pass its URL directly, and for blocked sites you can add a proxy under Advanced settings.

Why did a big site return nothing? Some sites (for example Wikipedia, Stack Overflow, The Guardian, IMDb) refuse requests from automated clients, even for their sitemaps. You get one free row with errorType: "blocked" and the HTTP status of every location tried; adding an Apify proxy under Advanced settings may help. The Actor identifies itself honestly (TidyToolsSitemapBot) and does not try to disguise itself as a browser.

What is charged and what is free? Each URL returned costs $0.50 per 1,000 on the Free plan, down to $0.15 on Business and higher, with no start fee; Check each URL's HTTP status is an optional extra event. Websites without a readable sitemap, invalid input lines, URLs dropped by your filters, unchanged URLs with Only output changes, and status checks that could not reach the URL are free.

Can I run it on a schedule to track new and removed pages? Yes. Turn on Compare with the previous run, give it a Monitor name, and schedule it with Apify Schedules (for example daily): each URL is marked new, updated or unchanged, and URLs that disappeared are listed as removed. With Only output changes, you get only what changed, and a run with no changes outputs one row saying so (not charged).

Related Actors

More tools by TidyTools that work well with this one:

• Website Markdown Crawler: crawl a whole site or docs portal into clean Markdown for LLMs and RAG
• SEO Audit Crawler: audit titles, links, speed and structured data across a whole site
• Bulk URL Status Checker: check HTTP status and redirects for a list of URLs
• Broken Link Checker: find 404 and dead links on any website

Support

Open an issue in the Issues tab with the website URL. Issues are checked regularly.

Found this useful? A short review on the Store helps other users find it.