Sitemap Extractor: All URLs from Any Website, $0.50/1K

This sitemap extractor gets every page URL of a website from its XML sitemaps. Enter a bare domain such as gov.uk or a sitemap address, and it finds the sitemaps through robots.txt, falls back to the usual addresses (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml), follows sitemap indexes, unpacks .gz files and also reads plain text sitemaps. Each URL comes with last modified date, change frequency and priority.

Cost: $0.0005 per URL ($0.50 per 1,000). Websites without a sitemap, unreadable sitemaps and invalid entries are free. No login or API key.

How to extract all URLs from a sitemap

1. Enter websites or sitemap addresses under Websites or sitemaps.
2. Optionally set URL contains to keep one section such as /blog/ or /docs/, and URLs per website.
3. Click Start, then open the Sitemap URLs table or download the list.

The form opens with GOV.UK and one NASA post sitemap, 50 URLs each. It takes a few seconds and costs about 5 cents. Typical uses are SEO audits (every indexable URL with its last modified date), content inventories and migration checks (compare the URL lists of an old and a new site), and URL lists for other Actors such as a scraper, a screenshot or a Markdown Actor.

How much does the sitemap URL extractor cost?

Event	Price
Sitemap URL (sitemap-url)	$0.0005 ($0.50 per 1,000)
Website without a sitemap, unreadable sitemap or invalid entry	Free

Apify adds only its small standard fee per run start. You can set a spending limit on any run, and the Actor stops cleanly when it is reached.

Example: a full inventory of a 40,000-page store costs 40,000 x $0.0005 = $20.

Input

{
  "startUrls": [
    "https://www.gov.uk",
    "https://www.nasa.gov/wp-sitemap-posts-post-1.xml"
  ],
  "maxUrlsPerSite": 50
}

Output

One row per result:

{
  "input": "https://www.gov.uk",
  "url": "https://www.gov.uk/browse/benefits",
  "lastModified": "2026-09-20T10:15:00.000Z",
  "changeFrequency": null,
  "priority": null,
  "sitemapUrl": "https://www.gov.uk/sitemaps/sitemap_1.xml",
  "status": "ok",
  "charged": true
}

status is ok for URLs, which are charged. A website where no sitemap could be read gives one free row with no_sitemap, no_urls or robots_disallowed and the reason. A STATS record counts URLs, free rows and requests.

Run it every day

Use this to get each new URL once.

1. Enter your websites or sitemap addresses and the options you need.
2. Turn on Only new since the last run and give the monitor a Monitor name.
3. Click Save as a new task.
4. Open Schedules, create a schedule, pick the task and choose daily.

The first run returns the URLs and sets the baseline. Each later run returns only URLs that this monitor did not deliver before. URLs delivered before are not charged. URLs per website still limits each run: the URLs that were not delivered yet come first, and the next run returns the rest. A page whose last modified date changes is not delivered again. The memory is in your own Apify key-value store named sitemap-monitor- followed by the Monitor name. It holds ids only. Use one Monitor name for each task.

Example: 20 websites that add 300 new pages in total a day cost 300 x $0.0005 = $0.15 a day (the 300 pages are an example, not a measured value). A run without new URLs costs only the small Apify run start fee.

Bulk lists

Use a long list of websites or sitemap addresses from an Apify Dataset or from a file. Pick the Dataset under Dataset with the list and write the name of its field under Field in the dataset. Leave the field name empty to use the first text field. Or paste the web address of a .txt, .csv or .json file under List file URL. A .csv file uses its first column. A header row is skipped. The file can be up to 10 MB. Entries from the Dataset and the file are added to the ones typed under Websites or sitemaps, and duplicates are removed. The price per result does not change.

Related Actors

• Broken Link Checker: Use it to find broken internal and external links on a website.
• Web Page to Markdown for AI: Use it to turn web pages into clean Markdown for LLMs, RAG and AI agents.
• AI Crawler Access Audit: Use it to check which AI crawlers a website allows or blocks in robots.txt.
• Website Screenshot API: Use it to capture full-page screenshots or PDFs of a list of URLs.

FAQ

Does it crawl pages? No. It reads sitemap files only, which is fast and cheap. Pages that are missing from the sitemap are not found.

Does it respect robots.txt? Yes. Sitemaps are read only where robots.txt allows the token DonMangu-SitemapExtractor or all crawlers.

Which formats work? XML sitemaps and sitemap indexes, gzipped files, image sitemap entries (counted) and plain text sitemaps.

Can I limit it to one section? Set URL contains to a path such as /blog/ or /docs/.