Sitemap URL Extractor Pro avatar

Sitemap URL Extractor Pro

Pricing

from $0.50 / 1,000 url extracteds

Go to Apify Store
Sitemap URL Extractor Pro

Sitemap URL Extractor Pro

Extract every URL from any website's sitemaps: auto-discovery via robots.txt, sitemap indexes, gzipped .xml.gz, TXT and RSS sitemaps, lastmod, hreflang, images, robots.txt conflicts and optional HTTP status codes.

Pricing

from $0.50 / 1,000 url extracteds

Rating

0.0

(0)

Developer

SteadyUtil

SteadyUtil

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

an hour ago

Last modified

Features

Limited permissions

Share

Sitemap URL Extractor Pro: get every URL of any website from its sitemaps

Sitemap URL Extractor Pro turns any domain or sitemap link into a clean list of every page URL the site publishes, with lastmod, changefreq, priority, hreflang alternates, image URLs and (optionally) the live HTTP status of each page. Give it example.com and it finds the sitemaps by itself: robots.txt, sitemap indexes nested several levels deep, gzipped .xml.gz files, plain-text .txt sitemaps and RSS/Atom feeds.

It's built for SEO audits, content inventories, migrations, competitor research and as a fast pre-step for crawlers and AI/RAG pipelines. You get the URLs in seconds without crawling the whole site.

Why use it?

  • Auto-discovery that actually finds sitemaps. It reads Sitemap: lines in robots.txt, falls back to common locations (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt and more), and retries http:// sitemaps over https://, which is a common robots.txt mistake on big news sites.
  • Every format. Sitemap indexes (any depth), urlsets, .xml.gz (with or without gzip headers), TXT sitemaps, RSS and Atom feeds, namespaced XML and CDATA.
  • Rich output. lastmod, changefreq, priority, the sitemap each URL came from, image URLs, hreflang alternates, and Google News and video titles.
  • SEO checks built in. It flags sitemap URLs that robots.txt blocks (a classic SEO bug), and with one switch checks the HTTP status, redirects and final URL of every page.
  • Filters that cut cost. Include/exclude regex patterns, "modified after" date, same-host only, and per-site and total URL limits. You only pay for URLs you keep.
  • A bad site never breaks your run. Dead sitemaps, HTML soft-404 pages, DNS errors and timeouts are recorded in a per-site SUMMARY and the run still succeeds. Retries use exponential backoff and honour Retry-After.
  • Fast and cheap. Plain HTTP only (no browser). Over 1,000,000 URLs from apify.com in about 2 minutes on 512 MB in our tests.

How to use it

  1. Add one or more websites or sitemap URLs (apify.com, https://example.com/sitemap_index.xml, https://site.com/sitemap.xml.gz).
  2. Optionally set Max URLs (your cost cap), filters, or turn on Check HTTP status.
  3. Click Start and download the results as JSON, CSV, Excel or HTML, or use the API.

Input example

{
"startUrls": [{ "url": "https://apify.com" }, { "url": "https://www.nytimes.com" }],
"maxUrls": 5000,
"maxUrlsPerSite": 2500,
"includeUrlPatterns": ["/blog/", "/2026/"],
"excludeUrlPatterns": ["\\.pdf$"],
"lastmodAfter": "2026-01-01",
"checkStatus": false,
"respectRobotsTxt": true
}

Output example

{
"url": "https://www.ebay.com/itm/800940000815",
"lastmod": null,
"changefreq": "daily",
"priority": 1,
"sitemapUrl": "https://www.ebay.com/lst/GTC-0_10.xml.gz",
"sitemapType": "urlset",
"site": "https://www.ebay.com/lst/VIS-0-index.xml",
"host": "www.ebay.com",
"blockedByRobotsTxt": false,
"imageCount": 1,
"imageUrls": ["https://secureir.ebaystatic.com/pictures/aw/pics/stockimage1.jpg"],
"alternates": [],
"news": null,
"videoTitle": null
}

With Check HTTP status on, each item also has statusCode, finalUrl and redirected.

The run also stores a SUMMARY record in the key-value store. For each site it lists the sitemaps processed, the sitemaps that failed and why, whether robots.txt was found, the URL counts and the duration.

FieldDescription
urlPage URL listed in the sitemap
lastmodLast modification date (ISO 8601) if the sitemap provides it
changefreq, prioritySitemap hints, if present
sitemapUrl, sitemapTypeThe sitemap file the URL came from (urlset, feed, text)
site, hostWhich of your inputs it belongs to and its hostname
blockedByRobotsTxttrue if robots.txt disallows the URL even though the sitemap lists it
imageUrls, imageCountImages attached to the URL in an image sitemap
alternateshreflang alternates { hreflang, url }
news, videoTitleGoogle News and video sitemap data
statusCode, finalUrl, redirectedOnly with Check HTTP status

Run it from the API

Replace YOUR_API_TOKEN with your Apify API token. The call returns when the run finishes; then read the dataset.

cURL

curl "https://api.apify.com/v2/acts/steadyutil~sitemap-url-extractor-pro/runs?waitForFinish=300" \
-X POST \
-H "Authorization: Bearer YOUR_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"startUrls": [{ "url": "https://apify.com" }],
"maxUrls": 1000,
"checkStatus": false
}'

Fetch results (use the defaultDatasetId from the run):

curl "https://api.apify.com/v2/datasets/DATASET_ID/items?format=json" \
-H "Authorization: Bearer YOUR_API_TOKEN"

JavaScript

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('steadyutil/sitemap-url-extractor-pro').call({
startUrls: [{ url: 'https://apify.com' }],
maxUrls: 1000,
checkStatus: false,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((item) => item.url));

Python

from apify_client import ApifyClient
client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("steadyutil/sitemap-url-extractor-pro").call(run_input={
"startUrls": [{"url": "https://apify.com"}],
"maxUrls": 1000,
"checkStatus": False,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print([item["url"] for item in items])

The Actor is also available through the Apify MCP server, so AI agents can discover and call it from the Store page.

These SteadyUtil Actors combine well with this one:

  • Website Screenshot Pro — after you have the URL list, capture desktop and mobile screenshots (or PDFs) of every page, or of only the new and changed ones.
  • Website Tech Stack Detector & SEO Audit — take the unique domains from the sitemap and detect each site's CMS, ecommerce platform, analytics and on-page SEO score.

Automation recipes (Make, n8n, Zapier)

Each recipe starts the same way: in Apify Console, save this Actor as a Task with your input, then connect that Task from Make, n8n or Zapier (search for the Apify app). You can also call the API examples above from an HTTP module.

Weekly sitemap diff to Google Sheets

  1. Create a Task with your domain, a lastmodAfter of seven days ago (or no date filter if you want the full list), and Max URLs as a cost cap.
  2. In Apify Console, add a Schedule that runs the Task every Monday.
  3. In Make, n8n or Zapier, trigger on a finished Apify run (or poll the dataset), then upsert each url into a Google Sheet. Keep lastmod, statusCode and blockedByRobotsTxt as columns.
  4. Optional: add a filter so the sheet only gets URLs with statusCode 404 or blockedByRobotsTxt true, and email yourself the new rows.

New blog posts to Slack or email

Use the example Task Extract blog URLs from a sitemap (includeUrlPatterns such as /blog/), schedule it daily, and post any URL whose lastmod is after yesterday to Slack, email or a CMS draft queue.

Feed a crawler or screenshot run

Pass the dataset's url field into Website Screenshot Pro or your own crawler. In n8n, this is two Apify nodes in a row: extract URLs, then screenshot (or audit) them.

Pricing

This Actor uses pay-per-event pricing, so you pay only for results and never for compute time:

EventPrice
url-extracted: one URL in your dataset$0.50 per 1,000 URLs
url-status-checked: HTTP status check of one URL (optional)$1.00 per 1,000 URLs
residential-proxy-mb: only if you turn on Apify residential proxy, per started MB downloaded through it$0.012 per MB

Sites without a sitemap, failed sitemaps and filtered-out URLs cost nothing. Without a proxy or with the default datacenter proxy there are no other charges. Residential proxy traffic costs us about $8/GB, so it's passed on at $0.012/MB, and a typical gzipped sitemap is 0.2–2 MB. Use Max URLs to cap the bill, for example maxUrls: 10000 costs at most $5. Apify Store discounts for higher subscription plans apply automatically.

FAQ

How do I get all URLs of a website from its sitemap in 30 seconds? Paste the domain (for example apify.com) into Websites or sitemap URLs, set Max URLs if you want a cap, and click Start. The Actor finds robots.txt and the sitemaps, then writes every URL to the dataset. Download JSON, CSV or Excel, or use the API examples above. You don't need the exact sitemap URL.

How do I find 404s and broken pages in a sitemap? Turn on Check HTTP status and, if you like, set Max URLs. Each result gets statusCode, finalUrl and redirected. Filter the dataset (or the SEO issues view) for 404, 5xx or unexpected redirects. Try the public example Task Find 404s in your sitemap.

The site has no sitemap. What happens? The site is listed in the SUMMARY record with status no-sitemap and the reason. You aren't charged and the run doesn't fail. Some sites, such as python.org and docs.github.com, genuinely publish no sitemap.

Does it crawl the website pages? No. It only downloads robots.txt and sitemap files, unless you enable Check HTTP status, which sends a lightweight HEAD request per URL.

Can it handle huge sites? Yes. Nested indexes and millions of URLs are processed as a stream with de-duplication. Use Max URLs or Max URLs per site when you only need a sample.

Do I need a proxy? Usually not, because sitemaps are meant for bots. If a site rate-limits datacenter IPs, enable Apify Proxy in the input.

Can I get only new or changed pages? Yes. Set Only URLs modified after to a date and schedule the Actor daily or weekly. Pair this with the Google Sheets recipe above.

Can I use it through the API or with AI agents? Yes. It works with the Apify API, the JS and Python clients, Make, Zapier, n8n and the Apify MCP server. Copy the examples in Run it from the API.

Support and feedback

Sitemap URL Extractor Pro is built and maintained by SteadyUtil, a small publisher of reliable, low-cost web utilities on Apify. Found a site that doesn't work, or need a field we don't output yet? Open an issue in the Issues tab of this Actor (fastest, usually answered within 1–2 business days) or email support@steadyutil.com.

Other SteadyUtil Actors: Website Screenshot Pro and Website Tech Stack Detector & SEO Audit.

This Actor only reads robots.txt and sitemap files, which websites publish specifically so that automated agents can discover their public pages. It collects URLs and sitemap metadata only, never personal data. By default it respects robots.txt for the sitemap files it downloads. You are responsible for using the results in line with the target website's terms and applicable law.