Sitemap URL Extractor: All Pages of a Website avatar

Sitemap URL Extractor: All Pages of a Website

Pricing

$2.00 / 1,000 site mappeds

Go to Apify Store
Sitemap URL Extractor: All Pages of a Website

Sitemap URL Extractor: All Pages of a Website

List every URL a website publishes in its sitemaps: finds them via robots.txt, follows sitemap indexes and .gz files, returns lastmod dates, recently updated pages and counts by section.

Pricing

$2.00 / 1,000 site mappeds

Rating

0.0

(0)

Developer

Tudor Lane

Tudor Lane

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

12 hours ago

Last modified

Features

Limited permissions

Share

Use when: an SEO, crawl or research agent needs the full list of pages a website publishes in its sitemaps, with lastmod dates and counts by section, without crawling the site.

Try this input (also the default, just click Start)

{"requests": [{"url": "https://www.gov.uk", "max_urls": 500, "max_sitemaps": 3}, {"url": "https://docs.apify.com", "max_urls": 500}]}

Sample output (real run, 16 Nov 2026, trimmed)

{"site": "https://www.gov.uk", "found_via": "robots.txt", "sitemaps_read": 2, "sitemaps_pending": 35, "url_count": 500, "truncated": true, "sections": {"/government": 303, "/employment-tribunal-decisions": 76, "/hmrc-internal-manuals": 52}, "newest_lastmod": "2026-11-09T09:27:06+00:00", "recently_updated": [{"url": "https://www.gov.uk/government/news/princes-trust-celebrates-40th-anniversary", "lastmod": "2026-11-09T09:27:06+00:00", "priority": "0.29"}], "error": null}
{"site": "https://docs.apify.com", "found_via": "robots.txt", "sitemaps_read": 2, "url_count": 500, "truncated": true, "sections": {"/api": 295, "/academy": 202}, "newest_lastmod": null, "error": null}

Options

  • Pass a site URL (robots.txt Sitemap: lines, then common paths) or a sitemap / sitemap-index URL directly.
  • include is an optional regex to keep only matching URLs (e.g. /blog/). Follows sitemap indexes and .gz sitemaps.
  • max_urls (default 5000, cap 50,000) and max_sitemaps (default 20, cap 200) keep large sites cheap. truncated is true when a cap was hit.
  • Up to 200 sites per run in requests. No API key needed.

Pricing: pay per event, $0.002 per site mapped. Errors are not charged. No subscription.

Limits & failure modes

  • It only reads the site's public robots.txt and sitemap files; it does not crawl pages, and sites with no sitemap return an uncharged error.
  • Caps apply: this default run stopped at 500 URLs on both sites (truncated: true). Some sitemaps have no lastmod (docs.apify.com in the sample).
  • Empty or invalid URLs return an uncharged error (url must be http(s)://...). Private/local addresses are blocked.

Last tested: 16 Nov 2026.

Use from AI agents (MCP)

  • Apify MCP server: add https://mcp.apify.com/?tools=tudorlane/sitemap-url-lister to Claude, Cursor, ChatGPT, VS Code or any MCP client.
  • REST: POST https://api.apify.com/v2/acts/tudorlane~sitemap-url-lister/run-sync-get-dataset-items?token=YOUR_TOKEN with the input JSON.