Sitemap Extractor: All URLs of a Website with Lastmod

Get every URL of a website from its sitemaps in seconds, without crawling the site. Enter a domain and the Actor finds the sitemaps itself (robots.txt and the usual paths), follows sitemap indexes, unpacks .xml.gz files and returns each URL with its last-modified date, change frequency, priority, images and hreflang language versions.

• ✅ Automatic discovery: Sitemap: lines in robots.txt, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt
• ✅ Every format: sitemap indexes (nested), urlsets, gzip, image and news sitemaps, hreflang alternates, RSS and Atom feeds, plain-text URL lists
• ✅ Find new or changed pages: keep only URLs changed on or after a date
• ✅ Filters: include or exclude URLs by regular expression (only /blog/, no /tag/ …)
• ✅ Optional status check: HTTP status, redirect target and result (ok, redirected, broken, blocked) for every URL
• ✅ Fast: 35,000 URLs from 3 sites in 8 seconds in our test, de-duplicated across sitemaps
• ✅ Fair billing: you pay per URL returned; a site without sitemap costs nothing

Typical uses: SEO audits and migrations, feeding URL lists into scrapers or AI pipelines, watching competitors for new product or blog pages, checking that every sitemap URL still works.

Input example

{
    "urls": ["https://www.ikea.com/sitemaps/sitemap.xml", "https://wordpress.org/news/", "https://www.heise.de/rss/heise-atom.xml"],
    "includePattern": "",
    "excludePattern": "/tag/|/author/",
    "lastmodAfter": "2026-10-01",
    "checkStatus": false,
    "maxUrls": 100000
}

maxUrls applies to each website or sitemap in the list.

Output example

Real output from a test run (images and alternates shortened):

{
    "url": "https://www.ikea.com/ee/et/p/yxstaby-vitriinkarp-maengupuldile-valge-70620011/",
    "lastmod": "2026-10-20T10:48:56.790Z",
    "changefreq": "weekly",
    "priority": 1,
    "imageCount": 4,
    "images": ["https://www.ikea.com/ee/et/images/products/yxstaby-vitriinkarp-maengupuldile-valge__1505544_pe1008573_s5.jpg"],
    "alternates": [
        { "hreflang": "en-ee", "url": "https://www.ikea.com/ee/en/p/yxstaby-gaming-controller-display-box-white-70620011/" },
        { "hreflang": "ru-ee", "url": "https://www.ikea.com/ee/ru/p/yxstaby-vitrina-dlya-geympada-belyy-70620011/" }
    ],
    "newsTitle": null,
    "sitemap": "https://www.ikea.com/sitemaps/sitemap.xml",
    "site": "https://www.ikea.com/sitemaps/sitemap.xml"
}

With Check HTTP status on, each item also gets status, category, finalUrl and statusError. For feeds, newsTitle holds the article title and lastmod the publish or update date.

The OUTPUT record lists per input which sitemap files were read, how many URLs they gave and any problems (for example "No sitemap found" or "HTTP 500").

Pricing

A small fixed price per URL returned, plus a price per URL when the status check is on.

Paid Apify plans pay less: 10% off on Starter, 20% on Scale and 30% on Business and higher plans.

FAQ

The site has no sitemap. Then the run ends for that site with a note in OUTPUT and nothing is billed. Use our Broken Link Checker to crawl such sites.

Why are there fewer URLs than in the sitemap? URLs that appear in several sitemaps are returned once, and your filters or maxUrls may drop some. The OUTPUT record shows the details.

Does lastmodAfter keep URLs without a date? No. With a date filter only URLs that state a last-modified date on or after it are kept.

More tools from Tidelane Data

• Broken Link Checker: find 404s, redirects and dead links
• SEO Audit: on-page SEO checks for any list of URLs
• Website to Markdown: clean Markdown of web pages for AI, LLMs and RAG
• Wayback Machine Scraper: snapshots, archived URLs and site history

All our tools: apify.com/tidelane

---

Impressum

Levi Griesinger, Tidelane Data
Besigheimer Straße 18
74369, Löchgau, Deutschland
E-Mail: LeviGriesinger@gmail.com