Pricing

$0.20 / 1,000 urls

Try for free

Go to Apify Store

Sitemap URL Extractor - All URLs from Any Website's Sitemaps

Try for free

Get every URL a website publishes in its XML sitemaps, with lastmod, changefreq and priority. Finds sitemaps via robots.txt, follows sitemap indexes and gzipped sitemaps. Pay only per URL returned.

Pricing

$0.20 / 1,000 urls

Rating

0.0

(0)

Developer

Conor Wilson

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

Sitemap URL Extractor

Get every URL a website publishes in its XML sitemaps, with lastmod, changefreq and priority. Give it a homepage and it finds the sitemaps itself through robots.txt, follows sitemap indexes, and reads gzipped and plain-text sitemaps.

$0.20 per 1,000 URLs. Websites with no sitemap are reported, never charged.

Why this one

Automatic discovery. Reads the Sitemap: lines in robots.txt, then falls back to /sitemap.xml and /sitemap_index.xml.
Handles real-world sitemaps. Nested sitemap indexes, .xml.gz files, plain-text sitemaps, namespaces, and de-duplication across files.
Filters that save money. Keep only URLs containing /blog/ or /products/, or only pages modified since a date, and you're charged only for what's kept.
Reliable. Built-in retries with backoff, safety caps on files and URLs, and one clear error row per website if something goes wrong.

Quick start

{
    "startUrls": [{ "url": "https://apify.com" }],
    "urlFilter": ["/blog/"],
    "modifiedSince": "2026-01-01",
    "maxUrls": 50000
}

Use cases

SEO audits and content inventories
Feeding a crawler or URL-to-Markdown pipeline with every page of a site
Monitoring a competitor's newly published pages with a daily schedule and modifiedSince
Migration checks: compare old and new site URL lists

Output

Field	Description
`url`	Page URL listed in the sitemap
`lastmod`, `changefreq`, `priority`	As published by the site
`sitemap`	The sitemap file it came from
`website`	The website you supplied
`ok`, `error`	A per-website error row appears only if nothing could be read (free)

Input options

Option	Default	Notes
`startUrls`	required	Homepages or direct sitemap URLs
`urlFilter`	`[]`	Keep URLs containing any of these strings
`modifiedSince`	-	`YYYY-MM-DD`; URLs without `lastmod` are dropped when set
`maxUrls`	`50000`	Per website
`maxSitemaps`	`500`	Safety cap on sitemap files per website

Good use

Sitemaps are published so machines can read them. The actor still obeys robots.txt for its user agent WilsDataTools and never bypasses access controls.

Pricing

Pay per event: one event per URL saved to the dataset after your filters.

Sitemap Generator - Crawl Website & Create XML Sitemap

scrappy_garden/sitemap-generator

Generate an XML sitemap for any website. Crawls internal pages from start URLs (with depth + page limits), deduplicates URLs, and stores a ready-to-submit sitemap.xml plus a structured dataset and summary for SEO audits.

Bikram Adhikari

Sitemap Generator

datawinder/sitemap-generator

Automatically crawl a website and generate an SEO-ready sitemap in XML, HTML, or TXT format. Supports crawl depth limits, URL include/exclude patterns, and optional merging with an existing sitemap.xml. Ideal for SEO audits, site migrations, and automation.

DatawinderLabs

Sitemap Extractor

smorgi_apps/sitemap-extractor

Smorgi Apps

Sitemap URL Extractor — All Pages of Any Website

tidydata/sitemap-url-extractor

Extract every URL from any website's sitemap: finds sitemaps via robots.txt, follows sitemap indexes and .gz files, returns lastmod, priority and change frequency. $0.30 per 1,000 URLs.

tidydata

Bill 96 French-Parity Website Checker

master_invictus/bill96-french-parity-checker

Check that English pages have working French equivalents (Quebec Bill 96 / Charter of the French Language).

Frank Nagy

Website Traffic Checker - Bulk Domain Traffic Estimate

tenfoldfleet/website-traffic-checker

Check the website traffic of a list of domains: estimated monthly Google search traffic (organic and paid) and ranking keywords, plus optional traffic value and top 10 rankings. 1,000 domains per request, 94 countries. Not total visits. $4 per 1,000 domains + $0.025 per run; no data is free.

Tenfold Fleet

Sitemap URL Extractor

axiomworks/sitemap-url-extractor

Find and read website sitemaps, then return one dataset row per page URL with lastmod, priority, changefreq, image, video, news, and hreflang data. The Actor discovers sitemap locations from robots.txt, common paths, and homepage links, and follows sitemap indexes recursively.

Axiom Works

Robots.txt Tester — Is a URL Allowed to Crawl?

hipersoft/robots-txt-checker

Test whether URLs are allowed or blocked by a site's robots.txt for any crawler user-agent. Returns allowed/blocked, the matched rule, crawl-delay and the site's sitemaps. Great for SEO audits and crawler compliance. One row per URL as JSON, CSV or Excel.

hiper soft

Trivia Question Generator (Open Trivia DB)

smilemask/trivia-question-generator

Pull trivia questions with the correct answer and distractors, filterable by category (general knowledge, science, geography, history, entertainment and more), difficulty and question type. Free Open Trivia DB API, no key, CC BY-SA 4.0 licensed question data.