Sitemap URL Extractor
Pricing
from $0.20 / 1,000 urls
Sitemap URL Extractor
Get every page URL a website publishes in its sitemaps: auto-discovers sitemaps from robots.txt, follows sitemap indexes and gzip, filters by lastmod date and URL pattern, optional HTTP status check. $0.30 per 1,000 URLs.
Pricing
from $0.20 / 1,000 urls
Rating
0.0
(0)
Developer
Aaron Rabinowe
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
4 days ago
Last modified
Categories
Features
Share
Sitemap URL Extractor: every page a website publishes, in seconds
Get the full list of page URLs from any website's sitemaps, for $0.30 per 1,000 URLs. Enter a domain and the Actor finds its sitemaps (via robots.txt and common locations), follows sitemap indexes, unpacks .gz files, and returns every URL with its lastmod, changefreq and priority. Filter by date or URL pattern, and optionally check each URL's live HTTP status to find broken pages.
Sitemaps exist so machines can read them, so this is fast, light and reliable: apify.com's 51,000+ URLs are listed in seconds.
What people use it for
- SEO audits: find every indexable page, spot 404s and redirects (Check HTTP status), and compare against what's ranking.
- Content monitoring: get pages added or updated since a date, such as a competitor's new blog posts or product pages.
- Crawling & AI/RAG pipelines: seed a crawler or an LLM ingestion job with a clean URL list instead of spidering the whole site.
- Migrations & QA: inventory every URL before a redesign, then verify nothing broke.
Input
{"startUrls": ["stripe.com", "https://example.com/sitemap_index.xml"],"lastmodAfter": "2026-09-01","urlIncludes": ["/blog/"],"checkStatus": false,"maxUrls": 10000}
Output
| Field | Example |
|---|---|
url | https://stripe.com/blog/... |
lastmod | 2026-10-01 (as published in the sitemap) |
status | 200 (only with Check HTTP status) |
changefreq, priority | daily, 0.8 |
site, sitemap | the site and the exact sitemap file the URL came from |
The SUMMARY record lists, per site, how many sitemaps and URLs were found, which sites had no sitemap, and how many URLs a date filter skipped because they had no lastmod.
Pricing
$0.30 per 1,000 URLs ($0.0003 per URL) on the free plan, cheaper on larger Apify plans. You pay only for URLs saved; duplicates are free. Set Max URLs or a maximum cost per run and the Actor stops exactly there.
FAQ
A site returned "no sitemap found". It doesn't publish one in robots.txt or the standard locations. If you know its sitemap URL, paste it directly.
Why were URLs skipped with a date filter? Many sitemaps (Stripe's, for example) don't include lastmod. Those URLs can't be date-filtered, so they're skipped and counted in the summary.
Is this allowed? Sitemaps are published specifically for automated tools to read. The Actor only fetches robots.txt and sitemap files (plus a HEAD request per URL if you enable status checks).
More questions
What if a site has no sitemap.xml?
The Actor checks robots.txt for sitemap locations and common sitemap paths, and follows sitemap indexes and gzip files automatically.
Can I check which URLs are broken?
Yes. Turn on the status check to get the HTTP status of every URL, which is useful for SEO audits and migrations.
Can I get only recently updated pages?
Yes. Filter by the lastmod date to monitor new or changed pages.
How do I run it on a schedule?
In Apify Console, open the Actor and click Schedule: hourly, daily or any cron expression. Each run's results land in a new dataset you can export as CSV, Excel or JSON.
Can I call it from my own code or no-code tools?
Yes. Use the Apify API or the official Python and JavaScript clients, or connect it to Make, Zapier, n8n, Google Sheets or a webhook through Apify's built-in integrations.
Is there a free trial?
Apify's free plan includes monthly platform credit, enough to test this Actor on your real data before paying anything.
More tools from the same developer
- Ats Jobs Scraper: every open job from a company's careers page (Workday, Greenhouse, Lever, Ashby and more)
- Tech Job Search: search 4,400+ tech and startup career pages by keyword
- Remote Jobs Search: remote roles straight from company career pages
- Job Salary Data: real salary ranges from live job postings
- Hiring Companies Finder: companies hiring for a role right now (sales hiring signals)
- Website Contact Extractor: emails, phones and social profiles from a list of websites
- Domain Intelligence: WHOIS/RDAP dates, DNS, email provider and tech stack for domains
- Article Extractor: clean Markdown of any web page, ready for LLMs and RAG
- Website Screenshot: full-page and mobile screenshots without cookie banners
- Pdf Text Extractor: text from PDFs at any URL, page by page
- Rss Feed Reader: any RSS or Atom feed as clean JSON
Questions or a feature request? Open an issue on the Actor's Issues tab and I'll usually reply within a day.