Sitemap URL Scraper
Pricing
from $1.19 / 1,000 results
Sitemap URL Scraper
Sitemap Scraper SD - Sitemap Scraper is an SEO crawling tool that extracts every URL from a website's sitemaps (robots.txt, sitemap.xml, indexes, gzip, RSS) with lastmod, changefreq, priority, hreflang and news data.
Pricing
from $1.19 / 1,000 results
Rating
0.0
(0)
Developer
Neuro Scraper
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
5 days ago
Last modified
Categories
Features
Share
Sitemap URL Scraper
The Sitemap URL Scraper collects public Sitemap url data and returns it as a clean dataset you can download as JSON, CSV or Excel, or read through the Apify API. No login, no cookies and no code: give the Sitemap URL Scraper your input, start it and get structured Sitemap data for research, monitoring, analytics or lead generation. You pay only for the results you receive.
⭐ Featured by Neuro Scraper: Facebook Reels Scraper - scrape Facebook Reels from any page, profile or reel link with exact views and plays, every reaction type (Like, Love, Care, Haha, Wow, Sad, Angry), comments, shares, captions, music and video links. No login and no cookies. Try the Facebook Reels Scraper →
Sitemap Scraper — every URL in a website's sitemaps
Sitemap Scraper finds a website's sitemaps — from robots.txt Sitemap: lines, or /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt — follows sitemap indexes to the end, unpacks gzip files and exports every URL with lastmod, changefreq and priority, plus hreflang alternates, image counts and Google News titles and dates. Give it a website or a sitemap URL.
Filter by URL globs or by last-modified date (for incremental runs), and optionally check each URL's HTTP status.
Input
| Field | Type | Default | Meaning |
|---|---|---|---|
startUrls | URL list | crawlee.dev (prefill) | Websites or sitemap / sitemap-index URLs (.xml, .xml.gz, .txt) |
maxItems | integer | 1000 | URLs saved |
maxSitemaps | integer | 200 | Sitemap files read (indexes included) |
includeUrlGlobs / excludeUrlGlobs | glob lists | – | e.g. https://site.com/blog/** |
lastmodAfter | date | – | Only URLs with <lastmod> on/after YYYY-MM-DD |
checkUrlStatus | boolean | false | Request each URL and add statusCode, finalUrl, statusNote |
tryWellKnownPaths | boolean | true | Probe common sitemap paths when robots.txt lists none |
respectRobotsTxt | boolean | true | Skip sitemap files / URL checks robots.txt disallows |
requestDelayMs | integer | 1000 | Pause between requests to one host |
proxyConfiguration | proxy | Apify datacenter | RESIDENTIAL for sites that block datacenter IPs |
Output (one row per URL)
| Field | Meaning |
|---|---|
url | Page URL from the sitemap |
lastmod, changefreq, priority | As published in the sitemap (null when absent) |
imageCount, alternates | Image-sitemap entries; hreflang alternates [{hreflang, href}] |
newsTitle, newsPublishedAt | Google News sitemap data |
sitemapUrl, sitemapType, inputUrl | Which file it came from (urlset, rss, atom, text) and which input |
statusCode, finalUrl, statusNote | Only with checkUrlStatus |
scrapedAt | ISO time |
A per-site summary (sitemaps found and how, files read, URLs saved, problems) is stored in the key-value store record SUMMARY.
Measured (2026-10-03, see TEST_REPORT.md)
- crawlee.dev: 1,000 URLs in 8 s; target.com gzip store index: 2,026 URLs in 10 s; wordpress.org: 30 files (17 indexes) -> 2,576 unique URLs in 36 s.
Limits
- Only URLs the site lists in its sitemaps (python.org publishes none: 0 rows, said plainly).
checkUrlStatusmakes one request per URL at the per-host delay — about 1 URL per second per site.- Sitemaps behind anti-bot pages cannot be read; try a RESIDENTIAL proxy.
Example
{"startUrls": [{"url": "https://www.theguardian.com"}], "lastmodAfter": "2026-10-01", "maxItems": 500}
Why use the Sitemap URL Scraper
The Sitemap URL Scraper is a Sitemap scraper and Sitemap data extractor for anyone who needs public Sitemap url data as clean data. The Sitemap URL Scraper handles web scraping, proxy rotation and retries for you, so you get a ready dataset instead of raw pages. Use the Sitemap URL Scraper for market research, competitor analysis, lead generation, analytics or monitoring, and export the results to JSON, CSV, Excel or Google Sheets.
- No login and no code: the Sitemap URL Scraper runs on Apify with one click or one API call.
- Pay per result: the Sitemap URL Scraper charges only for rows delivered - no monthly Sitemap API fee.
- Complete fields: the Sitemap URL Scraper returns every public field, with a stable schema for automation.
- Integrations: connect the Sitemap URL Scraper to Zapier, Make, n8n, Google Sheets, a webhook or your CRM.
Sitemap URL Scraper use cases
- Market research: the Sitemap URL Scraper collects Sitemap url data at scale, so you can compare trends across weeks and niches.
- Competitor analysis: run the Sitemap URL Scraper on a schedule and export the dataset to Google Sheets or Excel to watch what competitors publish.
- Lead generation and B2B prospecting: the Sitemap URL Scraper gives a sales or agency team a clean list to load into a CRM.
- Content and social media monitoring: the Sitemap URL Scraper keeps a fresh copy of Sitemap data through an Apify schedule and a webhook.
- Analytics and dashboards: send the Sitemap URL Scraper output as JSON through the Apify API into Python, Node.js or a BI tool.
- Academic and data science projects: the Sitemap URL Scraper returns a consistent schema, ready for pandas, R or a notebook.
- Price, product and brand tracking: the Sitemap URL Scraper turns public Sitemap data into a dataset you can diff run by run.
- Automation workflows: connect the Sitemap URL Scraper to Zapier, Make or n8n so every new row starts your next step.
How to run it with the API, Python and Node.js
Start the Sitemap URL Scraper from the Apify Console, or call it from code. The Sitemap URL Scraper returns its dataset as JSON, CSV or Excel. With the Apify API you can run the Sitemap URL Scraper from Python or Node.js, wait for it to finish and read the rows, which makes the Sitemap URL Scraper easy to plug into an automation, a data pipeline or a scheduled export.
- Open the Sitemap URL Scraper on Apify and fill in the input (or send the same input as JSON to the API).
- Start the Sitemap URL Scraper. Follow the run log while the Sitemap URL Scraper collects public Sitemap data.
- Download the dataset as CSV, JSON, Excel or XLSX, or let a webhook send it on.
- Schedule the Sitemap URL Scraper to keep your Sitemap data fresh for monitoring and analytics.
Tips for better results
- Start small: run the Sitemap URL Scraper with a low maximum first, check the fields, then scale up the same input.
- Split big jobs: several smaller Sitemap URL Scraper runs (by keyword, URL list or date) finish faster and are easier to resume.
- Keep a history: name a dataset or use a schedule so each Sitemap URL Scraper run adds to your Sitemap data over time.
- Filter early: use the input filters of the Sitemap URL Scraper so you only pay for the Sitemap url data you really need.
- Automate the hand-off: a webhook or the Zapier and Make integrations can move every Sitemap URL Scraper result straight to Google Sheets, a CRM or a database.
- Watch the run log: the Sitemap URL Scraper writes what it is doing and why an item was skipped, so a surprising result is easy to explain.
Sitemap URL Scraper FAQ
Is it legal to scrape Sitemap?
The Sitemap URL Scraper collects only publicly available data. If results contain personal data, process it under GDPR or the rules that apply to you, and only for a legitimate purpose.
Do I need a Sitemap account or login?
No. The Sitemap URL Scraper works with no login and no cookies. It reads only public data, the same data any visitor can see.
What does this Sitemap scraper do?
The Sitemap URL Scraper collects public Sitemap url data and returns it as a structured dataset. It is web scraping without code: you set the input, start the Sitemap URL Scraper and download JSON, CSV, Excel or XLSX.
Do you offer custom scrapers?
Yes. If you need a field or a filter the Sitemap URL Scraper does not have yet, contact Neuro Scraper through the Apify Store page and we can extend the Sitemap URL Scraper for you.
How much does it cost?
The Sitemap URL Scraper is pay per result: you pay only for the rows you receive, with lower prices on higher Apify plans. You can cap the cost of every run with the maximum cost setting.
Which formats can I download?
The Sitemap URL Scraper dataset downloads as JSON, CSV, Excel (XLSX), XML, RSS or an HTML table, and the Apify API returns the same Sitemap URL Scraper data page by page.
What is the best Sitemap scraper for url data?
The Sitemap URL Scraper is built for exactly this job: public Sitemap url data, complete fields, no login, pay per result and a clean export to JSON, CSV or Excel.
Can I run it for many inputs at once?
Yes. Give the Sitemap URL Scraper a list of inputs in one run; each row says which input it came from, so one Sitemap URL Scraper run can cover a whole list.
How fast is it?
The Sitemap URL Scraper runs requests in parallel and rotates proxy sessions automatically, so a typical run of a few hundred items finishes in minutes.
Does it work with proxies?
Yes. The Sitemap URL Scraper uses Apify Proxy by default and you can pick the proxy group or country in the input.
How is this different from other Sitemap scrapers?
The Sitemap URL Scraper combines complete public fields, no login, pay-per-result pricing and a stable output schema, so the Sitemap URL Scraper fits both quick one-off exports and long-running automation.
Can I export the results to Excel, CSV or Google Sheets?
Yes. Every Sitemap URL Scraper run stores a dataset you can export as CSV, JSON, Excel (XLSX), XML or HTML, or send to Google Sheets with the Apify integration.
What happens if a run is interrupted?
The Sitemap URL Scraper saves its progress, so a resurrected or restarted run continues where it stopped instead of paying for the same rows again.
Why is a field empty in my results?
The Sitemap URL Scraper fills every field Sitemap publishes for that item. When Sitemap does not show a value - for example a figure the owner hid - the field stays empty rather than being guessed.
Can I use it through the API?
Yes. Start the Sitemap URL Scraper and read its dataset through the Apify API from Python, Node.js or any HTTP client, or call it from Zapier, Make and n8n.
Can it replace the official Sitemap API?
For public data, yes. The Sitemap URL Scraper gives you Sitemap data through the Apify API without an official Sitemap API key, quota approval or developer account.
Can I send the data to my CRM?
Yes. Push Sitemap URL Scraper rows to HubSpot, Salesforce, Pipedrive or any CRM through Zapier, Make or a webhook - a common setup for an agency or B2B sales team.
How do I get more results?
Raise the maximum items, add more inputs (keywords, URLs or profiles), or run the Sitemap URL Scraper several times on a schedule. Results are de-duplicated.
Can I schedule it?
Yes. Apify schedules run the Sitemap URL Scraper hourly, daily or weekly, and a webhook can notify your system when new Sitemap data is ready.
More scrapers from Neuro Scraper
| Actor | What it extracts |
|---|---|
| ⭐ Facebook Reels Scraper | Facebook Reels with exact views, plays, every reaction type, comments and shares |
| Snapchat Profile Scraper | Snapchat profile data with all public fields, exported to JSON, CSV or Excel |
| Amazon Scraper | Amazon data with all public fields, exported to JSON, CSV or Excel |
| Instagram Reels Scraper | Instagram reels with all public fields, exported to JSON, CSV or Excel |
| Facebook Video Details Scraper | Facebook video details with all public fields, exported to JSON, CSV or Excel |
| Facebook Reels Scraper Advanced | Facebook reels advanced data with all public fields, exported to JSON, CSV or Excel |
| Facebook Reels Downloader | Facebook reels with all public fields, exported to JSON, CSV or Excel |
| Instagram Search Scraper | Instagram search data with all public fields, exported to JSON, CSV or Excel |
| TikTok Video Details Scraper | TikTok video details with all public fields, exported to JSON, CSV or Excel |
| Instagram Hashtag Scraper | Instagram hashtag data with all public fields, exported to JSON, CSV or Excel |
| TikTok Profile Scraper | TikTok profile data with all public fields, exported to JSON, CSV or Excel |
| Amazon Best Sellers Scraper | Amazon best sellers with all public fields, exported to JSON, CSV or Excel |
| Facebook Photos Scraper | Facebook photos with all public fields, exported to JSON, CSV or Excel |