Sitemap URL Extractor

Sitemap URL Extractor pulls structured data from Robots.txt & Sitemap XML and returns it as a clean dataset you can export as JSON, CSV or Excel. It reads a site's robots.txt and sitemap files, follows nested sitemap indexes, and returns one row per URL with the lastmod, changefreq and priority the sitemap declares.

No API key, no login. Click Start and it works with the prefilled input.

What does Sitemap URL Extractor do?

It queries Robots.txt & Sitemap XML and returns one row per record, with only the fields that matter. Runs on the Apify platform, so you get an API endpoint, scheduling, monitoring and integrations (Zapier, Make, Google Sheets, webhooks) without maintaining any infrastructure.

Why use it?

• Robots.txt & Sitemap XML data without the plumbing: no proxy setup, no pagination code, no scraping maintenance.
• Schedule it to monitor changes daily or weekly and trigger alerts on new records.
• Feed it straight into a spreadsheet, database, or an AI agent via the Apify API or MCP server.
• Runs in seconds and returns a typed dataset that is ready for analysis.

How to use it

1. Click Try for free / Start.
2. Leave the default input to see a sample run, or type your own values (see Input below).
3. Press Start and wait a few seconds.
4. Open the Output tab and download as JSON, CSV, Excel or HTML.

Input

Field	Type	Default	Description
startUrl	string	"https://www.bbc.com"	Site to scan. Accepts a full URL or a bare domain.
maxResults	integer	200	Maximum number of URLs to return.
includeSubSitemaps	boolean	true	When the sitemap is an index, follow the sitemaps it lists (up to 15).

Example input:

{
  "startUrl": "https://www.bbc.com",
  "maxResults": 5
}

Output

One item per record. Sample:

{
  "url": "https://www.bbc.com/news",
  "lastmod": "2026-09-17T22:14:49+00:00",
  "changefreq": "weekly",
  "priority": "0.8",
  "source_sitemap": "https://www.bbc.com/sitemap.xml",
  "sitemap_type": "urlset"
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Data fields

Field	Type	Meaning
url	string	URL
lastmod	string	Last Modified
changefreq	string	Change Frequency
priority	string	Priority
source_sitemap	string	Source Sitemap
sitemap_type	string	Sitemap Type
domain	string	Domain

How much does it cost to run?

This Actor uses pay-per-event pricing: you pay per result item pushed to the dataset, and nothing for an empty run. Each run is a few seconds of compute, so typical costs are fractions of a cent per run, and the Apify free tier covers plenty of testing.

Tips and advanced options

• Raise maxResults for a bigger pull; the Actor pages through the source for you.
• Schedule it (e.g. every morning) and combine with a webhook to get only the new records.
• Add the Actor to an AI agent through the Apify MCP server for natural-language queries.

FAQ, disclaimers and support

• Is this legal? It uses publicly available data from Robots.txt & Sitemap XML. You are responsible for how you use the data and for complying with the source's terms.
• It returned nothing? Enter a full domain such as https://www.bbc.com - most large sites publish a sitemap via robots.txt.
• Something broken? Open an issue on the Actor's Issues tab. Data sources change and get fixed quickly.
• Need a custom variant (extra fields, a different source, a scheduled report)? Get in touch through the Issues tab.