Sitemap URL Extractor: robots.txt + sitemap.xml Crawl

Extract every URL a site exposes via its public sitemap chain. Reads robots.txt for Sitemap: declarations, falls back to /sitemap.xml, recursively descends sitemap-index files, and returns one row per discovered URL with lastmod, changefreq, and priority.

Example output row

{
  "domain": "vercel.com",
  "url": "https://vercel.com/blog/nextjs-14",
  "lastmod": "2024-03-15",
  "changefreq": "weekly",
  "priority": 0.8,
  "source": "https://vercel.com/sitemap-blog.xml"
}

How to use

Input

Field	Type	Default	Description
domains	string[]	["stripe.com","shopify.com","vercel.com"]	Domains to crawl: no scheme, no trailing slash
maxUrlsPerDomain	integer	2000	Hard cap on URLs returned per domain
followSitemapIndex	boolean	true	Recursively follow <sitemapindex> child links (up to depth 5)

Minimal run

{
  "domains": ["example.com"],
  "maxUrlsPerDomain": 500,
  "followSitemapIndex": true
}

Output fields

Field	Type	Notes
domain	string	Input domain
url	string	Discovered URL from <loc>
lastmod	string	ISO date, null if absent
changefreq	string	e.g. weekly, null if absent
priority	float	0.0 to 1.0, null if absent
source	string	Sitemap file the URL was found in

Pricing

Pay per event. Price per 1,000 events:

Event	Base price	Bronze	Silver	Gold
Result	$1.00	$0.90	$0.80	$0.70

Bronze, Silver and Gold are the discounts Apify gives users of its paid plans. The Pricing tab always shows the current prices.

Buyer

• SEO teams auditing crawl coverage: verify every page is in the sitemap.
• Content operations checking lastmod staleness across thousands of URLs.
• Competitive intelligence: map a competitor's full URL structure.
• QA pipelines validating sitemap health after deploys.
• Link-building researchers finding indexable pages at scale.

Run it through the API

Python, with the apify-client package:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("v0iddo/sitemap-url-extractor").call(run_input={"domains": ["stripe.com", "shopify.com", "vercel.com"], "maxUrlsPerDomain": 20})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript, with the apify-client package:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('v0iddo/sitemap-url-extractor').call({"domains": ["stripe.com", "shopify.com", "vercel.com"], "maxUrlsPerDomain": 20});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

One HTTP call that waits for the run and returns the rows (for runs that finish within 5 minutes):

curl -X POST "https://api.apify.com/v2/acts/v0iddo~sitemap-url-extractor/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["stripe.com", "shopify.com", "vercel.com"], "maxUrlsPerDomain": 20}'

Use it from an agent (MCP)

Claude, Cursor, VS Code and other MCP clients can call this actor as a tool through the Apify MCP server. Add it to the client's MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=v0iddo/sitemap-url-extractor",
      "headers": {
        "Authorization": "Bearer <YOUR_APIFY_TOKEN>"
      }
    }
  }
}

FAQ

Do I need an account, cookies or an API key?

No. The actor reads public data without logging in. You only need an Apify account to start it.

How many results can one run return?

Set maxUrlsPerDomain to the number you need. Large jobs can also be split into several runs or scheduled.

How am I charged?

Pay per event, at the prices on the Pricing tab for your Apify plan (the Pricing section of this page lists them). The platform usage of the run is included in those prices, and your spending limit for a run is always respected.

Why did my run return no results?

The status message and the log of the run say why: usually nothing matched the input, or the source did not answer for a moment. Check the input against the example above and run it again.

Can I run it on a schedule?

Yes. Save your input as a task and add a schedule in the Apify Console (daily, hourly or with a cron expression). Each run gets its own dataset, so you keep a history.

Which formats can I export?

JSON, CSV, Excel, XML, HTML table and RSS from the Output tab or the API. Integrations send the rows to Google Sheets, Slack, Make, Zapier, n8n or a webhook.

Is collecting this data allowed?

The actor collects data that the source publishes openly, without a login. Public data can still be personal data under laws such as the GDPR, and sources have their own terms: make sure your use of the data follows the rules that apply to you.

What happens when the source changes?

An automated test runs the actor every day on the example input and checks the rows, not only that the run finished. When a check fails, the actor is repaired and rebuilt; the build history on this page shows each update.

Something is missing or wrong. Where do I report it?

Open an issue on the Issues tab of this actor with the run link. Requests for a new field are welcome too.

Related actors

• SEO Audit Tool (On-Page Checker, Canonical, Alt Text)
• Website Content Crawler: Pages to Clean Text

Source

Crawl order per domain:

1. GET https://{domain}/robots.txt: parse all Sitemap: lines.
2. If none found, fall back to GET https://{domain}/sitemap.xml.
3. For each sitemap URL: fetch + parse XML.
4. If <sitemapindex>, enqueue each <sitemap><loc> (up to depth 5).
5. If <urlset>, emit one row per <url> until maxUrlsPerDomain is reached.

All requests use a polite User-Agent and are paced at 250 to 600 ms between calls. 404 and empty responses are skipped gracefully.