Sitemap URL Discovery

Discover every URL a website's own sitemap declares, starting from the Sitemap: directive in robots.txt (falling back to the conventional /sitemap.xml path if none is declared). No API key, no crawling the site page-by-page — this reads the site's own XML sitemap, the same file search engines use to index it.

Features

• Robots.txt-first discovery. Reads Sitemap: lines from robots.txt rather than guessing at a URL, so it follows whatever sitemap the site actually declares.
• Sitemap index support. If the declared sitemap is an index (<sitemapindex>), it automatically follows the first child sitemap one level deep to reach real page URLs.
• Fallback path. Sites with no Sitemap: line in robots.txt are still checked at the conventional /sitemap.xml location.
• Bulk URL list. Returns up to 500 discovered URLs per site in one row, with a truncated flag when the real sitemap has more.
• Pay only for real discoveries. Domains with no usable sitemap cost nothing — see Pricing.

How to use Sitemap URL Discovery — XML Sitemap & robots.txt API

1. In the Apify Console. Open the actor page and click Start — the items field is already pre-filled with a working example. Results land in the run's dataset as soon as each item is found.
2. Via the API. Call it directly with a POST request — no Console needed once you have an API token:

   curl "https://api.apify.com/v2/acts/accountable_eel~sitemap-url-discovery/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
     -X POST \
     -H "Content-Type: application/json" \
     -d '{"items":["bbc.co.uk"]}'
   

3. On a schedule. Save this actor as an Apify Task with the input you want, then add a Schedule (hourly, daily, weekly) so it runs on its own — no server of your own required.

Input

{
  "items": ["nytimes.com", "bbc.co.uk"],
  "maxConcurrency": 5,
  "proxyConfiguration": { "useApifyProxy": true }
}

items is a list of domains or URLs (a bare domain is normalized to https:// and reduced to its root). One dataset row is returned per item; rows with "found": false are never charged. maxConcurrency (default 5) caps parallel requests — kept conservative since this target has no browser fallback. proxyConfiguration routes requests through Apify Proxy.

Output

{
  "query": "bbc.co.uk",
  "found": true,
  "data": {
    "domain": "bbc.co.uk",
    "sitemapIndexUrls": ["https://www.bbc.co.uk/sitemap.xml"],
    "sampledSitemap": "https://www.bbc.co.uk/sitemaps/https-index-uk-archive.xml",
    "urlCount": 500,
    "urls": [
      "https://www.bbc.co.uk/news/uk-12345678",
      "https://www.bbc.co.uk/sport/football/98765432"
    ],
    "truncated": true
  },
  "scrapedAt": "2026-08-20T14:03:11.000Z"
}

A row is only marked found: true — and only then billed — once a sitemap resolves and yields at least one <loc> URL. A domain with no Sitemap: line in robots.txt and no sitemap at the conventional fallback path (or a sitemap index whose first child sitemap is unreachable) comes back as found: false and is never charged. urls is capped at 500 entries per row; truncated: true means the actual sitemap had more.

Use cases

• Seed a broader crawl or scraping job with a real, authoritative list of a site's known URLs instead of guessing at site structure.
• Audit whether a site publishes a sitemap at all, and where it points, as part of a technical SEO review.
• Compare urlCount across a portfolio of sites to spot ones with unusually thin or missing sitemap coverage.
• Pull a sample of a competitor's published content/product URLs for content-gap analysis.
• Verify that a sitemap index correctly points to a working child sitemap after a site migration or CMS change.

Pricing

$5 per 1,000 results, plus a $0.005 start fee. Misses (found:false) are never charged.

Use it from Clay, n8n, Make, or an AI agent

This actor runs synchronously over plain HTTP — call it directly from a script, a workflow tool, or an AI agent, no Apify Console needed once you have an API token.

curl "https://api.apify.com/v2/acts/accountable_eel~sitemap-url-discovery/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"items":["bbc.co.uk"]}'

n8n. Add an HTTP Request node: Method POST, URL https://api.apify.com/v2/acts/accountable_eel~sitemap-url-discovery/run-sync-get-dataset-items?token=<YOUR_TOKEN>, Body Content Type JSON, JSON Body {"items":["bbc.co.uk"]} (swap in an expression from an earlier node for a real value).

Clay. Add an "HTTP API" column: Method POST, URL https://api.apify.com/v2/acts/accountable_eel~sitemap-url-discovery/run-sync-get-dataset-items?token=<YOUR_TOKEN>, Body {"items":["{{value}}"]}, mapping the row's value into the items array.

MCP. In Claude, Cursor, or any MCP client with the Apify MCP server, ask for "Sitemap URL Discovery | Apify" — the agent will find and run this actor.

FAQ

Why did a well-known site come back found: false? Not every site declares a Sitemap: line in robots.txt or serves one at /sitemap.xml — that's a real, correct outcome for sites without a public sitemap, not a failure of the actor.

Does it crawl the whole site to find URLs? No — it only reads robots.txt and follows the sitemap URL(s) declared there (or the conventional fallback path). It never crawls arbitrary site pages.

What happens with a sitemap index (a sitemap of sitemaps)? The actor follows the first child sitemap listed in the index one level deep and returns URLs from that child. It does not aggregate every child sitemap in a large index.

Is there a limit on how many URLs come back? Yes — up to 500 per row. urlCount reports the number found in the sampled sitemap, and truncated: true indicates more exist than were returned.

Does it respect robots.txt disallow rules? No — this actor only reads the Sitemap: directive from robots.txt, it does not evaluate Disallow/Allow crawl rules, since it isn't crawling site pages.

Can I pass a URL with a path instead of a bare domain? Yes — any path is stripped down to the domain's root before checking robots.txt, since sitemaps are declared per-domain, not per-page.