Sitemap Extractor & Page Counter - URLs, Sections, Lastmod

For SEO consultants, agencies, sales and competitive-intelligence teams who need to know how big a website is and how fresh it is. Give it a list of domains and get one row per domain: the page count from the site's sitemaps, the section split (the 10 biggest first path segments such as /blog or /products), the newest and oldest lastmod, how many sitemap files and index levels it took, and where the sitemaps were found. Switch on includeUrls and you also get one row per page URL with lastmod, changefreq and priority, in the same dataset.

$0.004 per domain with at least one page URL found. 1,000 domains = $4. URL rows are free for the first 100 per domain, then $0.0002 each. Domains with no sitemap, blocked by robots.txt, unreachable or invalid are delivered with a clear status and are never charged.

Reliability is the product. Sitemap extraction fails on real sites in predictable ways: a sitemap index that points to more indexes, .xml.gz files, a sitemap that is really a text list or an RSS feed, a robots.txt that lists the sitemap under another host, files that are hundreds of megabytes, redirect loops and the same sitemap served under five URLs. This Actor reads robots.txt Sitemap: lines first, falls back to /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml, follows indexes up to 3 levels, inflates gzip, accepts text and RSS/Atom sitemaps, de-duplicates sitemaps and URLs (a repeated sitemap is delivered and charged once) and puts a hard cap on every dimension (URLs, files, bytes, redirects, time). A broken or truncated file produces a status row and a warning, never a crashed run.

Public files only: no proxy, no login, no CAPTCHA solving, no anti-bot evasion. It identifies itself with an honest User-Agent, honours robots.txt (including Disallow rules for the sitemap URLs themselves and Crawl-delay) and waits at least a second between requests to the same site.

What you get

Field	What it tells you
status	ok, partial (some files failed or hit a cap), no_sitemap, blocked, unreachable, error or invalid_input. Only ok and partial with URLs are charged.
pageCount, urlCapReached	Distinct page URLs across all the site's sitemaps, and whether counting stopped at maxUrlsPerDomain.
sections, sectionCount	The 10 largest sections as {section, count} and the total number of sections. A section is the first path segment; root-level pages (/, /about) count under /.
newestLastmod, oldestLastmod, urlsWithLastmod	The freshest and the stalest lastmod published, and how many pages carry one. A site whose newest lastmod is two years old has stopped publishing.
sitemapCount, sitemapIndexDepth	Sitemap files read and levels of index files (0 = a plain sitemap, 1 = one index, 2 = an index of indexes).
sitemapSource, sitemapUrls, robotsTxt	Where the sitemaps were found (robots.txt, a fallback path, or input when you passed a sitemap URL), the top-level sitemap URLs, and the state of robots.txt.
warnings	Anything that was skipped or cut short, in plain words.

With includeUrls on, URL rows carry type: "url", domain, url, lastmod, changefreq, priority and sitemap (the file the URL was first found in). Filter on type to split the two kinds of row.

Quick start

1. Open the Input tab and paste domains or URLs, one per line.
2. Click Start, then export the dataset to CSV, Excel or JSON.

{
  "domains": ["apify.com", "shopify.com", "stripe.com"],
  "maxItems": 3
}

Input is forgiving: https://www.Acme.co.uk/blog/post?x=1 and acme.co.uk both become acme.co.uk, duplicates and the same site with and without www are merged, and a direct sitemap URL such as https://acme.co.uk/sitemap_index.xml is read first; if it yields no URLs, the site's robots.txt Sitemap: lines and the fallback paths are tried as usual (a sitemap URL for the same site given next to its plain domain is merged into one entry).

Recipes

Size up competitors. Paste ten competitor domains and sort by pageCount. The sections column shows where their content is: a site with 40,000 pages under /products and 300 under /blog is a catalogue, not a publisher.

Check who is still publishing. Sort by newestLastmod. Prospects whose newest lastmod is more than a year old are good targets for content and SEO services; a recent newestLastmod means an active team.

Extract every URL. Turn includeUrls on to get the URL list for a content audit, a migration or a crawl list. The first 100 URL rows of each domain cost nothing.

{
  "domains": ["apify.com", "https://www.example-store.com/sitemap_index.xml"],
  "includeUrls": true,
  "maxUrlsPerDomain": 5000
}

Qualify leads. Pair with Website Contact Scraper to get emails and phones for the domains with the biggest sites, and with Website Tech Stack Detector to see which CMS or ecommerce platform those sites run.

Input

Field	Default	What it does
domains	required	Domains, URLs or direct sitemap URLs.
includeUrls	false	Also output one row per page URL.
maxUrlsPerDomain	5000	Hard cap on distinct URLs per domain, 1 to 500,000. The default keeps a first run quick; raise it for big sites.
maxSitemapsPerDomain	20	Hard cap on sitemap files requested per domain, 1 to 1,000. Big sites split their sitemap into hundreds of files: raise it.
maxItems	100	Hard cap on domains read, 1 to 5,000.

Example output

A domain row:

{
  "type": "domain",
  "domain": "acme-example.com",
  "input": "https://www.acme-example.com/pricing",
  "status": "ok",
  "error": null,
  "pageCount": 12840,
  "urlCapReached": false,
  "sections": [
    { "section": "/products", "count": 9120 },
    { "section": "/blog", "count": 2410 },
    { "section": "/docs", "count": 880 },
    { "section": "/", "count": 430 }
  ],
  "sectionCount": 11,
  "urlsWithLastmod": 12790,
  "newestLastmod": "2026-10-11T09:14:00+00:00",
  "oldestLastmod": "2019-03-02",
  "sitemapCount": 14,
  "sitemapIndexDepth": 1,
  "sitemapSource": "robots.txt",
  "sitemapUrls": ["https://www.acme-example.com/sitemap_index.xml"],
  "robotsTxt": "found",
  "warnings": [],
  "checkedAt": "2026-10-13T08:30:00.000Z"
}

A URL row (only with includeUrls):

{
  "type": "url",
  "domain": "acme-example.com",
  "url": "https://www.acme-example.com/blog/how-to-pick-a-widget",
  "lastmod": "2026-09-30",
  "changefreq": "monthly",
  "priority": 0.6,
  "sitemap": "https://www.acme-example.com/sitemap-posts-3.xml.gz"
}

Pricing

Pay-per-event, charged only for delivered value:

• domain: $0.004 per domain whose sitemaps contained at least one page URL (status ok or partial).
• url: $0.0002 per URL row beyond the first 100 URL rows of a domain, and only with includeUrls on. The first 100 per domain are free.
• No sitemap, blocked by robots.txt, unreachable, unreadable sitemap and invalid input are free.

Run	Cost
100 domains, summary rows only	$0.40
1,000 domains, summary rows only	$4.00
1 domain with 50,000 URLs (maxUrlsPerDomain raised), URL rows on	$0.004 + 49,900 x $0.0002 = $9.98
1 domain with 100 URLs or fewer, URL rows on	$0.004

maxItems, maxUrlsPerDomain and your Apify per-run spending limit all stop delivery before they are passed. When the spending limit runs out the run still finishes normally and the status message says what was left out. Price changes follow Apify's notice rules; the Pricing tab always shows the rate you are charged.

Limitations

• Only what the site publishes. The page count is the number of distinct URLs in the sitemaps, not a crawl. Sites without a sitemap are reported as no_sitemap; pages missing from a sitemap are not counted, and some sitemaps list pages that no longer exist.
• Same-site URLs only. URLs and child sitemaps on other sites are ignored (a sitemap may only list its own site's pages), unless robots.txt itself lists a sitemap on another host. Hosts that are IP addresses, localhost, internal names or that resolve to a private, loopback, link-local or metadata address are never requested.
• robots.txt is respected absolutely. A robots.txt answering 401, 403, 429 or a persistent 5xx marks the domain blocked and nothing else is requested; any other 4xx (404, 410 and so on, RFC 9309) means no restrictions. Crawl-delay is honoured up to 10 seconds between requests; a longer delay is capped at 10 s and the row gets a warning saying so. Sitemap URLs disallowed for User-agent: * are not fetched.
• Caps, not surprises. Index depth 3, 50 MB uncompressed per file (streamed; anything beyond is cut and flagged), maxSitemapsPerDomain, maxUrlsPerDomain, 5 redirects per request, at most 10 minutes per domain (retries and waits stop there too), entries over 64 KB skipped, and a stop after 25 sitemaps in a row that add no new URL. A capped or cut domain is marked partial or urlCapReached with a warning.
• Dates are as published. lastmod is shown as the site wrote it; invalid dates are ignored. RSS and Atom items use their publication or update date.
• Memory. URL lists are held in memory per domain while they are delivered. With includeUrls on, give the run at least 1 GB (the Actor limits itself to about 900 MB of heap); above 100,000 URLs per domain, domains are read one at a time. Very large sites are best split by passing their section sitemaps as input.
• Restarts. If the platform restarts a run, rows already delivered and paid for are not charged twice, but free rows of the interrupted domain may appear twice.
• Not affiliated with any site named here. Sitemaps are public; use the data in line with the site's terms and applicable law.

Use it from your code, Make, Zapier or n8n

Built for automated pipelines: small runs, stable field names, one row per item, and you pay only for rows actually delivered.

One API call that runs the Actor and returns the data (replace $APIFY_TOKEN with your token from Apify Console → Settings → Integrations):

curl -s -X POST "https://api.apify.com/v2/acts/lukehunter~sitemap-page-counter/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"domains": ["apify.com", "shopify.com", "stripe.com"], "maxUrlsPerDomain": 5000, "maxSitemapsPerDomain": 20, "maxItems": 3}'

JavaScript (npm i apify-client):

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('lukehunter~sitemap-page-counter').call({
  "domains": [
    "apify.com",
    "shopify.com",
    "stripe.com"
  ],
  "maxUrlsPerDomain": 5000,
  "maxSitemapsPerDomain": 20,
  "maxItems": 3
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

Python (pip install apify-client):

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('lukehunter~sitemap-page-counter').call(run_input={"domains": ["apify.com", "shopify.com", "stripe.com"], "maxUrlsPerDomain": 5000, "maxSitemapsPerDomain": 20, "maxItems": 3})
items = client.dataset(run['defaultDatasetId']).list_items().items

Schedules, webhooks and no-code tools: save your input as a Task, add a Schedule (e.g. hourly or daily), and attach a webhook or one of the Integrations tab's connectors (Make, Zapier, n8n, Google Sheets, Slack) to push each run's results into your own system.

Feedback: an honest rating on the Store page, good or bad, helps others decide if this Actor fits their needs. Missing a field or found a problem? Open an issue on the Actor page.

FAQ

Why is my domain no_sitemap? The site has no robots.txt Sitemap: line and none of /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml exists. If the sitemap lives elsewhere, paste its URL as the input.

Why blocked? The site's robots.txt refuses automated access (401, 403, 429 or server errors), or disallows its own sitemap URLs for generic crawlers. The Actor does not try to get around that, and the row is free.

What does partial mean? URLs were found, but at least one sitemap file failed, was cut at a cap or was malformed. The warnings field says which. A partial row is charged like ok, because it delivers URLs.

Why is the count lower than a crawl? The count is what the site declares. Many sites leave out parameter pages, old pages or entire sections.

Is a duplicated sitemap charged twice? No. Sitemap files and page URLs are de-duplicated per domain, so a sitemap listed twice, served under two URLs or redirecting to another listed sitemap is read, delivered and charged once.

Can I get more than 500,000 URLs per domain? No, that is the hard maximum. Use a section-specific sitemap URL as input to split a very large site.

Related Actors

Other data tools from the same developer, built to the same standard: official or public sources, hard cost caps, and honest documentation of limits.

• Domain Email Provider Lookup: the email provider (Google Workspace, Microsoft 365, Zoho and more), MX, SPF and DMARC setup of any domain list, with provider-change alerts.
• Bulk WHOIS & Domain Age Lookup (RDAP): domain age, expiry date, registrar, nameservers and availability for any domain list over the official RDAP protocol, without personal contact data.
• French Company Leads: French companies (entreprises) from the official Sirene-based API, filtered by NAF code, department, headcount and creation date, with SIREN/SIRET, officers and new-company alerts.
• Poland KRS Company Lookup: Polish companies from the official KRS API by KRS number: NIP, REGON, address, share capital, PKD and the board as published, with optional VAT status.
• EU VAT Number Validator: bulk EU VAT number checks against the official VIES service: valid or invalid, registered name and address, with optional consultation numbers.
• Phone Number Validator: bulk phone number validation and E.164 formatting, offline: valid or invalid, country, and mobile or landline type for every number in a list or scraped contact dataset.
• Website Contact Scraper: emails, phone numbers and social profile links from any list of company domains, with the page each came from and a changed-contacts-only mode for scheduled runs.
• Impressum Scraper: legal entity data from the imprint of any company website in Germany, Austria and Switzerland: legal name and form, managing directors, HRB/HRA or FN register, VAT id and address, with a changed-data-only mode for scheduled runs.