# Sitemap Extractor & Page Counter - URLs, Sections, Lastmod (`lukehunter/sitemap-page-counter`) Actor

Reliable sitemap extractor and page counter. For a list of domains it reads robots.txt and every sitemap (index, .xml.gz, text, RSS) and returns one row per site: page count, sections, newest lastmod. $0.004/domain, URL rows optional.

- **URL**: https://apify.com/lukehunter/sitemap-page-counter.md
- **Developed by:** [Luke Hunter](https://apify.com/lukehunter) (community)
- **Categories:** SEO tools, Lead generation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 domain with urls founds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor & Page Counter - URLs, Sections, Lastmod

**For SEO consultants, agencies, sales and competitive-intelligence teams who need to know how big a website is and how fresh it is.** Give it a list of domains and get one row per domain: the **page count** from the site's sitemaps, the **section split** (the 10 biggest first path segments such as `/blog` or `/products`), the **newest and oldest lastmod**, how many sitemap files and index levels it took, and where the sitemaps were found. Switch on `includeUrls` and you also get one row per page URL with `lastmod`, `changefreq` and `priority`, in the same dataset.

**$0.004 per domain with at least one page URL found. 1,000 domains = $4. URL rows are free for the first 100 per domain, then $0.0002 each.** Domains with no sitemap, blocked by `robots.txt`, unreachable or invalid are delivered with a clear status and are never charged.

Reliability is the product. Sitemap extraction fails on real sites in predictable ways: a sitemap index that points to more indexes, `.xml.gz` files, a sitemap that is really a text list or an RSS feed, a `robots.txt` that lists the sitemap under another host, files that are hundreds of megabytes, redirect loops and the same sitemap served under five URLs. This Actor reads robots.txt `Sitemap:` lines first, falls back to `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml`, follows indexes up to 3 levels, inflates gzip, accepts text and RSS/Atom sitemaps, de-duplicates sitemaps and URLs (a repeated sitemap is delivered and charged once) and puts a hard cap on every dimension (URLs, files, bytes, redirects, time). A broken or truncated file produces a status row and a warning, never a crashed run.

Public files only: no proxy, no login, no CAPTCHA solving, no anti-bot evasion. It identifies itself with an honest User-Agent, honours `robots.txt` (including `Disallow` rules for the sitemap URLs themselves and `Crawl-delay`) and waits at least a second between requests to the same site.

### What you get

| Field | What it tells you |
|---|---|
| `status` | `ok`, `partial` (some files failed or hit a cap), `no_sitemap`, `blocked`, `unreachable`, `error` or `invalid_input`. Only `ok` and `partial` with URLs are charged. |
| `pageCount`, `urlCapReached` | Distinct page URLs across all the site's sitemaps, and whether counting stopped at `maxUrlsPerDomain`. |
| `sections`, `sectionCount` | The 10 largest sections as `{section, count}` and the total number of sections. A section is the first path segment; root-level pages (`/`, `/about`) count under `/`. |
| `newestLastmod`, `oldestLastmod`, `urlsWithLastmod` | The freshest and the stalest lastmod published, and how many pages carry one. A site whose newest lastmod is two years old has stopped publishing. |
| `sitemapCount`, `sitemapIndexDepth` | Sitemap files read and levels of index files (0 = a plain sitemap, 1 = one index, 2 = an index of indexes). |
| `sitemapSource`, `sitemapUrls`, `robotsTxt` | Where the sitemaps were found (`robots.txt`, a fallback path, or `input` when you passed a sitemap URL), the top-level sitemap URLs, and the state of `robots.txt`. |
| `warnings` | Anything that was skipped or cut short, in plain words. |

With `includeUrls` on, URL rows carry `type: "url"`, `domain`, `url`, `lastmod`, `changefreq`, `priority` and `sitemap` (the file the URL was first found in). Filter on `type` to split the two kinds of row.

### Quick start

1. Open the **Input** tab and paste domains or URLs, one per line.
2. Click **Start**, then export the dataset to CSV, Excel or JSON.

```json
{
  "domains": ["apify.com", "shopify.com", "stripe.com"],
  "maxItems": 3
}
```

Input is forgiving: `https://www.Acme.co.uk/blog/post?x=1` and `acme.co.uk` both become `acme.co.uk`, duplicates and the same site with and without `www` are merged, and a direct sitemap URL such as `https://acme.co.uk/sitemap_index.xml` is read first; if it yields no URLs, the site's robots.txt `Sitemap:` lines and the fallback paths are tried as usual (a sitemap URL for the same site given next to its plain domain is merged into one entry).

### Recipes

**Size up competitors.** Paste ten competitor domains and sort by `pageCount`. The `sections` column shows where their content is: a site with 40,000 pages under `/products` and 300 under `/blog` is a catalogue, not a publisher.

**Check who is still publishing.** Sort by `newestLastmod`. Prospects whose newest lastmod is more than a year old are good targets for content and SEO services; a recent `newestLastmod` means an active team.

**Extract every URL.** Turn `includeUrls` on to get the URL list for a content audit, a migration or a crawl list. The first 100 URL rows of each domain cost nothing.

```json
{
  "domains": ["apify.com", "https://www.example-store.com/sitemap_index.xml"],
  "includeUrls": true,
  "maxUrlsPerDomain": 5000
}
```

**Qualify leads.** Pair with [Website Contact Scraper](https://apify.com/lukehunter/company-contact-scraper) to get emails and phones for the domains with the biggest sites, and with [Website Tech Stack Detector](https://apify.com/lukehunter/website-tech-stack-detector) to see which CMS or ecommerce platform those sites run.

### Input

| Field | Default | What it does |
|---|---:|---|
| `domains` | required | Domains, URLs or direct sitemap URLs. |
| `includeUrls` | `false` | Also output one row per page URL. |
| `maxUrlsPerDomain` | `5000` | Hard cap on distinct URLs per domain, 1 to 500,000. The default keeps a first run quick; raise it for big sites. |
| `maxSitemapsPerDomain` | `20` | Hard cap on sitemap files requested per domain, 1 to 1,000. Big sites split their sitemap into hundreds of files: raise it. |
| `maxItems` | `100` | Hard cap on domains read, 1 to 5,000. |

### Example output

A domain row:

```json
{
  "type": "domain",
  "domain": "acme-example.com",
  "input": "https://www.acme-example.com/pricing",
  "status": "ok",
  "error": null,
  "pageCount": 12840,
  "urlCapReached": false,
  "sections": [
    { "section": "/products", "count": 9120 },
    { "section": "/blog", "count": 2410 },
    { "section": "/docs", "count": 880 },
    { "section": "/", "count": 430 }
  ],
  "sectionCount": 11,
  "urlsWithLastmod": 12790,
  "newestLastmod": "2026-10-11T09:14:00+00:00",
  "oldestLastmod": "2019-03-02",
  "sitemapCount": 14,
  "sitemapIndexDepth": 1,
  "sitemapSource": "robots.txt",
  "sitemapUrls": ["https://www.acme-example.com/sitemap_index.xml"],
  "robotsTxt": "found",
  "warnings": [],
  "checkedAt": "2026-10-13T08:30:00.000Z"
}
```

A URL row (only with `includeUrls`):

```json
{
  "type": "url",
  "domain": "acme-example.com",
  "url": "https://www.acme-example.com/blog/how-to-pick-a-widget",
  "lastmod": "2026-09-30",
  "changefreq": "monthly",
  "priority": 0.6,
  "sitemap": "https://www.acme-example.com/sitemap-posts-3.xml.gz"
}
```

### Pricing

Pay-per-event, charged only for delivered value:

- **`domain`: $0.004** per domain whose sitemaps contained at least one page URL (status `ok` or `partial`).
- **`url`: $0.0002** per URL row beyond the first 100 URL rows of a domain, and only with `includeUrls` on. The first 100 per domain are free.
- No sitemap, blocked by `robots.txt`, unreachable, unreadable sitemap and invalid input are **free**.

| Run | Cost |
|---|---:|
| 100 domains, summary rows only | $0.40 |
| 1,000 domains, summary rows only | $4.00 |
| 1 domain with 50,000 URLs (`maxUrlsPerDomain` raised), URL rows on | $0.004 + 49,900 x $0.0002 = $9.98 |
| 1 domain with 100 URLs or fewer, URL rows on | $0.004 |

`maxItems`, `maxUrlsPerDomain` and your Apify per-run spending limit all stop delivery before they are passed. When the spending limit runs out the run still finishes normally and the status message says what was left out. Price changes follow Apify's notice rules; the Pricing tab always shows the rate you are charged.

### Limitations

- **Only what the site publishes.** The page count is the number of distinct URLs in the sitemaps, not a crawl. Sites without a sitemap are reported as `no_sitemap`; pages missing from a sitemap are not counted, and some sitemaps list pages that no longer exist.
- **Same-site URLs only.** URLs and child sitemaps on other sites are ignored (a sitemap may only list its own site's pages), unless `robots.txt` itself lists a sitemap on another host. Hosts that are IP addresses, `localhost`, internal names or that resolve to a private, loopback, link-local or metadata address are never requested.
- **`robots.txt` is respected absolutely.** A `robots.txt` answering 401, 403, 429 or a persistent 5xx marks the domain `blocked` and nothing else is requested; any other 4xx (404, 410 and so on, RFC 9309) means no restrictions. `Crawl-delay` is honoured up to 10 seconds between requests; a longer delay is capped at 10 s and the row gets a warning saying so. Sitemap URLs disallowed for `User-agent: *` are not fetched.
- **Caps, not surprises.** Index depth 3, 50 MB uncompressed per file (streamed; anything beyond is cut and flagged), `maxSitemapsPerDomain`, `maxUrlsPerDomain`, 5 redirects per request, at most 10 minutes per domain (retries and waits stop there too), entries over 64 KB skipped, and a stop after 25 sitemaps in a row that add no new URL. A capped or cut domain is marked `partial` or `urlCapReached` with a warning.
- **Dates are as published.** `lastmod` is shown as the site wrote it; invalid dates are ignored. RSS and Atom items use their publication or update date.
- **Memory.** URL lists are held in memory per domain while they are delivered. With `includeUrls` on, give the run at least 1 GB (the Actor limits itself to about 900 MB of heap); above 100,000 URLs per domain, domains are read one at a time. Very large sites are best split by passing their section sitemaps as input.
- **Restarts.** If the platform restarts a run, rows already delivered and paid for are not charged twice, but free rows of the interrupted domain may appear twice.
- **Not affiliated** with any site named here. Sitemaps are public; use the data in line with the site's terms and applicable law.

### Use it from your code, Make, Zapier or n8n

Built for automated pipelines: small runs, stable field names, one row per item, and you pay only for rows actually delivered.

**One API call that runs the Actor and returns the data** (replace `$APIFY_TOKEN` with your token from Apify Console → Settings → Integrations):

```bash
curl -s -X POST "https://api.apify.com/v2/acts/lukehunter~sitemap-page-counter/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"domains": ["apify.com", "shopify.com", "stripe.com"], "maxUrlsPerDomain": 5000, "maxSitemapsPerDomain": 20, "maxItems": 3}'
```

**JavaScript** (`npm i apify-client`):

```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('lukehunter~sitemap-page-counter').call({
  "domains": [
    "apify.com",
    "shopify.com",
    "stripe.com"
  ],
  "maxUrlsPerDomain": 5000,
  "maxSitemapsPerDomain": 20,
  "maxItems": 3
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

**Python** (`pip install apify-client`):

```python
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('lukehunter~sitemap-page-counter').call(run_input={"domains": ["apify.com", "shopify.com", "stripe.com"], "maxUrlsPerDomain": 5000, "maxSitemapsPerDomain": 20, "maxItems": 3})
items = client.dataset(run['defaultDatasetId']).list_items().items
```

**Schedules, webhooks and no-code tools:** save your input as a Task, add a Schedule (e.g. hourly or daily), and attach a webhook or one of the Integrations tab's connectors (Make, Zapier, n8n, Google Sheets, Slack) to push each run's results into your own system.

**Feedback:** an honest rating on the Store page, good or bad, helps others decide if this Actor fits their needs. Missing a field or found a problem? Open an issue on the Actor page.

### FAQ

**Why is my domain `no_sitemap`?** The site has no robots.txt `Sitemap:` line and none of `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` exists. If the sitemap lives elsewhere, paste its URL as the input.

**Why `blocked`?** The site's `robots.txt` refuses automated access (401, 403, 429 or server errors), or disallows its own sitemap URLs for generic crawlers. The Actor does not try to get around that, and the row is free.

**What does `partial` mean?** URLs were found, but at least one sitemap file failed, was cut at a cap or was malformed. The `warnings` field says which. A `partial` row is charged like `ok`, because it delivers URLs.

**Why is the count lower than a crawl?** The count is what the site declares. Many sites leave out parameter pages, old pages or entire sections.

**Is a duplicated sitemap charged twice?** No. Sitemap files and page URLs are de-duplicated per domain, so a sitemap listed twice, served under two URLs or redirecting to another listed sitemap is read, delivered and charged once.

**Can I get more than 500,000 URLs per domain?** No, that is the hard maximum. Use a section-specific sitemap URL as input to split a very large site.

### Related Actors

Other data tools from the same developer, built to the same standard: official or public sources, hard cost caps, and honest documentation of limits.

- **[Domain Email Provider Lookup](https://apify.com/lukehunter/email-provider-mx-lookup)**: the email provider (Google Workspace, Microsoft 365, Zoho and more), MX, SPF and DMARC setup of any domain list, with provider-change alerts.
- **[Bulk WHOIS & Domain Age Lookup (RDAP)](https://apify.com/lukehunter/domain-whois-rdap-lookup)**: domain age, expiry date, registrar, nameservers and availability for any domain list over the official RDAP protocol, without personal contact data.
- **[French Company Leads](https://apify.com/lukehunter/french-companies-sirene)**: French companies (entreprises) from the official Sirene-based API, filtered by NAF code, department, headcount and creation date, with SIREN/SIRET, officers and new-company alerts.
- **[Poland KRS Company Lookup](https://apify.com/lukehunter/poland-krs-company-lookup)**: Polish companies from the official KRS API by KRS number: NIP, REGON, address, share capital, PKD and the board as published, with optional VAT status.
- **[EU VAT Number Validator](https://apify.com/lukehunter/eu-vat-number-validator)**: bulk EU VAT number checks against the official VIES service: valid or invalid, registered name and address, with optional consultation numbers.
- **[Phone Number Validator](https://apify.com/lukehunter/phone-number-validator)**: bulk phone number validation and E.164 formatting, offline: valid or invalid, country, and mobile or landline type for every number in a list or scraped contact dataset.
- **[Website Contact Scraper](https://apify.com/lukehunter/company-contact-scraper)**: emails, phone numbers and social profile links from any list of company domains, with the page each came from and a changed-contacts-only mode for scheduled runs.
- **[Impressum Scraper](https://apify.com/lukehunter/impressum-scraper)**: legal entity data from the imprint of any company website in Germany, Austria and Switzerland: legal name and form, managing directors, HRB/HRA or FN register, VAT id and address, with a changed-data-only mode for scheduled runs.

# Actor input Schema

## `domains` (type: `array`):

Websites to read the sitemaps of. Accepts domains, page URLs and direct sitemap URLs; scheme, www and paths are dropped, except a sitemap URL, which is read first. Example: \["apify.com", "https://www.shopify.com/blog", "https://example.com/sitemap_index.xml"]. Duplicates are merged.

## `includeUrls` (type: `boolean`):

Off by default: one summary row per domain. On, e.g. true: also one row per page URL (url, lastmod, changefreq, priority, sitemap) in the same dataset, marked type "url". The first 100 URL rows per domain are free; the rest cost $0.0002 each.

## `maxUrlsPerDomain` (type: `integer`):

Hard cap on distinct page URLs read per domain, 1 to 500000, e.g. 5000, up to 500000 for big sites. The count stops there and urlCapReached is set. With URL rows on, this also caps the URL rows you can be charged for per domain.

## `maxSitemapsPerDomain` (type: `integer`):

Hard cap on sitemap files requested per domain, 1 to 1000, e.g. 20, up to 1000 for big sites. Large sites split their sitemap into many files; reading stops at this cap and the row is marked partial.

## `maxItems` (type: `integer`):

Hard cap on domains read, 1 to 5000, e.g. 100. At $0.004 per domain with at least one URL found, 100 domains cost at most $0.40 before URL rows. Domains beyond the cap are not read.

## Actor input object example

```json
{
  "domains": [
    "apify.com",
    "shopify.com",
    "stripe.com"
  ],
  "includeUrls": false,
  "maxUrlsPerDomain": 5000,
  "maxSitemapsPerDomain": 20,
  "maxItems": 3
}
```

# Actor output Schema

## `domains` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "apify.com",
        "shopify.com",
        "stripe.com"
    ],
    "maxUrlsPerDomain": 5000,
    "maxSitemapsPerDomain": 20,
    "maxItems": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("lukehunter/sitemap-page-counter").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": [
        "apify.com",
        "shopify.com",
        "stripe.com",
    ],
    "maxUrlsPerDomain": 5000,
    "maxSitemapsPerDomain": 20,
    "maxItems": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("lukehunter/sitemap-page-counter").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "apify.com",
    "shopify.com",
    "stripe.com"
  ],
  "maxUrlsPerDomain": 5000,
  "maxSitemapsPerDomain": 20,
  "maxItems": 3
}' |
apify call lukehunter/sitemap-page-counter --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,lukehunter/sitemap-page-counter"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/q6ow6tFhAVWdeB50F/builds/m0drlxo6hYG5I8VLs/openapi.json