# Sitemap Extractor: Every Page URL of Any Website (`planetlinux98/sitemap-extractor`) Actor
Get every page URL of a website from its sitemaps, found automatically (robots.txt, sitemap indexes, compressed, text and RSS sitemaps), with a link-following fallback when there's no sitemap. Track new and removed pages, and check pages for broken links, redirects and noindex.
- **URL**: https://apify.com/planetlinux98/sitemap-extractor.md
- **Developed by:** [Elliott Tyson](https://apify.com/planetlinux98) (community)
- **Categories:** Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $0.25 / 1,000 url founds
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event
## What's an Apify Actor?
An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.
For examples already wired to this Actor's own input schema, see the [API](#api) section below.
Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).
# README
**Sitemap Extractor** gets **every page URL of a website** in one clean list. Type a website and it finds the sitemaps for you: in robots.txt, at the usual addresses, inside sitemap indexes, compressed (.gz), plain-text and RSS or Atom sitemaps. **If a site has no sitemap, it follows the site's links instead**, so you still get its pages rather than an error.
It can also **watch a site for new, updated and removed pages** on a schedule, and **check every listed page** for the problems search engines complain about: broken pages, redirects, "noindex" and canonical links that point elsewhere.
It's built for SEO specialists auditing sites, developers and AI builders who need a site's pages to crawl or feed into a pipeline, and anyone keeping an eye on a competitor's new products or articles.
### What can this sitemap extractor do?
- **Find sitemaps on its own:** robots.txt first, then the usual places (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` and others) and links on the homepage. Sitemap indexes are followed down to every child sitemap.
- **Read sloppy sitemaps too:** compressed files, missing namespaces, stray characters, unclosed tags, relative addresses and odd date formats. A web page served at a sitemap address is recognised and skipped instead of producing junk.
- **Fall back to following links** when there's no sitemap, staying on the site, respecting robots.txt and "nofollow", and stopping at a page limit you choose.
- **Find pages missing from the sitemap:** in "Both" mode it reads the sitemaps and follows the links, and marks pages that are linked on the site but not in its sitemap.
- **Track changes:** in monitoring mode it remembers what it found and returns only **new**, **updated** and **removed** URLs on the next run.
- **Check pages (optional):** final status after redirects, the redirect chain, response time, noindex (meta tag or header), canonical URL, whether robots.txt blocks Google, and the page title.
- **Explain sitemap problems in plain words:** a "Websites" table lists each site's sitemaps and issues such as URLs on the wrong site, http URLs on an https site, unreadable dates, duplicates, oversized files, or a sitemap missing from robots.txt.
- **Filter before you pay:** keep only URLs containing `/blog/`, drop `/tag/`, or keep only pages changed in the last 7 days.
- **Group by section:** each URL comes with its top-level folder (`/blog/`, `/products/`, or `/en/services/` on sites that start with a language), and each site gets a count per section.
### How to get all URLs of a website
1. Click **Try for free** (you'll need a free Apify account).
2. Add one or more websites, one per line: `example.com`, a sitemap address, or a section like `https://example.com/blog/`.
3. Optionally add filters, turn on **Check each page**, or turn on **monitoring**.
4. Click **Start**.
5. Download the URLs as JSON, CSV or Excel, or connect the results to Google Sheets, Slack, Make or Zapier. For monitoring, schedule the run daily or weekly.
### Input example
```json
{
"websites": ["crawlee.dev", "https://www.python.org"],
"discoveryMode": "auto",
"includeUrlPatterns": ["/blog/"],
"maxUrlsPerWebsite": 1000,
"checkPages": false,
"onlyChanges": false
}
```
### Output example
One URL as JSON, from a real run with page checks on:
```json
{
"url": "https://crawlee.dev/blog/archive",
"website": "crawlee.dev",
"source": "sitemap",
"sitemapUrl": "https://crawlee.dev/sitemap.xml",
"inSitemap": null,
"lastmod": null,
"changefreq": "weekly",
"priority": 0.5,
"title": "Archive | Crawlee for JavaScript ยท Build reliable crawlers. Fast.",
"section": "/blog/",
"alternates": [],
"changeType": null,
"previousLastmod": null,
"firstSeenAt": null,
"checkStatus": "checked",
"statusCode": 200,
"finalUrl": "https://crawlee.dev/blog/archive",
"redirectChain": [],
"responseTimeMs": 29,
"canonicalUrl": "https://crawlee.dev/blog/archive",
"noindex": false,
"issues": [],
"checkError": null,
"foundAt": "2026-10-14T05:25:48.770Z"
}
```
The page check finds what a sitemap hides. In this example, based on a real run on a large news site (its name is changed here), the sitemap still lists an old address that goes through four redirects before reaching the page (shortened):
```json
{
"url": "http://www.example.co.uk/news/2014/09/140924_story.shtml",
"statusCode": 200,
"finalUrl": "https://www.example.com/news/2014/09/140924_story",
"redirectChain": [
{ "url": "http://www.example.co.uk/news/2014/09/140924_story.shtml", "statusCode": 302 },
{ "url": "https://www.example.co.uk/news/2014/09/140924_story.shtml", "statusCode": 301 },
{ "url": "https://www.example.com/news/2014/09/140924_story.shtml", "statusCode": 301 },
{ "url": "http://www.example.com/news/2014/09/140924_story", "statusCode": 302 }
],
"issues": ["redirects"]
}
```
The **Websites** table has one row per site. A real example (shortened) from a monitoring run that kept only `/blog/` URLs:
```json
{
"website": "crawlee.dev",
"status": "ok",
"discoveryMethod": "sitemaps",
"sitemapsFoundBy": "robots-txt",
"sitemapCount": 2,
"urlsFound": 4844,
"urlsMatchingFilters": 36,
"urlsSaved": 36,
"sections": [
{ "section": "/js/api/", "urls": 3123 },
{ "section": "/python/", "urls": 971 },
{ "section": "/js/docs/", "urls": 708 }
],
"sitemapIssues": [
{ "code": "duplicate-urls", "count": 12, "message": "Some URLs are listed more than once (we list each once)." }
],
"changes": { "firstRun": true, "new": 36, "updated": 0, "removed": 0, "note": null }
}
```
### How much does it cost?
You pay **$0.25 per 1,000 URLs**, plus a tiny start fee per run. With **Check each page** on, each page checked adds **$1.00 per 1,000 pages**.
- A 500-page website: about $0.13 (about $0.63 with page checks)
- A 20,000-URL online store: about $5.00
- A daily monitor of a blog that adds a few posts a day: a few cents a month, because only new and changed URLs are charged
You aren't charged for websites that block us, for URLs filtered out, or for pages we couldn't check. Apify's free plan includes $5 of monthly credit, which covers about 20,000 URLs. To cap any single run, set **Max URLs** or a maximum cost when you start it.
### Sites that block bots
Some websites use bot protection (Cloudflare, Akamai, DataDome and similar) that turns away automated tools. This Actor identifies itself honestly and never tries to get around these protections. Instead:
- It checks each site first, so a site that blocks us is stopped within seconds.
- The site is listed in the **Websites** table as `blocked`, with the protection service when we can tell.
- **You aren't charged** for URLs from a blocked site, because there aren't any.
- Sitemaps are often left open even when pages are protected; in that case you still get the URLs.
If it's your own site, allow our bot (its name starts with `SitemapExtractor`) in your bot protection settings.
### Good to know
- **Polite by default:** at most 4 requests a second to any one site, and a site's robots.txt Crawl-delay always wins. Reading a sitemap is usually only a handful of requests, however many URLs it lists.
- **robots.txt is respected.** If a site asks bots to stay out, we do, and the Websites table says so.
- **Big sites:** sitemaps can list millions of URLs. Reading them is fast (200,000 URLs took under 3 minutes in our tests). Use **Max URLs per website** for a first look, and the run's maximum cost setting to cap spending. A run also stops by itself shortly before its time limit, so it ends successfully with the URLs found so far.
- **Last-modified dates come from the site.** Many sites leave them out or set them all to today, so treat them as hints.
- **Following links finds what's linked.** Pages that aren't linked from anywhere (and aren't in a sitemap) can't be found. Sites that build their pages with JavaScript may show only a few links; the Websites table says when that seems to be the case.
- **Removed pages** are only reported when every sitemap was read successfully, and not after a sudden large drop (more often a temporary fault on the site than real deletions), so you don't get false alarms.
- **No personal data is collected.** Only page addresses and the site's own sitemap details.
### FAQ
#### How is this different from other sitemap extractors?
Most stop with an error when a site has no sitemap, or when the sitemap isn't where they expect it. This one looks in every usual place, reads broken and compressed sitemaps, and falls back to following links, so you get a result for almost any website. It also tracks changes, checks pages, and explains sitemap problems in plain words.
#### Can I find pages that are missing from my sitemap?
Yes. Choose **Both** under "How to find URLs". Pages that are linked on the site but not listed in its sitemaps have `inSitemap: false`.
#### Can I get only new pages, for example a competitor's new products?
Yes. Turn on monitoring, add a filter such as `/products/`, and schedule the run. The first run returns everything; after that you get only new, updated and removed URLs.
#### Can I use it through the API or with AI agents?
Yes. Start runs and read results through the Apify API (JavaScript and Python clients are available), call it from AI agents through Apify's MCP server, or schedule it and send results to Google Sheets, Slack, Make or Zapier.
#### Something's wrong or unclear?
Please open an issue on the **Issues** tab with the website and what you expected.
# Actor input Schema
## `websites` (type: `array`):
One per line. Type a website (example.com) and we find its sitemaps for you: in robots.txt, at the usual addresses, or as compressed, text or RSS sitemaps. If a site has no sitemap, we follow its links instead. You can also paste a sitemap address (https://example.com/sitemap.xml) to read just that one, or a section of a site (https://example.com/blog/) to keep only URLs under it.
## `discoveryMode` (type: `string`):
Sitemaps, or follow links if there's no sitemap works for almost any site. Sitemaps only never crawls. Follow links only ignores sitemaps. Both reads the sitemaps and also follows links, and marks pages that are linked on the site but missing from its sitemap (inSitemap: false).
## `maxCrawlPages` (type: `integer`):
Only used when we follow a site's links. Each page visited is one request to the site. Reading sitemaps doesn't count against this.
## `includeUrlPatterns` (type: `array`):
Keep only URLs that contain at least one of these. Use * for any text: /blog/, /products/*-red or \*.pdf. Matching ignores case. Leave empty to keep every URL.
## `excludeUrlPatterns` (type: `array`):
Drop URLs that contain any of these, for example /tag/, ?page= or /fr/. Same rules as above.
## `modifiedWithinDays` (type: `integer`):
Keep only URLs whose last-modified date (from the sitemap) is within this many days. URLs without a date are left out when this is set, because we can't tell. Leave empty or 0 for no date filter.
## `includeImages` (type: `boolean`):
Adds an images list to each URL when the sitemap lists images for that page.
## `checkPages` (type: `boolean`):
Visits every URL that's saved and reports what search engines would see: the HTTP status after redirects, the redirect chain, response time, whether the page says "noindex", where its canonical link points, whether robots.txt blocks Google, and the page title. Problems are listed in plain words in the issues field. Charged as an extra event per page checked; pages we couldn't check (for example behind bot protection) aren't charged.
## `onlyChanges` (type: `boolean`):
Remembers the URLs it has already given you and returns only what changed: new pages, pages whose last-modified date moved forward, and pages that disappeared (changeType). The first run returns every URL. Ideal for scheduled runs that watch a site for new pages, products or articles. Keep the same filters between runs.
## `includeRemoved` (type: `boolean`):
In monitoring mode, also add a row for each URL that was there last time but is gone now (changeType: removed). To avoid false alarms, removals are only reported when every sitemap was read successfully.
## `monitorName` (type: `string`):
Optional. Give each monitor a name (for example competitor-products) so several schedules can watch the same site separately. Monitor memory is kept in a key-value store called sitemap-extractor-monitors in your account.
## `maxUrlsPerWebsite` (type: `integer`):
Stop saving URLs for a website after this many. 0 means no limit. Big sites can list millions of URLs, so a limit is a good idea for a first try.
## `maxUrls` (type: `integer`):
Stop once this many URLs have been saved across all websites. 0 means no limit. You can also cap spending with the run's maximum cost setting.
## `maxRequestsPerSecond` (type: `integer`):
How fast we may load pages from one website when following links or checking pages. The default is polite for any site. Only raise it for sites you own or manage. A site's robots.txt Crawl-delay always wins.
## Actor input object example
```json
{
"websites": [
"https://crawlee.dev",
"https://www.python.org"
],
"discoveryMode": "auto",
"maxCrawlPages": 500,
"includeImages": false,
"checkPages": false,
"onlyChanges": false,
"includeRemoved": true,
"maxUrlsPerWebsite": 50,
"maxUrls": 0,
"maxRequestsPerSecond": 4
}
```
# Actor output Schema
## `urls` (type: `string`):
One row per URL found, with its sitemap details (and page check results, when on).
## `websites` (type: `string`):
One row per website: how its URLs were found, its sitemaps and any sitemap problems.
## `summary` (type: `string`):
Totals for the run, invalid entries, and anything not processed.
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"websites": [
"https://crawlee.dev",
"https://www.python.org"
],
"maxUrlsPerWebsite": 50
};
// Run the Actor and wait for it to finish
const run = await client.actor("planetlinux98/sitemap-extractor").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`๐พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// ๐ Want to learn more ๐? Go to โ https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {
"websites": [
"https://crawlee.dev",
"https://www.python.org",
],
"maxUrlsPerWebsite": 50,
}
# Run the Actor and wait for it to finish
run = client.actor("planetlinux98/sitemap-extractor").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print(f"๐พ Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
print(item)
# ๐ Want to learn more ๐? Go to โ https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"websites": [
"https://crawlee.dev",
"https://www.python.org"
],
"maxUrlsPerWebsite": 50
}' |
apify call planetlinux98/sitemap-extractor --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"type": "http",
"url": "https://mcp.apify.com/?tools=fetch-actor-details,planetlinux98/sitemap-extractor"
}
}
}
```
The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).
## OpenAPI specification
Download the OpenAPI definition: https://api.apify.com/v2/actors/IADstuQLmZGayBPQy/builds/X2sSYLDhGYMBIedru/openapi.json