# Sitemap Extractor: All URLs, Lastmod, Status Codes (`joesimo/sitemap-extractor`) Actor

Get every URL from a website's sitemaps. Finds sitemaps through robots.txt and common paths, follows nested indexes, reads gzip, text, RSS and Atom. Returns lastmod, hreflang, images and news tags, with optional HTTP status and redirect checks. No login.

- **URL**: https://apify.com/joesimo/sitemap-extractor.md
- **Developed by:** [Joe Simo](https://apify.com/joesimo) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor: All URLs, Lastmod, Status Codes

Give it a website or a sitemap URL and get every page URL the site publishes in its sitemaps, one clean row per URL with `lastmod`, `changefreq`, `priority`, hreflang alternates, images, videos and Google News tags. Turn on status checks to also get the HTTP status code, redirect chain, final URL and X-Robots-Tag of every URL, so you can find broken pages, redirects and noindex pages that are still listed in the sitemap.

Empty input runs a short demo (50 URLs from crawlee.dev), so you can try the Actor from the Store without filling in a form.

**$0.30 per 1,000 URLs** ($0.0003 per URL). **Status checks add $0.70 per 1,000 checked URLs** ($1.00 per 1,000 in total). Sites with no sitemap, failed sites, invalid inputs, duplicates and URLs removed by your filters are free.

### What it does

- **Finds the sitemaps for you.** Reads `Sitemap:` lines in robots.txt, then tries `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml` and `/sitemap.txt`. Works with WordPress (Yoast, Rank Math, core), Shopify, Wix, Squarespace, Webflow, Next.js, Docusaurus and hand-written sitemaps.
- **Follows nested sitemap indexes** (up to 6 levels) and reads **gzip** (`.xml.gz`), **plain text**, **RSS** and **Atom** sitemaps.
- **Extensions:** hreflang `xhtml:link` alternates, `image:image`, `video:video` and `news:news` entries.
- **Filters:** keep or skip URLs by substring or regular expression, keep only URLs changed after or before a date, and drop URLs on other hosts.
- **Status checks (optional):** one HEAD request per URL (GET when HEAD is refused), manual redirect following with the full chain, final status, content type, content length, Last-Modified and X-Robots-Tag (with a `noindex` flag).
- **Many sites in one run.** Each URL is saved and charged once per run even if several sitemaps list it.

### Use cases

- Seed a crawler or a RAG pipeline with the full list of pages, without crawling.
- SEO audits: broken (4xx, 5xx) and redirected URLs in the sitemap, noindex pages, missing lastmod, hreflang coverage.
- Track new and updated pages on a competitor's site with `lastmodAfter` on a schedule.
- Export all product, blog or category URLs of a store with `includePatterns`.

### Input examples

| Goal | Input |
| --- | --- |
| All URLs of a site | `"startUrls": ["example.com"]` |
| A specific sitemap or index | `"startUrls": ["https://example.com/sitemap_index.xml"]` |
| Only blog posts changed since a date | `"startUrls": ["example.com"], "includePatterns": ["/blog/"], "lastmodAfter": "2026-10-01"` |
| Product URLs only (regex) | `"includePatterns": ["/\\/products\\/[^/]+$/"]` |
| Broken link and redirect check | `"startUrls": ["example.com"], "checkStatus": true` |
| First 1,000 URLs of each site | `"startUrls": ["a.com", "b.com"], "maxUrlsPerSite": 1000` |

**Max URLs in total** caps the run. You can also set a maximum cost per run in the Apify Console; the Actor stops exactly at that amount.

### Output

Each paid row has `status: ok` and:

- `url`, `site`, `input`, `lastmod` (ISO 8601), `lastmodRaw`, `changefreq`, `priority`, `title` (RSS and Atom only)
- `alternates` (hreflang, url), `imageCount`, `images` (url, title, caption), `videos`, `news` (title, publication date, publication name, language)
- `sitemapUrl`, `sitemapType` (urlset, text, rss, atom), `parentSitemap`, `sitemapLastmod`, `depth`
- With status checks: `statusChecked`, `statusCode`, `ok`, `redirected`, `finalUrl`, `redirectChain`, `contentType`, `contentLength`, `lastModifiedHeader`, `xRobotsTag`, `noindex`, `responseTimeMs`, `checkError`, `checkNote`
- `scrapedAt`

Example row:

```json
{
  "status": "ok",
  "url": "https://crawlee.dev/blog",
  "site": "https://crawlee.dev",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "sitemapType": "urlset",
  "statusChecked": true,
  "statusCode": 200,
  "ok": true,
  "redirected": false,
  "contentType": "text/html; charset=utf-8"
}
```

#### Free status rows

| status | Meaning |
| --- | --- |
| `no_sitemap` | No Sitemap line in robots.txt and none of the common paths returned a sitemap. Pass the sitemap URL directly if it lives elsewhere. |
| `empty` | Sitemaps were read but no URL was left after filters and duplicates |
| `failed` | Sitemaps were found but none could be downloaded or parsed (the reason is in `error`) |
| `invalid_input` | Not a public http(s) website or sitemap URL |

A run summary is saved as `OUTPUT` in the default key-value store: per site, how the sitemap was found, sitemap files read and failed, URLs seen, saved, filtered and duplicated, and the stop reason.

### How it behaves on the sites it reads

- Reads only robots.txt and sitemap files, which sites publish for automated readers. It does not crawl or render pages.
- Status checks send one lightweight HEAD request per URL (GET with the body discarded when HEAD is refused), 5 at a time per site by default, and skip paths that robots.txt disallows unless you turn that off.
- Sends a descriptive User-Agent, waits between requests to the same host, and backs off on HTTP 429 and 5xx answers.
- No login, no proxy, no captcha solving. Private and local network addresses are refused.

### FAQ

**Does it crawl the website?** No. It reads the sitemap files only, which is why it is fast and cheap. If a page is not in any sitemap, it will not be in the results.

**How big a site can it handle?** Sitemap files can hold 50,000 URLs each and indexes can list thousands of files. Runs of a few hundred thousand URLs work at the default 256 MB. Use `maxUrlsPerSite` or `maxItems` for a sample.

**Why is lastmod empty for some URLs?** The site did not include it. Many sitemap generators leave it out.

**Why did a site return `no_sitemap`?** Some sites have no sitemap or keep it at an unusual path. Find it in Google Search Console or the site's CMS and pass the URL directly.

### Related Actors

- [Wayback Machine Scraper](https://apify.com/joesimo/wayback-machine)
- [DNS Lookup](https://apify.com/joesimo/dns-lookup)
- [Tech Stack Detector](https://apify.com/joesimo/tech-stack-detector)
- [Website Screenshot](https://apify.com/joesimo/website-screenshot)

Questions or a field you need? Open an issue on the Actor's Issues tab.

# Actor input Schema

## `startUrls` (type: `array`):

One per line. A website (example.com or https://example.com) is searched for sitemaps through robots.txt and common paths such as /sitemap.xml and /wp-sitemap.xml. A sitemap URL (.xml, .xml.gz, .txt, RSS or Atom feed) is read directly, and sitemap indexes are followed. Leave empty for a demo of 50 URLs from crawlee.dev.

## `maxUrlsPerSite` (type: `integer`):

Stop reading a site after this many saved URLs. 0 means no limit (large stores can have millions).

## `maxItems` (type: `integer`):

Stop the whole run after this many saved URLs. 0 means no limit. The run also stops exactly at your maximum cost per run.

## `includePatterns` (type: `array`):

Keep a URL only if it matches at least one of these. Plain text is a case-insensitive substring (for example /blog/). Wrap in slashes for a regular expression, for example //products/\[^/]+$/. Filtered URLs are free.

## `excludePatterns` (type: `array`):

Drop URLs that match any of these (same rules as above). Filtered URLs are free.

## `lastmodAfter` (type: `string`):

Keep only URLs whose sitemap lastmod is on or after this date (YYYY, YYYY-MM or YYYY-MM-DD). URLs without a lastmod are dropped when this is set. Good for finding new or updated pages.

## `lastmodBefore` (type: `string`):

Keep only URLs whose lastmod is on or before this date. Good for finding stale pages.

## `sameHostOnly` (type: `boolean`):

Drop URLs on other hosts (www. is ignored), for example CDN or partner links listed in the sitemap.

## `checkStatus` (type: `boolean`):

Send one HEAD request per URL (GET if HEAD is refused) and record the status code, redirect chain, final URL, content type, Last-Modified and X-Robots-Tag. Finds broken pages, redirects and noindex pages listed in the sitemap. Costs an extra status-check event per checked URL.

## `respectRobotsTxt` (type: `boolean`):

Skip status checks on paths that robots.txt disallows for all user agents (the URL is still saved, marked as skipped, and the status-check event is not charged).

## `statusConcurrency` (type: `integer`):

Parallel status checks per site. Keep it low to be polite to the site.

## `includeExtras` (type: `boolean`):

Read the hreflang alternates, image, video and Google News extensions of each URL entry. Turn off for smaller rows.

## `maxSitemapsPerSite` (type: `integer`):

Safety cap on sitemap files read per site (indexes included).

## `timeoutSecs` (type: `integer`):

Timeout for each sitemap download or status check.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 0,
  "maxItems": 0,
  "sameHostOnly": false,
  "checkStatus": false,
  "respectRobotsTxt": true,
  "statusConcurrency": 5,
  "includeExtras": true,
  "maxSitemapsPerSite": 500,
  "timeoutSecs": 30
}
```

# Actor output Schema

## `results` (type: `string`):

One item per URL found in the sitemaps, plus free status rows for sites with no sitemap, failed sites and invalid inputs.

## `runSummary` (type: `string`):

JSON summary per site: how the sitemap was found, sitemaps read and failed, URLs seen, saved, filtered and duplicated, plus the stop reason.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("joesimo/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://crawlee.dev"] }

# Run the Actor and wait for it to finish
run = client.actor("joesimo/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ]
}' |
apify call joesimo/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,joesimo/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/r4YM0XFPnWHj9vcsd/builds/Hk6pcCmeScGoRi1Am/openapi.json
