# Sitemap URL Extractor: All Website URLs + New Page Alerts (`manageable_gully/sitemap-url-extractor`) Actor

Extract every URL of a website from its XML sitemaps: auto-finds sitemaps via robots.txt, follows sitemap indexes and .gz files, returns lastmod, priority, images, hreflang and news tags. Optional status check and only-new page alerts. $0.20 per 1,000 URLs.

- **URL**: https://apify.com/manageable_gully/sitemap-url-extractor.md
- **Developed by:** [WebIntel Tools](https://apify.com/manageable_gully) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: All Website URLs + New Page Alerts

Get **every URL of a website** from its **XML sitemaps** in one run. Give it a domain and the Actor finds the sitemaps itself (robots.txt `Sitemap:` lines, then `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` and more), follows **sitemap indexes** and **.xml.gz** files, and returns each page URL with **lastmod, changefreq, priority, images, hreflang alternates and Google News tags**. Schedule it with **only new URLs** to get alerts for new pages, new products or new articles on any website.

**$0.20 per 1,000 URLs.** Filtered, duplicate, off-site and already-seen URLs are free.

### What you get for each URL

| Field | Example |
|---|---|
| `url` | https://blog.apify.com/ |
| `lastmod`, `lastmodIso` | 2026-10-22T09:45:12.067Z (raw), 2026-10-22T09:45:12Z (normalised UTC) |
| `changefreq`, `priority` | daily, 0.8 |
| `imageCount`, `images` | image URLs from image sitemaps |
| `alternates` | hreflang language versions: `[{"hreflang": "de", "url": "https://.../de/"}]` |
| `news` | Google News sitemap title, publication date, publication name, language |
| `videoCount` | videos tagged in a video sitemap |
| `sitemapUrl`, `sitemapType`, `site` | the sitemap file the URL came from (urlset, text, rss, atom) |
| `statusCode`, `finalUrl`, `redirected`, `ok` | optional HTTP status check: find 404s and redirects in your sitemap |

A summary per site (how the sitemap was found, sitemaps read, URLs listed/saved/filtered, errors, broken URLs) is saved to the `OUTPUT` record.

### Use cases

- **SEO audits and site migrations:** a full URL inventory with lastmod dates; check status codes to find 404s and redirects your sitemap still lists.
- **New page monitoring:** schedule daily with `onlyNew` to see every new blog post, product, job or landing page a competitor publishes.
- **Content and news tracking:** news sitemaps give fresh article URLs with titles and publication dates within minutes.
- **Crawl seeding:** feed the URL list into a crawler or into an LLM / RAG pipeline (e.g. a page-to-Markdown Actor) instead of crawling the whole site.
- **E-commerce:** list all product URLs of a shop, filter with `/products/*`, and track new ones.
- **International SEO:** hreflang alternates for every URL in one table.

### Input examples

All URLs of a site (the sitemap is found automatically):

```json
{ "startUrls": [{ "url": "https://blog.apify.com" }], "maxUrlsPerSite": 0 }
```

Only blog posts modified in the last 7 days, several sites:

```json
{ "startUrls": [{ "url": "https://site-a.com" }, { "url": "https://site-b.com/sitemap_index.xml" }], "includePatterns": ["/blog/*"], "lastmodFrom": "7 days" }
```

Daily new-page alerts (schedule it):

```json
{ "startUrls": [{ "url": "https://competitor.com" }], "onlyNew": true, "stateKey": "competitor", "maxUrlsPerSite": 0 }
```

The first run returns every URL and remembers them; later runs return only URLs that are new since then.

Find broken URLs in your sitemap:

```json
{ "startUrls": [{ "url": "https://your-site.com" }], "checkStatus": true, "maxUrlsPerSite": 0 }
```

### Supported sitemap formats

XML sitemaps and sitemap indexes (any depth), gzip (`.xml.gz`), plain-text sitemaps (one URL per line), RSS 2.0 and Atom feeds used as sitemaps, image / video / news / hreflang (xhtml:link) extensions, W3C and RFC 822 dates, relative URLs.

### Pricing

Pay per event: **$0.0002 per URL** saved + $0.0003 per run start. 1,000 URLs cost $0.20; 100,000 URLs cost $20. Optional status check: $0.0005 per URL checked. Set *Maximum cost per run* and the Actor stops exactly there.

### Good to know

- Sitemaps and robots.txt are published by site owners for crawlers, so this is their intended use. Only page URLs and the metadata in the sitemap are returned; no page content and no personal data.
- Status checks use one HEAD request per URL, a few in parallel, and skip URLs disallowed by the site's robots.txt (those are not charged).
- A site without any sitemap is reported in the summary and costs nothing beyond the run start.
- `maxSitemapsPerSite` (default 500) protects you from giant sitemap trees; raise it for big shops.

### FAQ

**How is this different from crawling?** A crawler follows links page by page; a sitemap lists the pages the owner wants indexed, including ones with no internal links, in seconds and for a fraction of the cost.
**What if a site has no sitemap?** The summary says so. Use a crawler for those sites.
**Can I export to CSV/Excel or Google Sheets?** Yes, from the dataset tab, or via the Apify API and integrations.

Found a problem or need a field? Open an issue on the Actor page and we'll fix it fast.

# Actor input Schema

## `startUrls` (type: `array`):

A website (https://example.com: sitemaps are found via robots.txt, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml) or a direct sitemap URL (.xml, .xml.gz, .txt, RSS/Atom). Sitemap indexes are followed automatically.

## `maxUrlsPerSite` (type: `integer`):

Stop after this many URLs per website. 0 = all URLs.

## `maxItems` (type: `integer`):

Overall limit for the run. 0 = no limit.

## `includePatterns` (type: `array`):

Keep only URLs containing one of these patterns. '*' is a wildcard (e.g. '/blog/*', '/products/'); prefix 're:' for a regular expression. Filtered URLs are free.

## `excludePatterns` (type: `array`):

Drop URLs containing one of these patterns (e.g. '/tag/', '/author/', 're:\\.pdf$'). Filtered URLs are free.

## `lastmodFrom` (type: `string`):

Only URLs whose sitemap lastmod is on/after this date: YYYY-MM-DD or relative ('24 hours', '7 days', '3 months'). Great for 'pages changed this week'.

## `lastmodTo` (type: `string`):

Only URLs with lastmod on/before this date (YYYY-MM-DD or relative).

## `skipUrlsWithoutLastmod` (type: `boolean`):

With a date filter, drop URLs that have no lastmod in the sitemap (they are kept by default).

## `onlyNew` (type: `boolean`):

Return only URLs not returned by earlier runs with the same state key. Schedule daily to get new pages / new products / new articles of any website. Already-seen URLs are free.

## `stateKey` (type: `string`):

Name of the saved list of seen URLs. Use a different key per monitored site or task.

## `checkStatus` (type: `boolean`):

Send a HEAD request to each returned URL and add statusCode, finalUrl, redirected and ok (finds 404s and redirects listed in your sitemap). robots.txt is respected. Adds a small per-check charge.

## `includeExtras` (type: `boolean`):

Add image URLs, hreflang alternates, Google News title/date and video counts when the sitemap has them.

## `sameSiteOnly` (type: `boolean`):

Skip URLs on other domains (subdomains of the same site are kept). Skipped URLs are free.

## `maxSitemapsPerSite` (type: `integer`):

Safety limit on sitemap files read per site (large shops can have thousands).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://blog.apify.com"
    }
  ],
  "maxUrlsPerSite": 1000,
  "skipUrlsWithoutLastmod": false,
  "onlyNew": false,
  "stateKey": "default",
  "checkStatus": false,
  "includeExtras": true,
  "sameSiteOnly": true,
  "maxSitemapsPerSite": 500
}
```

# Actor output Schema

## `results` (type: `string`):

One row per page URL: lastmod, changefreq, priority, images, hreflang alternates, news tags, source sitemap, optional HTTP status.

## `summary` (type: `string`):

Per site: how sitemaps were found, sitemaps read, URLs listed/saved/filtered/already seen, errors, broken URLs.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://blog.apify.com"
        }
    ],
    "maxUrlsPerSite": 1000
};

// Run the Actor and wait for it to finish
const run = await client.actor("manageable_gully/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://blog.apify.com" }],
    "maxUrlsPerSite": 1000,
}

# Run the Actor and wait for it to finish
run = client.actor("manageable_gully/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://blog.apify.com"
    }
  ],
  "maxUrlsPerSite": 1000
}' |
apify call manageable_gully/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,manageable_gully/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0mawRNTtLVdbNjild/builds/uN8ZaU4NFiz2eINCo/openapi.json