# Sitemap URL Extractor (`signalfeed/sitemap-url-extractor`) Actor

Get every page URL a website publishes in its sitemaps: auto-discovers sitemaps from robots.txt, follows sitemap indexes and gzip, filters by lastmod date and URL pattern, optional HTTP status check. $0.30 per 1,000 URLs.

- **URL**: https://apify.com/signalfeed/sitemap-url-extractor.md
- **Developed by:** [Aaron Rabinowe](https://apify.com/signalfeed) (community)
- **Categories:** SEO tools, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: every page a website publishes, in seconds

**Get the full list of page URLs from any website's sitemaps, for $0.30 per 1,000 URLs.** Enter a domain and the Actor finds its sitemaps (via robots.txt and common locations), follows sitemap indexes, unpacks `.gz` files, and returns every URL with its `lastmod`, `changefreq` and `priority`. Filter by date or URL pattern, and optionally check each URL's live HTTP status to find broken pages.

Sitemaps exist so machines can read them, so this is fast, light and reliable: apify.com's 51,000+ URLs are listed in seconds.

### What people use it for

- **SEO audits:** find every indexable page, spot 404s and redirects (*Check HTTP status*), and compare against what's ranking.
- **Content monitoring:** get pages added or updated since a date, such as a competitor's new blog posts or product pages.
- **Crawling & AI/RAG pipelines:** seed a crawler or an LLM ingestion job with a clean URL list instead of spidering the whole site.
- **Migrations & QA:** inventory every URL before a redesign, then verify nothing broke.

### Input

```json
{
  "startUrls": ["stripe.com", "https://example.com/sitemap_index.xml"],
  "lastmodAfter": "2026-09-01",
  "urlIncludes": ["/blog/"],
  "checkStatus": false,
  "maxUrls": 10000
}
```

### Output

| Field | Example |
|---|---|
| `url` | `https://stripe.com/blog/...` |
| `lastmod` | `2026-10-01` (as published in the sitemap) |
| `status` | `200` (only with *Check HTTP status*) |
| `changefreq`, `priority` | `daily`, `0.8` |
| `site`, `sitemap` | the site and the exact sitemap file the URL came from |

The `SUMMARY` record lists, per site, how many sitemaps and URLs were found, which sites had no sitemap, and how many URLs a date filter skipped because they had no `lastmod`.

### Pricing

**$0.30 per 1,000 URLs** ($0.0003 per URL) on the free plan, cheaper on larger Apify plans. You pay only for URLs saved; duplicates are free. Set *Max URLs* or a maximum cost per run and the Actor stops exactly there.

### FAQ

**A site returned "no sitemap found".** It doesn't publish one in robots.txt or the standard locations. If you know its sitemap URL, paste it directly.

**Why were URLs skipped with a date filter?** Many sitemaps (Stripe's, for example) don't include `lastmod`. Those URLs can't be date-filtered, so they're skipped and counted in the summary.

**Is this allowed?** Sitemaps are published specifically for automated tools to read. The Actor only fetches robots.txt and sitemap files (plus a HEAD request per URL if you enable status checks).

### More questions

**What if a site has no sitemap.xml?**

The Actor checks robots.txt for sitemap locations and common sitemap paths, and follows sitemap indexes and gzip files automatically.

**Can I check which URLs are broken?**

Yes. Turn on the status check to get the HTTP status of every URL, which is useful for SEO audits and migrations.

**Can I get only recently updated pages?**

Yes. Filter by the lastmod date to monitor new or changed pages.

**How do I run it on a schedule?**

In Apify Console, open the Actor and click **Schedule**: hourly, daily or any cron expression. Each run's results land in a new dataset you can export as CSV, Excel or JSON.

**Can I call it from my own code or no-code tools?**

Yes. Use the Apify API or the official Python and JavaScript clients, or connect it to Make, Zapier, n8n, Google Sheets or a webhook through Apify's built-in integrations.

**Is there a free trial?**

Apify's free plan includes monthly platform credit, enough to test this Actor on your real data before paying anything.

### More tools from the same developer

- [Ats Jobs Scraper](https://apify.com/signalfeed/ats-jobs-scraper): every open job from a company's careers page (Workday, Greenhouse, Lever, Ashby and more)
- [Tech Job Search](https://apify.com/signalfeed/tech-job-search): search 4,400+ tech and startup career pages by keyword
- [Remote Jobs Search](https://apify.com/signalfeed/remote-jobs-search): remote roles straight from company career pages
- [Job Salary Data](https://apify.com/signalfeed/job-salary-data): real salary ranges from live job postings
- [Hiring Companies Finder](https://apify.com/signalfeed/hiring-companies-finder): companies hiring for a role right now (sales hiring signals)
- [Website Contact Extractor](https://apify.com/signalfeed/website-contact-extractor): emails, phones and social profiles from a list of websites
- [Domain Intelligence](https://apify.com/signalfeed/domain-intelligence): WHOIS/RDAP dates, DNS, email provider and tech stack for domains
- [Article Extractor](https://apify.com/signalfeed/article-extractor): clean Markdown of any web page, ready for LLMs and RAG
- [Website Screenshot](https://apify.com/signalfeed/website-screenshot): full-page and mobile screenshots without cookie banners
- [Pdf Text Extractor](https://apify.com/signalfeed/pdf-text-extractor): text from PDFs at any URL, page by page
- [Rss Feed Reader](https://apify.com/signalfeed/rss-feed-reader): any RSS or Atom feed as clean JSON

Questions or a feature request? Open an issue on the Actor's **Issues** tab and I'll usually reply within a day.

# Actor input Schema

## `startUrls` (type: `array`):

Domains (e.g. <code>stripe.com</code>) are auto-discovered via robots.txt and common sitemap locations. You can also paste sitemap URLs directly (XML, sitemap index, .gz or .txt).

## `lastmodAfter` (type: `string`):

Keep URLs whose sitemap lastmod is on/after this date. URLs without a lastmod are skipped when this is set (the run summary tells you how many).

## `urlIncludes` (type: `array`):

e.g. <code>/blog/</code> or <code>/products/.+</code>

## `urlExcludes` (type: `array`):

e.g. <code>/tag/</code>, <code>?page=</code>

## `checkStatus` (type: `boolean`):

Adds the live HTTP status (200, 301, 404, …) per URL to find broken pages. Slower.

## `maxUrls` (type: `integer`):

Stop after this many URLs. You are charged per URL saved.

## Actor input object example

```json
{
  "startUrls": [
    "apify.com"
  ],
  "checkStatus": false,
  "maxUrls": 1000
}
```

# Actor output Schema

## `urls` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "apify.com"
    ],
    "maxUrls": 1000
};

// Run the Actor and wait for it to finish
const run = await client.actor("signalfeed/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["apify.com"],
    "maxUrls": 1000,
}

# Run the Actor and wait for it to finish
run = client.actor("signalfeed/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "apify.com"
  ],
  "maxUrls": 1000
}' |
apify call signalfeed/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,signalfeed/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fNTyEhk7LZIXNcwbs/builds/cgQZ9zGdbXJnDqGup/openapi.json
