# Sitemap URL Extractor: Get All URLs from a Website (`weio/sitemap-url-extractor-bulk`) Actor

Sitemap URL extractor for websites you supply. Get all URLs from XML sitemaps, including sitemap indexes and .xml.gz files, with lastmod, priority, source and optional HTTP status. Robots.txt is respected; this actor does not crawl JavaScript pages.

- **URL**: https://apify.com/weio/sitemap-url-extractor-bulk.md
- **Developed by:** [Weio, Inc.](https://apify.com/weio) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 sitemap url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap extractor: get all URLs from a website

This **sitemap URL extractor** gets all URLs from a website's XML sitemaps, with each URL's `lastmod`, `changefreq`
and `priority`, the sitemap file it came from, and (optionally) its live HTTP status. Useful for SEO audits, site
migrations, content inventories, change monitoring and feeding other crawlers a clean URL list.

### How it finds the sitemaps

1. The `Sitemap:` lines in the site's `robots.txt`.
2. If robots.txt lists none: `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml`.
3. Sitemap index files are followed up to 5 levels deep. Gzipped sitemaps (`.xml.gz`), files with a BOM, files with
   or without the sitemaps.org namespace, plain-text sitemaps (one URL per line) and slightly broken XML are read.
   Each URL is output once per site.

### Input

| Input | Default | What it does |
|---|---:|---|
| `startUrls` | one prefilled site | Required website(s), or any URL on them. The prefill is an example; a blank API run is rejected before work starts. |
| `maxUrlsPerSite` | 200 | Maximum sitemap URLs from each site (1–50,000). |
| `maxItems` | 200 | Compatibility output cap per site; the lower of this and `maxUrlsPerSite` applies. |
| `lastmodAfter` | none | Optional `YYYY-MM-DD` sitemap `lastmod` filter. |
| `urlPattern` | none | Optional case-insensitive URL regex. |
| `checkStatus` | `false` | Opt in to HTTP status checks (HEAD, then GET if necessary). |

```json
{ "startUrls": [{ "url": "https://example.com" }], "maxUrlsPerSite": 200, "urlPattern": "/blog/", "checkStatus": false }
```

### Output

One dataset row per URL:

| field | meaning |
|---|---|
| `site` | the site's origin, e.g. `https://example.com` |
| `url` | the URL listed in the sitemap |
| `lastmod`, `changefreq`, `priority` | as written in the sitemap (empty when the sitemap leaves them out) |
| `sitemapSource` | the sitemap file that lists it |
| `depth` | 0 for a sitemap found directly, 1+ for sitemaps reached through an index |
| `status` | HTTP status when status checking is on |
| `error` | why a site (or a status check) failed; a failed site gets one row with only `site` and `error` |

The key-value store gets a `SUMMARY-<host>` record per site (sitemaps found and read, sitemaps that failed and why,
URLs listed, duplicates, how many filters removed, whether the run stopped at your maximum) and a `SUMMARY` record
listing them all.

### How it behaves on other people's sites

Requests identify themselves as `WeioBot`, obey robots.txt for every host and redirect,
are paced to about one request per second per site (slower if robots.txt asks), and back off on 429/503 answers.
A sitemap that robots.txt blocks is not read; the summary says so. Only public addresses are contacted.

### Limits

- No sitemap means **0 URLs and an error reason** in the dataset and per-site summary; it does not guess pages.
- This is not a JavaScript crawler: it only reads published sitemap files, so JavaScript-only routes are not discovered.
- Robots.txt is obeyed for sitemap discovery, sitemap files, status checks, redirects and every contacted host.
- Up to 1,000 sitemap files and 50,000 URLs per site per run; a gzipped sitemap may unpack to at most 60 MB.
- Status checking is paced per site, so 1,000 URLs take roughly 20 minutes.
- Sitemaps list what a site wants indexed; pages missing from them are not discovered (this actor does not crawl links).

### Pay per event

The default example is one site and up to 200 URLs: at most about **$0.062** before optional status checks
($0.002 for one processed site + up to 200 × $0.0003 extracted URLs). A site that cannot be processed, including an
invalid, unreachable or robots-refused site, is free. Enabling `checkStatus` adds $0.0005 only for each URL whose
status is actually checked. The free-tier price for extracted URLs is $0.0003, below the comparable $0.0005 tier
named in the market brief.

### Use the URL list in n8n or Make

Use this actor when an SEO audit, migration, or content inventory needs the URLs a site already publishes—not a web
crawl. Set your own output cap before the first paid run.

- **n8n:** add the Apify node, choose **Run Actor**, select `weio/sitemap-url-extractor-bulk`, and pass
  `{"startUrls":[{"url":"https://example.com"}],"maxItems":100}`. Follow it with **Get Dataset Items** and send
  the URL rows to a sheet, database, or audit step. Use your own Apify credential; do not put a token in workflow JSON.
- **Make:** add **Apify > Run an Actor** with the same input, then **Get Dataset Items** (or **Watch Actor Runs** for
  longer jobs). Set the dataset limit explicitly. Connect using your own Apify OAuth/token in Make.

The prefill is only an example. A blank API run is rejected before work starts. `checkStatus` is optional and only
checks sitemap URLs that the site's robots rules allow.

Built and maintained by Weio, Inc. The actor was written and is operated by AI agents (Anthropic Claude), with a
person accountable at Weio.

# Actor input Schema

## `startUrls` (type: `array`):

Websites or any URL on them, one per line. The actor finds each site's sitemaps itself (robots.txt Sitemap lines, then /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml).

## `maxUrlsPerSite` (type: `integer`):

Stop reading a site's sitemaps after this many output URLs (after filters).

## `maxItems` (type: `integer`):

An additional per-site output limit for generic Apify integrations. The lower of this and Max URLs per site applies.

## `lastmodAfter` (type: `string`):

Optional date (YYYY-MM-DD). Keeps URLs whose sitemap lastmod is on or after it; URLs without a lastmod are left out when this is set.

## `urlPattern` (type: `string`):

Optional regular expression (case-insensitive), e.g. /blog/ or .pdf$. A pattern that is not a valid regex is matched as plain text.

## `checkStatus` (type: `boolean`):

Also request every output URL (HEAD, then GET if HEAD fails) and add its status. Slower: requests to one site are paced to about one per second and obey robots.txt.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://weio.ai"
    }
  ],
  "maxUrlsPerSite": 200,
  "maxItems": 200,
  "checkStatus": false
}
```

# Actor output Schema

## `urls` (type: `string`):

The default dataset, URLs view.

## `summary` (type: `string`):

SUMMARY record: one entry per site with its own SUMMARY-<host> record.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://weio.ai"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("weio/sitemap-url-extractor-bulk").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://weio.ai" }] }

# Run the Actor and wait for it to finish
run = client.actor("weio/sitemap-url-extractor-bulk").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://weio.ai"
    }
  ]
}' |
apify call weio/sitemap-url-extractor-bulk --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,weio/sitemap-url-extractor-bulk"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/dr9Z1t2PJaBTkzQmn/builds/ZgedWLndCRKEbRr6j/openapi.json