# Sitemap URL Extractor with lastmod Filter (`daddyd-prints/sitemap-url-extractor`) Actor

Lists the page URLs in a site's sitemaps, with lastmod. Follows sitemap indexes and .xml.gz files. Optional changed-since date filter. Reads robots.txt first and never requests the pages.

- **URL**: https://apify.com/daddyd-prints/sitemap-url-extractor.md
- **Developed by:** [Derek Derringer](https://apify.com/daddyd-prints) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor with lastmod Filter

### What it does

This Actor lists the page URLs in a website's sitemaps. For each site you give, it reads robots.txt first, takes the sitemaps named there, follows sitemap indexes, and reads `.xml.gz` and plain text sitemaps. If robots.txt names no sitemap it can read, it can try four common sitemap paths, each only where robots.txt allows. Each page URL becomes one dataset row with its lastmod value, and an optional date filter keeps only pages changed on or after a day you pick. It requests only robots.txt files and files named as sitemaps, the files sites publish for machines. It never requests the pages themselves: each page URL is only checked against its host's robots.txt and, if allowed, written to your dataset.

### How to use it

1. Open the Actor in Apify Console.
2. The form opens with `https://docs.apify.com` in **Websites** and 20 in **Maximum URLs to save**, as a sample. Replace the sample site with your own sites, one per line, and set how many URLs you want.
3. If you know a sitemap's address, add it to **Sitemap URLs (optional)**. To get only recently changed pages, pick a date in **Only URLs changed since**.
4. Click **Start**.
5. When the run ends, open the **Output** tab for the URL list. REPORT and SKIPPED are in the run's key-value store, on the **Storage** tab.

### Input

| Field | What it does | Default | Limits |
| --- | --- | --- | --- |
| Websites (`sites`) | Domains or site addresses, one per line. `https://` is added when no scheme is given. Any path is ignored, so the whole site's sitemaps are read. An address ending in `.xml`, `.xml.gz` or `.txt` is read as a sitemap file instead | None. Prefilled with `https://docs.apify.com` | http and https only, no user name or password in the address, no control or invisible characters (an entry holding one is listed as `invalid_character`, even in a path that would be ignored). Up to 1,000 entries |
| Sitemap URLs (optional) (`sitemapUrls`) | Direct links to sitemap files or sitemap indexes, one per line. robots.txt is still read and obeyed first. For a host given only here, robots.txt Sitemap lines and common paths are not used | None | As for Websites. At least one website or sitemap URL is needed |
| Only URLs changed since (`changedSince`) | Saves only URLs whose lastmod is on or after this date. Empty saves every URL | Empty | YYYY-MM-DD or a full ISO date and time, read as UTC when it has no time zone. A date the Actor cannot read stops the run before any request, so no URL is charged |
| Keep URLs that have no lastmod (`includeUrlsWithoutLastmod`) | Used only with the date filter. On: URLs with no lastmod, or one that cannot be read, are saved too. Off: they are left out and counted in REPORT | Off | |
| Maximum URLs to save (`maxUrls`) | The run stops after saving this many URLs. Each saved URL is one charge, so this caps what a run can charge for URLs | 1,000. Prefilled with 20 | 1 to 500,000 |
| Maximum sitemap files to fetch (`maxSitemaps`) | Upper limit on sitemap files fetched in one run, index files and their children included | 200 | 1 to 5,000 |
| Maximum sitemap index depth (`maxSitemapDepth`) | Levels of sitemap indexes to follow. 0 reads only the top sitemap files, 1 also reads their children, and so on | 3 | 0 to 10 |
| Try common sitemap paths (`tryCommonPaths`) | When robots.txt names no sitemap that can be read, tries `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml` and `/sitemap.xml.gz`, one at a time, each only if robots.txt allows it, and stops at the first that works | On | Sites given in Websites only |
| Request timeout (`requestTimeoutSecs`) | Seconds to wait at each step of a request, such as connecting or the next part of the answer. A file that takes more than four times this in total is abandoned and listed as `timeout` | 30 seconds | 5 to 120 seconds |

### Output

Each saved URL is one dataset item. This real item is from a local test run of the prefilled input:

```json
{"url": "https://docs.apify.com/api", "lastmod": null, "sourceSitemap": "https://docs.apify.com/sitemap_base.xml", "site": "https://docs.apify.com"}
```

| Field | What it holds |
| --- | --- |
| `url` | The page URL as the sitemap lists it |
| `lastmod` | The lastmod text exactly as the sitemap gives it, or null when there is none. Text sitemaps have none |
| `sourceSitemap` | The sitemap file the URL was listed in, as the Actor requested it, before any redirect |
| `site` | The site from your input that the URL belongs to, as scheme and host |

That run requested `/robots.txt`, then `/sitemap.xml` (a sitemap index), then `/sitemap_base.xml`, and saved 20 URLs, none with a lastmod. With the date filter on, URLs like these are left out unless **Keep URLs that have no lastmod** is on.

### Pricing

This Actor is priced per event, with two charges:

- **Actor start:** Apify's automatic start event, $0.00005 per start (one event per GB of run memory, minimum one).
- **URL saved:** one charge for each URL written to the dataset.

**Price per URL: $0.20 per 1,000 URLs, which is $0.0002 each.**

Never charged: anything listed in SKIPPED (URLs robots.txt disallows, skipped sites and sitemap files, invalid URLs, including addresses with control or invisible characters), duplicates removed within the run, URLs left out by the date filter, anything past **Maximum URLs to save** or the sitemap limits, and the REPORT and SKIPPED records themselves.

If you set a maximum cost for the run and it is reached, the Actor saves no more URLs and requests no more files, writes REPORT saying the spending limit was reached and how many URLs were ready but not saved, and ends the run without an error. Nothing after the limit is saved or charged. If Apify restarts the run on another server, URLs already in the run's dataset are not saved or charged again. A new run starts from the beginning, so URLs saved by an earlier run are charged again.

### robots.txt and site rules

The robots.txt check is always on, and no input turns it off. Before any other request to a host (scheme, host and port), the Actor reads that host's robots.txt and follows the group for `SitemapUrlExtractor`, otherwise the `*` group. A group for `apify` also applies, and a path must pass both. Every skip below is listed in SKIPPED with the reason and is not charged.

| Case | What the Actor does |
| --- | --- |
| robots.txt disallows a sitemap file, a common sitemap path or a redirect target | Never requests it |
| robots.txt disallows a page URL listed in a sitemap | Does not save or charge it. Page URLs are never requested either way |
| A page URL or sitemap is on another host | Reads that host's robots.txt first, and only robots.txt for a page URL. Up to 25 hosts beyond your sites; past that, the URL or file is skipped |
| `Disallow: /` for this Actor, even with Allow lines | Skips the whole site |
| robots.txt answers 401, 403, 429 or a server error, cannot be reached, shows a login or CAPTCHA page, or redirects to another host somewhere other than its robots.txt | Skips the whole site. No other file is requested from it |
| robots.txt answers another 4xx, such as 404 | Treats the site as having no rules |
| A sitemap answers 401, 403 or 429, shows a login or CAPTCHA page, or redirects to a login page | Stops at the door: no more requests go to that host in this run, and it never tries to get past |
| Requests to one host | One at a time, at least 0.25 seconds apart, counted from the end of the last one. Up to 5 of your sites are read at once |
| robots.txt sets a Crawl-delay | Waits that long between requests to that host |
| The Crawl-delay is over 60 seconds | Skips the whole site rather than crawl it faster |

Only a 500, 502, 503 or 504 answer, a timeout or a network error is retried, once, after 2 seconds. The user agent is `SitemapUrlExtractor/0.1 (Apify Actor sitemap-url-extractor; reads robots.txt first)`, and it never borrows another crawler's name. No cookies are stored or sent and no proxy is used. There is no cookie, header, login, proxy or user agent input. On Apify, private and local network addresses are refused before any request, listed in SKIPPED as `private_address` and not charged.

### Your responsibility

You are responsible for submitting only sites you are allowed to access and for following each website's terms of use. The Actor checks robots.txt for you, but it cannot read or judge a site's terms.

### Run records

Two records in the run's key-value store, never charged:

| Record | What it holds |
| --- | --- |
| REPORT | The result, `finished`, `stopped early` or `input problem`, and the stop reason: `maxUrls was reached`, the spending limit, or `the run was aborted`. Start and finish times and duration. Input problems and adjustments, and the settings used, including the user agent and the caps. Counts: URLs saved, URLs listed in sitemaps, duplicates removed, URLs left out by the date filter (older, or without a lastmod), URLs disallowed by robots.txt, invalid URLs, URLs not saved at the spending limit, sitemap files fetched, requests made and robots.txt files read. Skips by reason. For each site: the sitemaps from robots.txt and from your input, the common paths tried, files read, URLs saved and the outcome. For each host: the robots.txt status and result, any skip or stop, the Crawl-delay used and the number of requests |
| SKIPPED | Every skipped site, sitemap file or URL: the address, the site, the stage (`input`, `robots.txt`, `sitemap`, `url` or `site`), a reason code such as `robots_txt_disallowed`, `site_disallows_all`, `robots_txt_server_error`, `crawl_delay_too_long`, `access_denied`, `rate_limited`, `login_page`, `captcha`, `timeout`, `too_large`, `unsafe_xml`, `private_address`, `invalid_character`, `max_depth` or `max_sitemaps`, a detail, and where it was found. Split into SKIPPED, SKIPPED-2 and so on every 10,000 entries. Past 50,000 entries the rest are only counted in REPORT. Duplicates and date-filtered URLs are counted in REPORT, not listed here |

### Limits

| Limit | Value |
| --- | --- |
| URLs | Maximum URLs to save, 1 to 500,000, 1,000 by default |
| Sitemap files | Maximum sitemap files to fetch, 1 to 5,000, 200 by default. Files past it are listed as `max_sitemaps` and never requested |
| Index depth | Maximum sitemap index depth, 0 to 10, 3 by default. Deeper files are listed as `max_depth` and not requested |
| File size | 50 MiB per sitemap file, before and after gzip. A larger file is skipped and listed. Gzip is unpacked 1 MiB at a time. Only the first 500 KiB of a robots.txt is read |
| Formats | XML sitemaps and sitemap indexes, `.xml.gz` files, answers sent with gzip encoding, and text sitemaps with one URL per line. A broken XML file keeps the URLs read before the error, and the error is listed |
| XML safety | Sitemaps are read with a safe XML parser that refuses DTDs, entity declarations and external references, so no sitemap can make the Actor expand or fetch anything, and such files are skipped as `unsafe_xml` |
| Date filter | A lastmod without a time zone is read as UTC, and `2026` or `2026-10` as the start of that year or month. With the filter on, a URL with no lastmod, or one that cannot be read, is left out unless Keep URLs that have no lastmod is on |
| Duplicates | Removed across the whole run. Two URLs count as the same when they match after lower-casing the scheme and host and dropping a default port and any `#` part |
| Hosts | At most 25 hosts beyond the sites you give |
| Input lists | Up to 1,000 entries in each list. Only the first 1,000 are used |
| Redirects | At most 5 per file. Each sitemap redirect is checked against robots.txt again |
| Memory | 256 MB by default, up to 1,024 MB |

# Actor input Schema

## `sites` (type: `array`):

Domains or site addresses, one per line, for example docs.apify.com. https is assumed when no scheme is given. Any path in an address is ignored: the Actor reads the sitemaps of the whole site, up to the maximum URLs setting. Sitemaps are found from the site's robots.txt Sitemap lines, then from common sitemap paths, but only where robots.txt allows. Exception: an address ending in .xml, .xml.gz or .txt is read as that one sitemap file.

## `sitemapUrls` (type: `array`):

Direct links to sitemap files or sitemap indexes, one per line. robots.txt is still read and obeyed for each host before anything is fetched.

## `changedSince` (type: `string`):

Optional date (YYYY-MM-DD). Only URLs whose sitemap lastmod is on or after this date are saved. A lastmod without a time zone is read as UTC. Leave empty to save every URL.

## `includeUrlsWithoutLastmod` (type: `boolean`):

Used only with the date filter. When on, URLs with no lastmod (or one that cannot be read) are saved too. When off, they are left out and counted in REPORT.

## `maxUrls` (type: `integer`):

The run stops after saving this many URLs. Each saved URL is one billable result, so this caps the cost of a run.

## `maxSitemaps` (type: `integer`):

Upper limit on sitemap files fetched in one run, counting index files and their children.

## `maxSitemapDepth` (type: `integer`):

How many levels of sitemap indexes to follow. 0 reads only the top sitemap files, 1 also reads their children, and so on.

## `tryCommonPaths` (type: `boolean`):

When robots.txt lists no sitemap, or none of the sitemaps it lists can be read, try /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /sitemap.xml.gz, one at a time, stopping at the first one that is a sitemap. Each path is tried only if robots.txt allows it. Not used for addresses given as sitemap files, or for a site that refused access.

## `requestTimeoutSecs` (type: `integer`):

Seconds to wait for each response. A file that takes longer than four times this value is abandoned.

## Actor input object example

```json
{
  "sites": [
    "https://docs.apify.com"
  ],
  "includeUrlsWithoutLastmod": false,
  "maxUrls": 20,
  "maxSitemaps": 200,
  "maxSitemapDepth": 3,
  "tryCommonPaths": true,
  "requestTimeoutSecs": 30
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "https://docs.apify.com"
    ],
    "maxUrls": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("daddyd-prints/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sites": ["https://docs.apify.com"],
    "maxUrls": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("daddyd-prints/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "https://docs.apify.com"
  ],
  "maxUrls": 20
}' |
apify call daddyd-prints/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,daddyd-prints/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aZA3VpjjAw7ehW9Jv/builds/lb5porfDQpBDiRHG0/openapi.json