# Sitemap New Pages Watch: Added & Removed URLs (`practical_ai_lab_jp/pal-website-sitemap-new-pages-watch`) Actor

Compare a site's robots-permitted public sitemap with the previous run and report added, removed and updated URLs as JSON or CSV.

- **URL**: https://apify.com/practical_ai_lab_jp/pal-website-sitemap-new-pages-watch.md
- **Developed by:** [Practical AI Lab](https://apify.com/practical_ai_lab_jp) (community)
- **Categories:** E-commerce, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 sitemap change reports

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap New Pages Watch: Added & Removed URLs

Give this Actor a public HTTPS site root. It reads the site's `robots.txt`, fetches only the sitemap files that robots.txt permits, stores a per-domain baseline of listed URLs, and on every later run reports which URLs were **added**, **removed**, or had their sitemap **`lastmod` changed**. Results are available as a dataset (JSON) and as a CSV file.

It does not crawl pages. It only reads sitemap XML, so runs are small and fast.

### Typical uses

- SEO and content teams tracking when a competitor or partner publishes new pages.
- Agencies watching client sites for pages that disappear after a redeploy.
- Directory, catalog and documentation owners who want a scheduled "what changed" list.

### Quick start (copy this input)

```json
{
    "startUrl": "https://www.python.org/",
    "maxUrls": 5000,
    "reportUpdated": true
}
```

Replace `startUrl` with a site you are allowed to monitor. Whether a given site publishes a usable sitemap is up to that site; the Actor reports `NO_SITEMAP` when none is found.

1. **First run** creates the baseline and outputs one `BASELINE_CREATED` row. No changes are reported.
2. **Later runs** compare against the baseline and output one row per change.

### Input

| Field | Default | Meaning |
|---|---|---|
| `startUrl` (required) | | Public HTTPS site root, no path or query. |
| `includePathPrefixes` | none | Track only URLs under these paths (e.g. `/blog/`). Each distinct set has its own baseline. |
| `reportUpdated` | `true` | Also report URLs whose `lastmod` changed. |
| `maxUrls` | `5000` | Hard cap (max 50000). Exceeding it ends the run as `INCOMPLETE_SCAN`. |
| `maxSitemapFiles` | `25` | Cap on sitemap files read, including those named in sitemap indexes (max 100). |
| `maxChangesReported` | `2000` | Cap on dataset rows; totals are still in `SUMMARY`. |
| `resetBaseline` | `false` | Replace the stored baseline with this run's snapshot. |
| `baselineStoreName` | `pal-sitemap-watch-baselines` | Named key-value store holding baselines. |

### Output

Dataset rows (`recordType: "CHANGE"`) contain `changeType` (`ADDED`, `REMOVED`, `UPDATED`), `url`, `lastmod`, `previousLastmod`, `sourceSitemap`, `startUrl`, `domain`, `detectedAt`, `previousCheckedAt` and `baselineCreatedAt`. Runs without changes produce one `RUN_STATUS` row with a `status` and `message`.

Key-value store records of the run: `SUMMARY` (counts, skip counters, status) and `CHANGES_CSV` (the same rows as CSV).

#### Run statuses

| Status | Meaning | Run result | Billable event |
|---|---|---|---|
| `BASELINE_CREATED` | First run for this site/filter | Succeeded | No |
| `NO_CHANGES` | Nothing changed | Succeeded | No |
| `CHANGES_DETECTED` | Added, removed or updated URLs found | Succeeded | Yes (one `sitemap-change-report` event) |
| `ROBOTS_BLOCKED` | robots.txt forbids access to the sitemaps | Succeeded | No |
| `NO_SITEMAP` | No sitemap found, or it contains no usable URLs | Succeeded | No |
| `INCOMPLETE_SCAN` | A limit was hit or a listed file failed; nothing saved | Failed | No |
| `SITEMAP_ERROR` / `FETCH_ERROR` | Broken XML, HTTP errors, unsafe redirect | Failed | No |
| `INVALID_INPUT` | Bad or unsafe input | Failed | No |

A failed or incomplete run never overwrites the baseline, so the next good run still compares against the last complete snapshot.

The code requests one `sitemap-change-report` charge event only when changes are reported. No price is configured in this project; pricing is set separately by the publisher.

### Scheduling

Create a Task with your input and add an Apify Schedule (for example daily). Keep the default `baselineStoreName`: baselines live in a named key-value store, which persists across runs, unlike the per-run default store. Use a different name per customer or project if you want separate baselines. Because baselines are keyed by hostname, two tasks watching the same host with the same filters and the same store share one baseline.

### Safety behavior

- HTTPS only, default port only, no credentials in URLs, no IP-address hosts, no localhost or internal-style hostnames.
- The hostname must resolve only to public addresses (checked before every request).
- Redirects are followed at most 3 times and only to the same HTTPS host; anything else is refused.
- `robots.txt` is read first. Sitemap files and listed URLs that it disallows for this Actor are skipped. A 401/403 on robots.txt is treated as a block; an unreachable robots.txt is an error.
- Response sizes are bounded (robots 512 KB, sitemap download 10 MB, decompressed 20 MB). Sitemap DTD/entity declarations are rejected.
- URLs that look like they contain an email address are dropped. The Actor never visits listed pages, logs in, uses proxies, or sends notifications.

### Limitations and caveats

- **A sitemap is not the whole site.** Pages missing from the sitemap are invisible, and a URL leaving the sitemap does not necessarily mean the page was deleted.
- **`lastmod` quality varies.** Many sites omit it or set it to the build time, which can create noisy `UPDATED` rows. Turn off `reportUpdated` for such sites.
- **Caps are strict.** Sites with more than `maxUrls` URLs (max 50,000) cannot be monitored in a single baseline; use `includePathPrefixes` to split them.
- Only URLs on exactly the monitored hostname are tracked; sitemaps on other hosts or subdomains are ignored.
- Robots handling uses Python's standard parser, which follows first-match rules and may differ from other crawlers on complex wildcard rules.
- Address checks happen before each request but the HTTP library resolves the name again when connecting, so DNS rebinding cannot be ruled out entirely.
- Sitemap index nesting is limited to 3 levels, and gzip sitemaps are supported.
- Time-of-run is when the Actor observed the change, not when the site published it.
- The offline test suite uses mocked HTTP only. Behavior on real sites has not been benchmarked here, so no performance, accuracy or demand figures are claimed.

### Development

```
python -m unittest discover -s tests
```

Tests need no network and no Apify runtime.

# Actor input Schema

## `startUrl` (type: `string`):

Public HTTPS site root, for example https://www.python.org/ . Must be a root URL without path or query. robots.txt is read first and obeyed.

## `includePathPrefixes` (type: `array`):

Limit tracking to URLs whose path starts with one of these prefixes, for example /blog/ . Each distinct set of prefixes keeps its own baseline.

## `reportUpdated` (type: `boolean`):

Also report URLs whose sitemap lastmod value changed since the previous run.

## `maxUrls` (type: `integer`):

If the sitemaps hold more URLs than this, the run stops as INCOMPLETE_SCAN and nothing is saved or billed. Raise it for larger sites.

## `maxSitemapFiles` (type: `integer`):

Upper bound on sitemap files fetched, including those listed in sitemap indexes.

## `maxChangesReported` (type: `integer`):

Caps dataset rows after a large site redeploy. Totals are still given in the SUMMARY record.

## `resetBaseline` (type: `boolean`):

Discard the stored baseline for this site and create a fresh one from this run.

## `baselineStoreName` (type: `string`):

Named key-value store that keeps baselines between runs (3-63 lowercase letters, digits, hyphens). Leave the default so scheduled runs share one baseline. An empty value uses the per-run default store, which does not persist between runs.

## Actor input object example

```json
{
  "startUrl": "https://www.python.org/",
  "reportUpdated": true,
  "maxUrls": 5000,
  "maxSitemapFiles": 25,
  "maxChangesReported": 2000,
  "resetBaseline": false,
  "baselineStoreName": "pal-sitemap-watch-baselines"
}
```

# Actor output Schema

## `results` (type: `string`):

Download rows from the latest run in JSON/CSV.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://www.python.org/"
};

// Run the Actor and wait for it to finish
const run = await client.actor("practical_ai_lab_jp/pal-website-sitemap-new-pages-watch").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrl": "https://www.python.org/" }

# Run the Actor and wait for it to finish
run = client.actor("practical_ai_lab_jp/pal-website-sitemap-new-pages-watch").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://www.python.org/"
}' |
apify call practical_ai_lab_jp/pal-website-sitemap-new-pages-watch --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,practical_ai_lab_jp/pal-website-sitemap-new-pages-watch"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Z6cmgIsZYYehfpR0g/builds/TyeM66LvQoy2dcP9D/openapi.json