# Sitemap New Urls (`eichan/sitemap-new-urls`) Actor

- **URL**: https://apify.com/eichan/sitemap-new-urls.md
- **Developed by:** [Eichan Jolly](https://apify.com/eichan) (community)
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap New URLs Snapshot

Read **one public XML sitemap URL** and export URL records. Optionally compare every URL to a caller-supplied `knownUrls` baseline, exposing `isNew`. A sitemap index is supported for the first five direct child `urlset` files.

### Who it is for

SEO analysts, website owners, publishing operations, and automation builders tracking which public URLs appear in a sitemap.

### Input example

```json
{
  "sitemapUrl": "https://docs.apify.com/sitemap.xml",
  "maxUrls": 100
}
```

The form prefills **one sitemap URL**. For a meaningful **new URL** comparison, provide previously known URL strings:

```json
{
  "sitemapUrl": "https://docs.apify.com/sitemap.xml",
  "knownUrls": ["https://docs.apify.com/"],
  "maxUrls": 100,
  "onlyNew": true
}
```

| Field | Required | Meaning |
| --- | --- | --- |
| `sitemapUrl` | Yes | One public XML `urlset` or `sitemapindex` URL. |
| `knownUrls` | No | Previously known URL strings (up to 2,000). If **omitted**, `isNew` is **null**. If explicitly `[]`, every URL is new. Matching is exact and case-sensitive. |
| `maxUrls` | No | Maximum unique entries in document order, 1–200, default 100. |
| `onlyNew` | No | Write only records with `isNew: true`; requires the `knownUrls` field, even if empty. Default `false`. |

### Output example

One Dataset item per qualifying URL (illustrative values):

```json
{
  "url": "https://docs.apify.com/actors/",
  "lastModified": "2026-10-31",
  "isNew": true,
  "sourceSitemap": "https://docs.apify.com/sitemap.xml",
  "checkedAt": "2026-11-01T09:00:00Z"
}
```

- `url`: absolute public HTTP(S) page URL from the XML `<loc>` element.
- `lastModified`: unmodified `<lastmod>` string if present (maximum 100 characters), otherwise `null`. Sitemap timestamps are publisher-supplied, **not** independently verified.
- `isNew`: `true` / `false` when a `knownUrls` baseline is explicitly present; otherwise `null` (unknown). This is **not** automatic cross-run new URL detection.
- `sourceSitemap`: XML file that contained this `<url>` record (useful when reading a sitemap index).
- `checkedAt`: UTC ISO 8601 timestamp assigned to all records from the run.

### Scheduled use case

Schedule daily to export a snapshot of your sitemap. Preserve previous run URLs in your workflow or external storage, then supply them as `knownUrls` to the next run and set `onlyNew: true` if you only need additions. **The Actor does not read previous runs or persist a baseline by itself**, and `knownUrls` is capped at 2,000, so it is intended for small sitemaps or bounded comparisons.

### Limits and behavior

- XML `urlset` sitemaps and one-level `sitemapindex` files only. A sitemap index reads **up to five** child sitemap files in index order. Nested indexes are rejected with a clear error.
- Source download limit **2 MB per sitemap**, max 200 output candidates, 20-second HTTP timeout per request. Processing is sequential; server rate limits and connectivity may affect results.
- HTML landing pages, HTML error pages, RSS feeds, unrelated XML, malformed XML, and DTD/entity declarations are explicitly rejected with plain-language messages. In particular an HTML page is **not** silently treated as an empty sitemap.
- The Actor does not fetch listed pages, crawl websites, render JavaScript, follow canonical tags, or prove that listed URLs are live. Output URL strings are checked for HTTP(S) syntax and obviously private literal IPs but not DNS-resolved.
- Only public source HTTP(S) web URLs on standard ports; redirects to private networks are blocked. DNS can change between validation and connection; do not treat this as a hardened network security boundary.
- Deduplication is by exact URL string within one run; content is emitted in XML order. For an empty sitemap there are **zero** rows. `maxUrls` may truncate the snapshot; an index with more than five child sitemaps is explicitly noted in terminal output.
- `BadStatusLine` and other malformed HTTP replies are reported as plain download errors, never tracebacks.
- No login, external API keys, persistent state, alerting, third-party scraping services, or enabled billing events.

### Run locally and verify

Create `storage/key_value_stores/default/INPUT.json` with the input JSON, or set `ACTOR_INPUT_JSON`. From the Actor root run:

```bash
python src/main.py
python -m unittest discover -s tests -p 'test_*.py' -v
python tests/smoke_run.py
```

Rows are saved in `storage/datasets/default/000000001.json`, etc. `tests/smoke_run.py` starts a local HTTP fixture and executes **`python src/main.py` in a subprocess**, then prints actual terminal output and Dataset items. It sets `EICHAN_TEST_LOOPBACK=1` only for isolated non-cloud testing, not production; cloud and ordinary local runs reject loopback URLs.

### Deploy to Apify

Upload this complete folder to a new Actor, build the included Dockerfile, and run it. The Apify runtime supplies the input key-value-store and default Dataset identifiers plus a run token; the code uses only `urllib` to access its own run storage. The runtime token is **not** a user-managed external API key.

### Store categories and suggested pricing

- **Category:** Automation (primary)
- **Secondary category:** Developer tools
- **Suggested pricing model:** Pay per event — **per-run** `actor-start` event, provisional **$0.005/run**, for one bounded sitemap query. Review usage and performance before setting a real price.
- **Actual billing:** Not configured or charged; no billing events emitted. Pricing and category selection would be set manually only if this Actor is later published.

# Actor input Schema

## `sitemapUrl` (type: `string`):

Direct public XML urlset or sitemapindex URL. One is prefilled.

## `knownUrls` (type: `array`):

Compare these exact strings with sitemap URLs. Omit for isNew=null; \[] means all URLs are new.

## `maxUrls` (type: `integer`):

At most this many unique URLs, in sitemap order.

## `onlyNew` (type: `boolean`):

When true, knownUrls must be supplied (an empty array is allowed).

## Actor input object example

```json
{
  "sitemapUrl": "https://docs.apify.com/sitemap.xml",
  "maxUrls": 100,
  "onlyNew": false
}
```

# Actor output Schema

## `dataset` (type: `string`):

Dataset containing the Actor's result items

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sitemapUrl": "https://docs.apify.com/sitemap.xml"
};

// Run the Actor and wait for it to finish
const run = await client.actor("eichan/sitemap-new-urls").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sitemapUrl": "https://docs.apify.com/sitemap.xml" }

# Run the Actor and wait for it to finish
run = client.actor("eichan/sitemap-new-urls").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sitemapUrl": "https://docs.apify.com/sitemap.xml"
}' |
apify call eichan/sitemap-new-urls --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,eichan/sitemap-new-urls"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ujwPJfEcxT4YuqpbA/builds/Hh2ceh6n4IinLjzh3/openapi.json
