# Sitemap URLs — unique listed URLs, dates and robots rules (`hermeskd/sitemap-urls`) Actor

Give it domains or sitemap URLs; it finds the sitemaps (robots.txt, `/sitemap.xml`, `/sitemap_index.xml`), follows sitemap indexes, reads gzip, text and RSS/Atom sitemaps, and returns every listed URL once with lastmod, changefreq and priority, plus a free summary of the site's robots.txt rules.

- **URL**: https://apify.com/hermeskd/sitemap-urls.md
- **Developed by:** [Logan Parker](https://apify.com/hermeskd) (community)
- **Categories:** SEO tools, Developer tools, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.24 / 1,000 url listeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Give it domains or sitemap URLs; it finds the sitemaps (robots.txt, `/sitemap.xml`, `/sitemap_index.xml`), follows sitemap indexes, reads gzip, text and RSS/Atom sitemaps, and returns every listed URL once with lastmod, changefreq and priority, plus a free summary of the site's robots.txt rules.

Development validation produced actual successful public-input records using the production Dockerfile and SDK entrypoint on GitHub-hosted Linux. The smoke input, all three tasks and every defined benchmark passed their exact expectations. The independent review completed on 2026-10-19; its findings are tracked in the committed fix report. Platform costs, settlement, publication and daily tests remain unverified.

### Call it from an agent

Through Apify's MCP server (https://mcp.apify.com) use `call-actor`:

```json
{
  "actor": "hermeskd/sitemap-urls",
  "input": {
    "urls": [
      "https://docs.apify.com/"
    ],
    "maxUrls": 50
  },
  "callOptions": {
    "maxTotalChargeUsd": 0.05,
    "timeout": 1800
  },
  "waitSecs": 30
}
```

Then call `get-dataset-items` with the returned defaultDatasetId. Dedicated tool URL: https://mcp.apify.com?tools=hermeskd/sitemap-urls

### Input

| Field | Type | Meaning |
|---|---|---|
| `urls` | array | Domains or absolute sitemap URLs; bare domains use HTTPS. |
| `maxUrls` | integer | Maximum unique matching URL records across the run. |
| `filter` | string | Optional regex; at most 512 characters and a 20 ms match timeout. |
| `includeRobots` | boolean | One free all-group robots summary per host; independent of HermesBot policy enforcement. |

### Output

| Field | Meaning |
|---|---|
| `type` | Listed URL or free robots summary. |
| `url` | Listed URL, robots URL, or original failed input. |
| `ok` | True for returned results; errors never charged. |
| `lastmod` | Publisher-provided modification/publication value. |
| `changefreq` | Publisher change frequency. |
| `priority` | Publisher priority between zero and one. |
| `sitemap` | Source sitemap final URL. |
| `robots` | Free all-group summary, separate from the HermesBot access decision. |
| `error` | Failure, always uncharged. |

Actual production-container record from 2026-10-18T20:27:13.734929+00:00 (code `f063187693578dd7551fa2bec552281e2f0f3e33`, smoke input, SDK local storage). The saved record is unchanged; local event logs do not prove Apify settlement.

```json
{
  "type": "url",
  "url": "https://docs.apify.com/api",
  "ok": true,
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemap": "https://docs.apify.com/sitemap_base.xml"
}
```

### Events and prices

| Event | Title | Charged when | FREE / BRONZE | SILVER | GOLD / PLATINUM / DIAMOND |
|---|---|---|---|---|---|
| `url-listed` | URL listed | One unique matching URL written from a sitemap. | $0.0003 | $0.00027 | $0.00024 |

Platform usage is not passed through. Apify's automatic actor-start event remains $0.00005 per started GB (one GB or less counts once). The synthetic dataset-item event is absent. Charge limits stop further paying work; unprocessed inputs receive uncharged errors. Completed readable pending events are settled on abort.

Worked cost example: ten successful primary events cost $0.003 at list, plus actor start. Failures are not included in that count.

### Limits and run time

1–20 domains or sitemap URLs; maxUrls 1–200,000 across the run, after filtering and deduplication. XML/index, gzip, UTF-8 text, RSS and Atom are supported. Every sitemap is bounded to 50 MiB expanded and 50,000 entries; traversal to 10,000 documents and depth 50, with cycles deduplicated. Regex filters are at most 512 characters with a 20 ms match timeout. Robots summaries are free and do not mask failed sitemap outcomes. Protocol extensions do not replace protocol namespace fields.

Default run timeout is 1800 seconds. Typical cold platform time, costs and the smallest passing memory have not been measured for this new implementation. Memory probes are evidence to collect later; configured memories are specification defaults, not measured claims.

### Errors

| Code | Meaning |
|---|---|
| `no_sitemap` | No readable sitemap URL result was found for the input. |
| `download_failed` | The request or connection failed within its bounded deadline. |
| `http_error` | The source answered an unsuccessful HTTP status. |
| `robots_disallowed` | robots.txt forbids fetching the URL; no request was made to it. |
| `blocked_host` | The host is on the deny-list or resolves to a private or local address; no request was made. |
| `too_large` | A documented response, expansion or traversal cap was exceeded. |
| `source_schema_changed` | A source or tool response no longer has the verified protocol shape. |
| `timeout` | The input did not finish within the run's time budget. |
| `invalid_input` | The run input passed the input schema but cannot be used; the message names the field. |
| `internal_error` | Unexpected failure; the message carries the error class. |
| `charge_limit_reached` | The run reached its maximum total charge; this input was not processed. |

No failure record is charged. A run fails only when all input outcomes are Actor-side failures; caller errors and a mixture of caller/Actor failures do not fail the run.

### Sources and terms

Only process content you have the right to process. URL fetches obey HermesBot robots-group selection, the shared deny-list, public-address checks on connections and redirects, honest identity, deadlines and rate limits. No caller credentials, cookies, CAPTCHA solving, residential proxies or evasion. Public robots files and sitemap documents are fetched; listed pages are not downloaded. The bounded parser supports XML urlsets/indexes, gzip, UTF-8 text, RSS and Atom. DTDs and entities are refused. The public documentation sitemap source was checked directly; its Apache-2.0 source repository and observed document counts are recorded with the manifests. The 2026-10-18 GitHub-hosted run completed actual production-container traversal, with raw records and provenance shown above. Cloud resolver access remains unavailable; this does not negate the separate hosted evidence.

### Not affiliated

This Actor is not affiliated with Google, Apify, IANA, Verisign or other named protocol/source providers. Third-party names describe compatibility and source material.

### Related Hermes Actors

website-screenshot, lighthouse-audit, seo-page-audit, domain-availability, dns-and-email-auth, broken-links and sitemap-urls.

### Changelog

2026-10-17 (historical): initial implementation and fixture contracts; packaging and public production validation were incomplete on that date.

2026-10-18: all six production images, public smoke inputs, three tasks per Actor and defined benchmarks passed on GitHub-hosted Linux; exact code, records and image provenance are retained with the examples. These local SDK runs do not prove platform costs, settlement, publication, minimum memory, daily tests or payouts.

2026-10-19: independent-review repairs cover DNS privacy, malformed publisher URLs, empty sitemap failures, responsive assets and schema prefill, as applicable to this Actor. See the committed unit 4 fix report for measured verification and remaining platform NO-COVERAGE.

# Actor input Schema

## `urls` (type: `array`):

Domains or absolute sitemap URLs; bare domains use HTTPS.

## `maxUrls` (type: `integer`):

Maximum unique matching URL records across the run.

## `filter` (type: `string`):

Optional regex; at most 512 characters and a 20 ms match timeout.

## `includeRobots` (type: `boolean`):

One free all-group robots summary per host; independent of HermesBot policy enforcement.

## Actor input object example

```json
{
  "urls": [
    "https://docs.apify.com/"
  ],
  "maxUrls": 10000,
  "includeRobots": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.apify.com/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("hermeskd/sitemap-urls").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://docs.apify.com/"] }

# Run the Actor and wait for it to finish
run = client.actor("hermeskd/sitemap-urls").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.apify.com/"
  ]
}' |
apify call hermeskd/sitemap-urls --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hermeskd/sitemap-urls"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/uw4hJUyyJ4ISGi3Ue/builds/bWxx1iPOpwQURc6gw/openapi.json