# Seo Sitemap Url Extractor (`eichan/seo-sitemap-url-extractor`) Actor

- **URL**: https://apify.com/eichan/seo-sitemap-url-extractor.md
- **Developed by:** [Eichan Jolly](https://apify.com/eichan) (community)
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor

**One small SEO job:** read a public XML sitemap and return one dataset item per distinct page URL, including optional `lastmod` and the source sitemap. Sitemap indexes are followed **one level only**, with at most 20 child files. Supports `.xml`, `.xml.gz`, and gzip-compressed HTTP responses.

### Who this is for

SEO specialists, content inventory owners, and small-site operators who need a quick, CSV-ready set of sitemap URLs without a site crawler or search engine scraping.

### Input (copy into the Actor input form)

```json
{
  "sitemapUrl": "https://www.djangoproject.com/sitemap.xml",
  "maxUrls": 5
}
```

The prefilled URL is a public Django project sitemap (checked as publicly reachable on 2026-11-01). `maxUrls` is optional (default 5, min 1, max 500). Only **one sitemap URL** is accepted per run.

### Output (example schema)

Every successfully extracted URL has the following stable fields:

```json
{
  "url": "https://www.djangoproject.com/",
  "lastmod": null,
  "sourceSitemap": "https://www.djangoproject.com/sitemap.xml",
  "status": "ok",
  "error": null
}
```

The example illustrates the field layout; individual URLs and dates depend on the live sitemap. `lastmod` is `null` when the tag is missing. Duplicate URLs are returned only once, in sitemap order. With no page URLs, the run returns an empty dataset.

If a sitemap cannot be read or parsed, one error item is produced instead of a traceback, for example:

```json
{
  "url": null,
  "lastmod": null,
  "sourceSitemap": "https://site.example/sitemap.xml",
  "status": "error",
  "error": "The sitemap returned HTTP 404."
}
```

A bad child within an index also produces an error item while other children may still succeed. `status: error` items are not page URLs; filter on `status: ok` for URL exports.

### Operation and limits

- Uses **only the Python standard library** (`urllib`, `xml.etree`, `gzip`, etc.). No Apify SDK, browser, search engine scraping, login, or installed packages.
- Makes requests only to the **input host**; redirects and child sitemap references to other hosts are rejected. Page URLs merely listed in the XML are **not visited**.
- Reads `/robots.txt` first and skips blocked sitemap paths. A missing robots.txt (404 or 410) permits fetching; 401/403 and ambiguous errors fail closed. Uses a descriptive Actor user agent.
- Up to **20 child sitemaps** from an index, up to **500 unique page URLs**, **4 MB transfer per sitemap**, **12 MB decompressed**, **500 KB robots.txt**, **10 s/request**, at most four same-host redirects. Only one index layer is followed; a nested index produces an error item. Later children are not fetched after the URL cap is reached.
- HTTPS downgraded to HTTP, private network targets, credentials in URLs, nonstandard ports, oversized XML, and XML DTD/entity declarations are refused. Test-only localhost access is gated behind `SEO_TEST_ALLOW_LOOPBACK=1`; **never set this variable in production**.
- Extracts `<loc>` and `<lastmod>` from XML sitemap `<url>` elements. It does not validate destination pages, read image/video tags, or guarantee that URLs are indexable.
- For public operation, HTTP requests require working DNS and internet access.

### Use locally / test

No dependencies to install:

```bash
ACTOR_INPUT_JSON='{"sitemapUrl":"https://www.djangoproject.com/sitemap.xml","maxUrls":5}' python src/main.py
```

JSON item files are written under `storage/datasets/default/`. Alternatively create `storage/key_value_stores/default/INPUT.json` and run `python src/main.py`. Cloud runs read the run input from the default key-value store and append results through the **Apify REST API**, requiring the platform's `APIFY_TOKEN`, `APIFY_DEFAULT_KEY_VALUE_STORE_ID`, and `APIFY_DEFAULT_DATASET_ID` environment variables (no SDK).

For offline, **actual script-mode** checks with local HTTP fixtures:

```bash
python tests/run_smoke.py
python -m unittest discover -s tests -p 'test_*.py' -v
```

`tests/evidence/` contains the actual smoke input, terminal output, and dataset item(s). Smoke uses a local fixture server plus a dedicated test-only localhost override because public DNS was unavailable in the packaging environment. This is **not** a live Apify or live public-network test.

### Store configuration suggestion (NOT applied)

- Categories (max 3): **SEO tools**, **Marketing**, **Developer tools**.
- Pricing: **$0.001 per successfully extracted URL** (`status: ok`); don't charge for error rows. This is an indicative proposal, not an Apify billing setup.
- Full suggestion is in `store_metadata.json` (documentation metadata; it is not a billing command).

No publishing, billing configuration, or platform run was performed.

# Actor input Schema

## `sitemapUrl` (type: `string`):

Direct URL to an XML sitemap or sitemap index (same-host children only).

## `maxUrls` (type: `integer`):

Stop after this many unique page URLs.

## Actor input object example

```json
{
  "sitemapUrl": "https://www.djangoproject.com/sitemap.xml",
  "maxUrls": 5
}
```

# Actor output Schema

## `dataset` (type: `string`):

Dataset containing the Actor's result items

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sitemapUrl": "https://www.djangoproject.com/sitemap.xml"
};

// Run the Actor and wait for it to finish
const run = await client.actor("eichan/seo-sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sitemapUrl": "https://www.djangoproject.com/sitemap.xml" }

# Run the Actor and wait for it to finish
run = client.actor("eichan/seo-sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sitemapUrl": "https://www.djangoproject.com/sitemap.xml"
}' |
apify call eichan/seo-sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,eichan/seo-sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ZWnFyU6uV0fHotLsO/builds/Vnx6MXMh88XAwQSLs/openapi.json