# Sitemap & robots.txt Extractor - All Site URLs (`antishock/sitemap-robots-url-extractor`) Actor

Extract every URL a website publishes. Reads robots.txt, discovers sitemaps, walks sitemap indexes recursively and returns URL, path, last modified date, change frequency, priority and hreflang. Optional robots.txt rule parsing. For SEO audits, content inventory and migration planning.

- **URL**: https://apify.com/antishock/sitemap-robots-url-extractor.md
- **Developed by:** [Ryan Zinburg](https://apify.com/antishock) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 result exporteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap & robots.txt Extractor - Every URL a Site Publishes

Get the **complete published URL list of any website**. The actor reads `robots.txt`, finds the declared sitemaps, walks sitemap indexes recursively and exports every URL with its last modified date, change frequency and priority.

No API key, no proxy needed.

### What you get per URL

| Field | Example |
|---|---|
| `url` | https://example.com/blog/how-to-migrate |
| `path` | /blog/how-to-migrate |
| `lastModified` | 2026-08-14 |
| `changeFrequency` | weekly |
| `priority` | 0.8 |
| `imageCount` | images declared in the image sitemap extension |
| `alternateLanguages` | hreflang codes declared for the URL |
| `sitemapUrl` | which sitemap the URL came from |

Optionally one extra record with the parsed `robots.txt`: disallow rules, allow rules and crawl delay.

### Input

- **website** - the site to inspect, e.g. `example.com`. robots.txt and the conventional sitemap paths are discovered automatically
- **sitemapUrls** - alternatively, pass sitemap URLs directly if you already know them
- **urlContains** - keep only URLs containing a string, e.g. `/blog/` or `/product/`
- **includeRobotsRules** - also emit the parsed robots.txt rules
- **maxResults** - how many URLs to save, up to 200 000

### Example input

```json
{
  "website": "example.com",
  "urlContains": "/blog/",
  "includeRobotsRules": true,
  "maxResults": 5000
}
```

### Use cases

- **Technical SEO audits** - compare what a site publishes against what is actually indexed
- **Content inventory** - get every page of a site you inherited, with modification dates
- **Site migration planning** - build the redirect map from the old URL list before cutover
- **Competitor content analysis** - see a competitor's entire content footprint and how recently each page changed
- **Crawl seeding** - feed the URL list into a scraper instead of discovering links page by page
- **Freshness monitoring** - `lastModified` shows which sections a competitor actually maintains

### Why start from the sitemap

A sitemap is the site's own declaration of what matters: it is complete by design, already deduplicated, and carries modification dates that no crawl can infer. Starting there is dramatically cheaper than crawling, and it finds pages that no internal link points to.

Sitemap indexes are the normal shape for large sites, so recursive walking is the difference between 50 URLs and 50 000.

### Notes

- Sitemap indexes are followed automatically, up to 300 sitemap files per run.
- If robots.txt declares no sitemap, the conventional locations `sitemap.xml`, `sitemap_index.xml` and `sitemap-index.xml` are tried.
- `lastModified`, `changeFrequency` and `priority` are optional in the sitemap standard, and many sites omit them or fill them in mechanically. Treat them as hints.
- A run that finds nothing fails with an explicit message rather than reporting an empty success, so a missing or blocked sitemap is never silent.

# Actor input Schema

## `website` (type: `string`):

Domain or URL to inspect, e.g. example.com. Sitemaps are discovered automatically.

## `sitemapUrls` (type: `string`):

Optional: sitemap URLs to read directly instead of discovering them.

## `urlContains` (type: `string`):

Keep only URLs containing this string, e.g. /blog/.

## `includeRobotsRules` (type: `boolean`):

Emit one extra record with the parsed robots.txt rules.

## `maxResults` (type: `integer`):

How many URLs to save.

## Actor input object example

```json
{
  "website": "apify.com",
  "includeRobotsRules": false,
  "maxResults": 500
}
```

# Actor output Schema

## `results` (type: `string`):

Scraped records in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "website": "apify.com",
    "maxResults": 500
};

// Run the Actor and wait for it to finish
const run = await client.actor("antishock/sitemap-robots-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "website": "apify.com",
    "maxResults": 500,
}

# Run the Actor and wait for it to finish
run = client.actor("antishock/sitemap-robots-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "website": "apify.com",
  "maxResults": 500
}' |
apify call antishock/sitemap-robots-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,antishock/sitemap-robots-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bMWH9VBTegX2AYBt9/builds/TGT8XELvevjJlSqFy/openapi.json