# Sitemap Extractor & Website URL Finder (no sitemap needed) (`kilndata/sitemap-url-extractor`) Actor

Get every URL of a website: reads robots.txt, nested sitemap indexes, .xml.gz and text sitemaps, then RSS/Atom feeds, and falls back to a light crawl when there is no sitemap. URL, lastmod, changefreq, priority, source. Filters by URL pattern and date. Failed sites are free.

- **URL**: https://apify.com/kilndata/sitemap-url-extractor.md
- **Developed by:** [Datakiln](https://apify.com/kilndata) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor and Website URL Finder (works without a sitemap)

Give it a domain, a homepage or a sitemap link, and it returns every page URL it can find, one row per unique URL. For each URL you get `lastmod`, `changefreq` and `priority` when the sitemap has them, plus where the URL came from.

It starts from `robots.txt`, follows nested sitemap indexes, reads `.xml`, `.xml.gz` and plain-text sitemaps, and falls back to RSS/Atom feeds. If a site has no sitemap and no feed, it follows internal links for a small number of pages. It opens no browser, needs no login and reads no personal data.

### Who uses it

| Use | How |
|---|---|
| SEO and content audits | Pull the full URL list with `lastmod`, filter to `/blog/`, compare against your analytics or Search Console |
| Site migrations | Export all URLs of the old site before you redirect them |
| Competitor research | List every product or category URL of a store you want to study |
| AI and RAG pipelines | Get the list of pages to feed into a scraper, embedder or crawler |
| Change monitoring | Schedule it with a "Modified after: 7 days" filter to see what was updated this week |
| Broken-link work | Use the URL list as the input for a link checker |

### How it works

For each website, in this order:

1. Reads `robots.txt` and takes every `Sitemap:` line. It also tries `http://` and the `www.` variant if the first host does not answer.
2. Downloads each sitemap. Indexes are followed down to the depth you set (default 5), and loops are detected. Gzipped files are unpacked. Text sitemaps (one URL per line) work too.
3. If `robots.txt` lists nothing usable, it tries the usual places: `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap.txt` and a few more. It also checks the homepage for `<link rel="sitemap">`.
4. If there is still nothing, it reads RSS/Atom feeds (the ones the homepage points to, or common paths like `/feed`). This is limited by the `feeds` setting.
5. If there is no feed either, it crawls. It opens up to `maxCrawlPages` pages (default 50) of the same domain and returns every internal link it finds on them. 50 pages usually give a few hundred URLs, not the whole site.

Each URL is returned once per run, even if it shows up in two sitemaps or two of your websites. A start URL with a path, such as `https://example.com/blog/`, returns only URLs under that path. Turn that off with `limitToStartPath` if you want the whole site.

Websites run in parallel (10 by default). A site that answers every request with 403, 429 or a bot-challenge page is dropped early instead of being retried for minutes, and it is not charged.

### Input

```json
{
    "startUrls": ["nodejs.org", "https://www.sitemaps.org/sitemap.xml"],
    "includeUrlPatterns": ["/blog/"],
    "lastmodAfter": "90 days",
    "maxUrlsPerSite": 1000
}
```

| Field | Type | Default | What it does |
|---|---|---|---|
| `startUrls` | list | required | Domains, homepages or direct sitemap/feed URLs. Duplicates are removed |
| `includeUrlPatterns` | list | empty | Keep only URLs matching at least one pattern. Regex (case-insensitive) or simple wildcards, e.g. `/products/.*`, `\.pdf$` |
| `excludeUrlPatterns` | list | empty | Drop URLs matching any pattern, e.g. `/tag/`, `\?` |
| `lastmodAfter` | text | empty | Keep URLs modified on or after this date. `2025-01-31` or `30 days`, `2 weeks`, `6 months` |
| `lastmodBefore` | text | empty | Keep URLs modified on or before this date. Same formats |
| `includeUrlsWithoutLastmod` | boolean | false | With a date filter on, also keep URLs that have no `lastmod` |
| `maxUrlsPerSite` | integer | 0 | Stop a site after this many URLs (after filters). 0 means no limit |
| `limitToStartPath` | boolean | true | Only URLs under the path of the start URL |
| `feeds` | `fallback` / `always` / `never` | `fallback` | `fallback` reads feeds only when no sitemap URLs were found. `always` adds feed URLs too |
| `crawlFallback` | boolean | true | Follow internal links when there is no sitemap and no feed |
| `maxCrawlPages` | integer | 50 | Pages opened by the crawl fallback, per site. Max 5000 |
| `includeExtras` | boolean | false | Add `images` and hreflang `alternates` from the sitemap |
| `respectRobotsTxt` | boolean | true | Skip disallowed paths when probing and crawling. Sitemaps listed in robots.txt are always read |
| `maxConcurrency` | integer | 10 | Websites processed at once |
| `sitemapConcurrency` | integer | 5 | Sitemap files downloaded at once from one site |
| `maxSitemapDepth` | integer | 5 | Levels of nested indexes to follow |
| `requestTimeoutSecs` | integer | 60 | Time allowed to download one file |
| `siteTimeoutSecs` | integer | 900 | Time limit per site. When reached, the site keeps what it already collected |
| `maxRetries` | integer | 2 | Retries on network errors, 429 and 5xx. Each retry switches HTTP version or client |
| `ignoreSslErrors` | boolean | true | Read sitemaps from sites with expired or broken certificates |
| `proxyConfiguration` | object | no proxy | Only needed when a site blocks the requests |

### Output

One row per URL. This is a real row from a run on allbirds.com with `includeExtras` on, shortened:

```json
{
    "inputUrl": "https://www.allbirds.com",
    "site": "allbirds.com",
    "url": "https://www.allbirds.com/products/mens-wool-runners",
    "lastmod": "2026-10-05T03:55:27.000Z",
    "changefreq": "daily",
    "priority": null,
    "source": "sitemap",
    "sourceUrl": "https://www.allbirds.com/sitemap_products_1.xml?from=1878193471557&to=7369944137808",
    "status": "ok",
    "error": null,
    "extractedAt": "2026-10-05T03:55:26.049Z",
    "images": ["https://cdn.shopify.com/s/files/1/1104/4168/files/Allbirds_WL_RN_SF_PDP_Natural_Grey_LAT.png?v=1751143404"],
    "alternates": []
}
```

`source` is `sitemap`, `rss`, `atom` or `crawl`. URLs from a crawl have no `lastmod`, `changefreq` or `priority`. `lastmod` is converted to ISO 8601; if the sitemap holds a value that cannot be parsed, the raw text is kept. Fragments (`#...`) are removed.

A website that returns no URLs gets one free status row instead, so you always know why:

| status | Meaning |
|---|---|
| `ok` | A URL row. This is the only status that is charged |
| `failed` | The site could not be reached, the sitemap could not be read, or the site blocks automated requests |
| `no-urls` | No sitemap, no feed and no crawlable links |
| `no-match` | URLs were found, but none passed your filters or the start path |
| `skipped` | robots.txt disallows the site and lists no sitemap |
| `invalid-input` | The line was not a valid domain or URL |
| `not-processed` | The run hit the cost limit or the memory limit before this site |

The dataset has two views: the URL table, and "Websites without URLs". A run summary (sitemaps found per site, sitemap errors, methods used, duration) is saved under `SUMMARY` in the key-value store.

### Pricing

You pay per extracted URL (event `url-extracted`), plus a tiny start fee per run.

| Apify plan | Price per 1,000 URLs |
|---|---|
| Free | $0.80 |
| Starter / Bronze | $0.40 |
| Scale / Silver | $0.30 |
| Business / Gold | $0.20 |

Actor Start: $0.00005 per run.

What counts as a charged URL: a row with `status: ok` that was saved to the dataset. Not charged: status rows, invalid input, sites that fail, duplicates, URLs removed by your filters, and URLs cut by `maxUrlsPerSite`. Charges are sent in batches of 500 rows, so the bill follows the rows in your dataset.

**Your spending cap works.** The Actor reads your maximum cost per run and stops cleanly when the next URL would go over it. It keeps everything saved so far and marks the unvisited sites as `not-processed`. The summary says `stoppedByCostLimit: true`.

Example costs, on the Free plan and on Business:

| URLs extracted | Free | Business |
|---|---|---|
| 10,000 | $8.00 | $2.00 |
| 1,000,000 | $800 | $200 |

Add $0.00005 per run to each. These are the Actor's own prices. Apify platform usage (compute) is billed separately according to your plan, and sitemap reading is light, so it is usually small next to the per-URL price.

Tip for a first run: set `maxUrlsPerSite` to 1000 and a low maximum cost per run, check the rows, then raise both.

### Limits

- **Very large sites.** Each sitemap file is downloaded in one piece, with a cap of 100 MB per file. Sites with millions of URLs have many files, which is fine, but each file takes memory while it is parsed. A million-URL run needs several GB of RAM, so raise the run memory (the Actor allows up to 4 GB). If memory reaches about 90%, the Actor stops on purpose, keeps the saved URLs and sets `stoppedByMemory`. Splitting the job with `includeUrlPatterns` or by sub-sitemap URL is safer than one giant run.
- **Time.** Each site has a time limit (900 seconds by default). A big site that hits it returns partial results and says so in the summary notes. Raise `siteTimeoutSecs` if you need the rest.
- **Sites that block bots.** Some sites answer 403, 429 or a challenge page. The Actor gives up on them after three bad answers in a row and reports `failed`. A proxy can help, but there is no promise. It does not solve captchas or log in.
- **Incomplete sitemaps.** A sitemap lists what the site owner chose to list. Pages missing from it are not returned. Many sitemaps are out of date, and `lastmod` is often missing or set to the build date for every page. If you need to know whether the sitemap is complete, compare against a crawl. The crawl fallback only runs when no sitemap was found, so it will not fill gaps in a sitemap that exists.
- **The crawl fallback is light on purpose.** It reads the HTML the server sends, not JavaScript-rendered links, and stops after `maxCrawlPages` pages. It is for sites with nothing else to read, not a full crawler.
- **Duplicates are counted once per run**, across all your start URLs. The same URL listed by two of your websites appears once.
- **A URL is a listing, not a check.** The Actor does not request each page. A URL in a sitemap can return 404 today.
- **Order.** Rows are not sorted. Sort in the dataset or after export.

### Responsible use

The Actor reads public files: robots.txt, sitemaps, feeds and, in the fallback, public HTML pages. It does not log in, does not use cookies from a session and does not store personal data, only URLs and their metadata. With `respectRobotsTxt` on (the default), it skips paths that robots.txt disallows when it probes common sitemap locations and when it crawls. Sitemaps that robots.txt itself lists are always read, because the site publishes them for crawlers. Requests per site are limited by `sitemapConcurrency` (5 by default); lower it for small sites. You are responsible for the terms of the sites you point it at and for what you do with the URLs.

### FAQ

**Does it work on a site with no sitemap?**
Yes, in a limited way. It tries feeds, then follows internal links for up to `maxCrawlPages` pages. The result is the part of the site reachable from those pages, not a guaranteed full list.

**Why did I get fewer URLs than the site has?**
Usually the sitemap is incomplete, or you hit `maxUrlsPerSite`, the site time limit, or your cost cap. Check the `SUMMARY` record for notes and `stoppedByCostLimit`.

**What do I get for a site that fails?**
One row with `status` and a plain `error` text, at no cost.

**Can I give it a sitemap URL directly?**
Yes. A link ending in `.xml`, `.xml.gz` or a feed path is read as is, without looking for robots.txt or other sitemaps.

**Can I get only the blog, or only pages changed recently?**
Yes. Use a start URL with a path (`https://example.com/blog/`) or `includeUrlPatterns`, and `lastmodAfter` such as `30 days`. Many sitemaps have no `lastmod`; set `includeUrlsWithoutLastmod` to keep those URLs.

**Do I need a proxy?**
Almost never. Sitemaps are public. Add one only when a site blocks the requests.

**Can I call it from the API or an AI agent?**
Yes. Run it through the Apify API, a webhook, a schedule or the Apify MCP server, and read the dataset items.

### Feedback

Found a site that does not work, or want a feature? Open an issue in the Issues tab of this Actor and include the start URL and the run link. Bug reports with a run link get fixed fastest.

Not affiliated with any of the websites you extract from.

# Actor input Schema

## `startUrls` (type: `array`):

One per line. A domain ("example.com") or homepage triggers automatic discovery. A direct sitemap or feed URL ("https://example.com/sitemap.xml", ".xml.gz", "/feed") is read as is. Duplicates are removed.

## `includeUrlPatterns` (type: `array`):

Optional. Keep only URLs that match at least one pattern. Regular expressions (case-insensitive) or simple wildcards: "/blog/", "/products/.*", "https://example.com/docs/*", ".pdf$".

## `excludeUrlPatterns` (type: `array`):

Optional. Drop URLs that match any of these patterns, e.g. "/tag/", "/page/\d+", "?".

## `lastmodAfter` (type: `string`):

Optional. Keep only URLs whose lastmod (or feed date) is on/after this date. An absolute date ("2025-01-31") or a relative one ("30 days", "2 weeks", "6 months").

## `lastmodBefore` (type: `string`):

Optional. Keep only URLs whose lastmod is on/before this date. Same formats as above ("1 year" = older than one year).

## `includeUrlsWithoutLastmod` (type: `boolean`):

Only matters when a date filter is set. Many sitemaps have no lastmod; turn this on to keep those URLs anyway.

## `maxUrlsPerSite` (type: `integer`):

Stop a website after this many URLs (after filters). 0 = no limit.

## `limitToStartPath` (type: `boolean`):

When a start URL has a path (e.g. "https://example.com/blog/"), return only URLs under that path. Domains and homepages always return the whole website.

## `feeds` (type: `string`):

"fallback" reads feeds only when no sitemap URL was found. "always" also adds feed URLs (useful for the freshest posts). "never" skips feeds.

## `crawlFallback` (type: `boolean`):

If a website has no usable sitemap and no feed, follow its internal links (same domain, robots.txt respected) to list its URLs.

## `maxCrawlPages` (type: `integer`):

How many HTML pages the crawl fallback downloads per website. Every internal link found on those pages is returned, so 50 pages usually yield hundreds of URLs.

## `includeExtras` (type: `boolean`):

Add "images" (image:image locations) and "alternates" (xhtml:link hreflang versions) from the sitemap to each URL.

## `respectRobotsTxt` (type: `boolean`):

Probing common sitemap locations and the crawl fallback skip paths disallowed by robots.txt. Sitemaps listed in robots.txt are always read.

## `maxConcurrency` (type: `integer`):

How many websites are processed at the same time.

## `sitemapConcurrency` (type: `integer`):

How many sitemap files of the same website are downloaded at the same time. Keep it low to be polite.

## `maxSitemapDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow (index → index → sitemap). Loops are detected automatically.

## `requestTimeoutSecs` (type: `integer`):

Time allowed to download one file (large sitemaps can take a while).

## `siteTimeoutSecs` (type: `integer`):

When reached, the website stops and keeps the URLs collected so far, so one huge site never blocks the run.

## `maxRetries` (type: `integer`):

Retries for network errors, 429 and 5xx. Each retry changes strategy (HTTP/2 → HTTP/1.1 → different client).

## `ignoreSslErrors` (type: `boolean`):

Read public sitemaps even from websites with expired or misconfigured certificates.

## `proxyConfiguration` (type: `object`):

Optional. Sitemaps are public and almost never need a proxy. Use one only if a website blocks the requests.

## Actor input object example

```json
{
  "startUrls": [
    "nodejs.org",
    "https://www.sitemaps.org/sitemap.xml",
    "news.ycombinator.com"
  ],
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "lastmodAfter": "",
  "lastmodBefore": "",
  "includeUrlsWithoutLastmod": false,
  "maxUrlsPerSite": 1000,
  "limitToStartPath": true,
  "feeds": "fallback",
  "crawlFallback": true,
  "maxCrawlPages": 50,
  "includeExtras": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 10,
  "sitemapConcurrency": 5,
  "maxSitemapDepth": 5,
  "requestTimeoutSecs": 60,
  "siteTimeoutSecs": 900,
  "maxRetries": 2,
  "ignoreSslErrors": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "nodejs.org",
        "https://www.sitemaps.org/sitemap.xml",
        "news.ycombinator.com"
    ],
    "maxUrlsPerSite": 1000
};

// Run the Actor and wait for it to finish
const run = await client.actor("kilndata/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        "nodejs.org",
        "https://www.sitemaps.org/sitemap.xml",
        "news.ycombinator.com",
    ],
    "maxUrlsPerSite": 1000,
}

# Run the Actor and wait for it to finish
run = client.actor("kilndata/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "nodejs.org",
    "https://www.sitemaps.org/sitemap.xml",
    "news.ycombinator.com"
  ],
  "maxUrlsPerSite": 1000
}' |
apify call kilndata/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kilndata/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/OrOxsFhfziKdMdB5e/builds/iZOtCc7pWAJaBYrlZ/openapi.json