# URL & Sitemap Link Extractor: Site Map, Health Checks & Monitor (`getascraper/url-link-extractor`) Actor

Extract every URL from a site's sitemap or a page, no sitemap link needed. Auto-discovers via robots.txt, follows sitemap-index nesting, keeps lastmod, changefreq, and priority. Adds a site-structure summary, link-health checks, and change monitoring. Export as JSON, CSV, or Excel. $0.00001 per URL.

- **URL**: https://apify.com/getascraper/url-link-extractor.md
- **Developed by:** [GetAScraper](https://apify.com/getascraper) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.01 / 1,000 url records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🔗 URL & Sitemap Link Extractor: Site Map, Health Checks & Monitor

<table width="100%">
<tr>
<td style="padding:24px 28px;background:#EAF4FB;border:1px solid #BFDCF2;border-top:4px solid #0B4C7A;border-radius:12px">
<span style="font-size:23px;font-weight:800;color:#1C1917;line-height:1.3">Every URL on a site, found automatically and tracked over time</span><br>
<span style="font-size:15px;color:#57534E;line-height:1.6">Give it a domain, not a sitemap link. It finds every page, keeps the dates the site itself reports, and tells you exactly what's new on every re-run.</span>
</td>
</tr>
</table>

<table width="100%">
<tr>
<td style="padding:14px 12px;width:25%;background:#FFFFFF;border:1px solid #BFDCF2;border-radius:10px 0 0 10px;vertical-align:top">
<span style="font-size:15px;font-weight:800;color:#0B4C7A">🌐 Auto-discovery</span><br>
<span style="font-size:12px;color:#57534E">Give it a domain, not a sitemap link. It finds and follows the sitemap on its own.</span>
</td>
<td style="padding:14px 12px;width:25%;background:#FFFFFF;border:1px solid #BFDCF2;border-left:none;vertical-align:top">
<span style="font-size:15px;font-weight:800;color:#0B4C7A">📅 Real update dates</span><br>
<span style="font-size:12px;color:#57534E">Keeps last-modified date, change frequency, and priority when the site provides them.</span>
</td>
<td style="padding:14px 12px;width:25%;background:#FFFFFF;border:1px solid #BFDCF2;border-left:none;vertical-align:top">
<span style="font-size:15px;font-weight:800;color:#0B4C7A">🗂️ Site structure map</span><br>
<span style="font-size:12px;color:#57534E">See how many pages live under each section, at a glance.</span>
</td>
<td style="padding:14px 12px;width:25%;background:#FFFFFF;border:1px solid #BFDCF2;border-left:none;border-radius:0 10px 10px 0;vertical-align:top">
<span style="font-size:15px;font-weight:800;color:#0B4C7A">🔔 Change tracking</span><br>
<span style="font-size:12px;color:#57534E">Get only what's new or changed, grouped by section, on every re-run.</span>
</td>
</tr>
</table>

### 🔍 What does URL & Sitemap Link Extractor do?

URL & Sitemap Link Extractor pulls every URL out of a website's sitemap, or the links on a single page, and returns them as clean structured data. Point it at a bare domain and it checks `robots.txt` for the sitemap itself, following any linked sub-sitemaps automatically, no need to already know the exact sitemap address. Run it on Apify with no setup, schedule it to catch new pages as they go live, and export the results as JSON, CSV, or Excel.

### 💡 Why use URL & Sitemap Link Extractor?

**"I'm feeding a site into my RAG pipeline."** You need a real, complete inventory of every page before you can scrape and embed it, not a hand-built list that misses half the site.

**"I'm auditing a site's structure and links."** You need to see how a site is actually organized, section by section, and catch broken links before your client or your boss does.

**"I'm watching what a competitor publishes."** You need to know the moment they post something new, not find out a week later when you happen to check.

The other URL and sitemap extractors on Apify Store all need you to already know and paste the exact sitemap link, hand back a flat list with no dates attached, and offer no way to track what changed between runs, even though the most popular one shows people re-running it around 280 times each on average. This Actor is built for that exact pattern: point it at a domain, keep the dates the site reports, and turn re-running it into an actual change feed.

### 🪜 How to use URL & Sitemap Link Extractor

<table width="100%">
<tr>
<td style="padding:16px 14px;width:33%;background:#EAF4FB;border:1px solid #BFDCF2;border-radius:10px 0 0 10px;vertical-align:top">
<span style="font-size:12px;font-weight:800;color:#0B4C7A;letter-spacing:1px">STEP 1</span><br>
<span style="font-size:14px;font-weight:700;color:#1C1917">Point it at a domain</span><br>
<span style="font-size:12px;color:#57534E">Or paste a specific sitemap or page URL if you already have one.</span>
</td>
<td style="padding:16px 14px;width:33%;background:#EAF4FB;border:1px solid #BFDCF2;border-left:none;vertical-align:top">
<span style="font-size:12px;font-weight:800;color:#0B4C7A;letter-spacing:1px">STEP 2</span><br>
<span style="font-size:14px;font-weight:700;color:#1C1917">Set your filters</span><br>
<span style="font-size:12px;color:#57534E">Narrow by path or file type, or turn on a link-health check.</span>
</td>
<td style="padding:16px 14px;width:33%;background:#EAF4FB;border:1px solid #BFDCF2;border-left:none;border-radius:0 10px 10px 0;vertical-align:top">
<span style="font-size:12px;font-weight:800;color:#0B4C7A;letter-spacing:1px">STEP 3</span><br>
<span style="font-size:14px;font-weight:700;color:#1C1917">Start and export</span><br>
<span style="font-size:12px;color:#57534E">Browse the results, or turn on Monitor Mode to track changes over time.</span>
</td>
</tr>
</table>

### ⚙️ Input

Configure the run from the Apify Console form, or pass the same fields via the API.

| Field | Type | Required | Description |
|---|---|---|---|
| `domain` | string | No | A bare domain to auto-discover its sitemap. Required if `links` is empty. |
| `links` | array of URLs | No | Specific sitemap or page URLs, when you already know them. Required if `domain` is empty. |
| `urlPattern` | string | No | Only keep URLs whose path contains one of these comma-separated patterns. |
| `customExcludePattern` | string | No | Drop URLs whose path contains any of these comma-separated patterns. |
| `excludeExtensions` | string | No | Drop URLs ending in these comma-separated file extensions. |
| `checkLinkHealth` | boolean | No | Send a request to a sample of the returned URLs and record their HTTP status. |
| `maxUrls` | integer | No | Maximum number of URLs to return in this run, after filters. |
| `maxSitemapsToFollow` | integer | No | Maximum number of sitemap or page URLs to actually fetch. |
| `onlyNewOrChanged` | boolean | No | Only return URLs that are new or changed since your last run with the same monitor state name. |
| `stateName` | string | No | Name for this run's incremental state, used only when Incremental Monitor Mode is on. |
| `proxyConfiguration` | object | No | Proxy settings. Defaults to datacenter, which works for every site checked during development. |

### 📦 Output

Each dataset item is one URL, with whatever date and priority information the site's sitemap actually provides.

```json
{
  "url": "https://example.com/blog/my-post",
  "lastmod": "2026-09-09",
  "sourceSitemap": "https://example.com/sitemap-page-1.xml"
}
```

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. Every run also writes a site-structure summary (URL counts grouped by section) to the run's key-value store, and, when Incremental Monitor Mode is on, a change summary grouped the same way.

#### Data table

| Field | Type | Description |
|---|---|---|
| `url` | string | The extracted URL. |
| `lastmod` | string | Last-modified date, when the sitemap provides it. |
| `changefreq` | string | How often the page is expected to change, when the sitemap provides it. |
| `priority` | string | The page's relative priority, when the sitemap provides it. |
| `sourceSitemap` | string | The sitemap or page this URL was found on. |
| `linkStatus` | number | The URL's HTTP status, only present when Check Link Health is on. |
| `changeType` | string | `NEW` or `UPDATED`, only present when Incremental Monitor Mode is on. |

### 💰 Pricing

This Actor uses Apify's pay-per-event pricing. You pay a small fee when a run starts plus a fee per URL returned, an empty run costs almost nothing, and there is no subscription.

### ⭐ Enjoying URL & Sitemap Link Extractor?

<table width="100%" style="display:table;width:100%">
<tr>
<td style="padding:20px 24px 14px;background:#EAF4FB;border:1px solid #BFDCF2;border-left:5px solid #0B4C7A;border-radius:10px 10px 0 0">
<span style="font-size:20px;letter-spacing:4px">⭐ ⭐ ⭐ ⭐ ⭐</span><br>
<span style="font-size:17px;font-weight:800;color:#1C1917">Stopped you from hand-checking a sitemap for the hundredth time?</span><br>
<span style="font-size:14px;color:#57534E">A 5-star rating takes 10 seconds and helps other RAG builders, SEO auditors, and site-watchers find this Actor. Your feedback also tells us what to build next.</span>
</td>
</tr>
<tr>
<td style="padding:0;background:#0B4C7A;border:1px solid #BFDCF2;border-top:none;border-radius:0 0 10px 10px;text-align:center">
<a href="https://apify.com/getascraper/url-link-extractor/reviews" style="display:block;padding:13px 16px;color:#FFFFFF;text-decoration:none;font-weight:800;font-size:15px;letter-spacing:0.3px">★&nbsp;&nbsp;Rate this Actor on Apify</a>
</td>
</tr>
</table>

### ❓ FAQ

**Do I need to know the exact sitemap URL?**
No. Give it a bare domain and it checks the site's `robots.txt` for the sitemap and follows it automatically, including sites that split their sitemap into several linked files. If you already have a specific sitemap or page URL, you can paste that directly instead.

**Why don't some URLs have a last-modified date?**
Not every site includes that information in its sitemap. When it's missing, the field is left out entirely rather than guessed, so every date you do see is genuine.

**Can I track a site for new or changed pages?**
Yes. Turn on Incremental Monitor Mode and run this Actor on a schedule. It only returns URLs that are new or whose date, priority, or change frequency changed since your last run with the same monitor state name, and it groups those changes by section so you can see where the site actually changed.

**Does this check if links are broken?**
Turn on Check Link Health and it sends a request to a bounded sample of the returned URLs and reports each one's HTTP status, so you can spot broken links without a separate tool.

**Is this legal?**
This Actor only reads publicly available sitemap and page data the same way a search engine would. You are responsible for using the data in line with each site's terms of service and any applicable law in your jurisdiction.

**Something looks off or missing. Can you fix it?**
Yes. Open an issue from the Actor's Issues tab with details, and we'll take a look.

### 🔗 Other actors

- [arXiv scraper for RAG: papers as chunked JSON](https://apify.com/getascraper/arxiv-rag-extractor) ↗ - Extracts arXiv papers as clean, chunked JSON ready for embedding.
- [PubMed Scraper for RAG: Papers as Chunked JSON](https://apify.com/getascraper/pubmed-rag-extractor) ↗ - Extracts PubMed papers as clean, chunked JSON ready for embedding.
- [SEC EDGAR Scraper for RAG: 10-K/10-Q/8-K as JSON](https://apify.com/getascraper/sec-edgar-rag-extractor) ↗ - Extracts SEC filings as clean, structured JSON ready for embedding.
- [Google Lens OCR API: Image to Text with a Live Standby Endpoint](https://apify.com/getascraper/google-lens-ocr) ↗ - Turns any image into text through a live, agent-callable endpoint.
- [RapidAPI Hub catalog intelligence](https://apify.com/getascraper/rapidapi-scraper) ↗ - Extracts API listings, pricing, and developer data from the RapidAPI Hub.

# Actor input Schema

## `domain` (type: `string`):

A bare domain (e.g. example.com). The Actor checks robots.txt for its declared sitemap(s) and follows any sitemap index automatically, no need to know the exact sitemap URL. Ignored whenever Specific Sitemap or Page URLs below is filled in, so clear this field or add your own URLs there, not both.

## `links` (type: `array`):

One or more exact sitemap URLs or webpage URLs to extract from, when you already know them. A sitemap index is still followed automatically. A plain webpage URL returns the links found on that one page (no further crawling).

## `urlPattern` (type: `string`):

Only keep URLs whose path contains one of these comma-separated patterns (e.g. "/blog, /products"). Leave blank to keep every URL.

## `customExcludePattern` (type: `string`):

Drop URLs whose path contains any of these comma-separated patterns (e.g. "/tag, /author"). Leave blank to skip this filter.

## `excludeExtensions` (type: `string`):

Drop URLs ending in these comma-separated file extensions (e.g. "pdf, jpg, zip"). Leave blank to skip this filter.

## `checkLinkHealth` (type: `boolean`):

For a bounded sample of the returned URLs, send a request to each and record its HTTP status, so you can spot broken links. Costs one extra request per checked URL, so it defaults off.

## `maxUrls` (type: `integer`):

Maximum number of URLs to return in this run, after filters are applied.

## `maxSitemapsToFollow` (type: `integer`):

Maximum number of sitemap or page URLs to actually fetch. A sitemap index can list hundreds or thousands of nested sitemaps (a real large site can have 2,000+); this caps how many of those are followed in one run.

## `onlyNewOrChanged` (type: `boolean`):

Only return URLs that are new or whose lastmod/changefreq/priority changed since your last run with the same monitor state name. Pair with an Apify schedule to catch new or updated pages without re-downloading the whole sitemap every time.

## `stateName` (type: `string`):

Name for this run's incremental state, used only when Incremental Monitor Mode is on. Use different names to track separate domains or filters independently.

## `proxyConfiguration` (type: `object`):

This Actor makes plain HTTP requests to fetch sitemaps and pages. Datacenter proxy (the default) is fast and works for every target checked during development; switch to Residential only if a specific site you're targeting starts blocking datacenter IPs.

## Actor input object example

```json
{
  "domain": "apify.com",
  "links": [],
  "urlPattern": "",
  "customExcludePattern": "",
  "excludeExtensions": "",
  "checkLinkHealth": false,
  "maxUrls": 25,
  "maxSitemapsToFollow": 5,
  "onlyNewOrChanged": false,
  "stateName": "default",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domain": "apify.com",
    "links": [],
    "urlPattern": "",
    "customExcludePattern": "",
    "excludeExtensions": "",
    "checkLinkHealth": false,
    "maxUrls": 25,
    "maxSitemapsToFollow": 5,
    "onlyNewOrChanged": false,
    "stateName": "default",
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("getascraper/url-link-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domain": "apify.com",
    "links": [],
    "urlPattern": "",
    "customExcludePattern": "",
    "excludeExtensions": "",
    "checkLinkHealth": False,
    "maxUrls": 25,
    "maxSitemapsToFollow": 5,
    "onlyNewOrChanged": False,
    "stateName": "default",
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("getascraper/url-link-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domain": "apify.com",
  "links": [],
  "urlPattern": "",
  "customExcludePattern": "",
  "excludeExtensions": "",
  "checkLinkHealth": false,
  "maxUrls": 25,
  "maxSitemapsToFollow": 5,
  "onlyNewOrChanged": false,
  "stateName": "default",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call getascraper/url-link-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,getascraper/url-link-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Qy1A4agtyelDfh7dt/builds/0POxGXtqbje0Fm3Kv/openapi.json