🔗 URL & Sitemap Link Extractor: Site Map, Health Checks & Monitor

Every URL on a site, found automatically and tracked over time Give it a domain, not a sitemap link. It finds every page, keeps the dates the site itself reports, and tells you exactly what's new on every re-run.

🌐 Auto-discovery Give it a domain, not a sitemap link. It finds and follows the sitemap on its own.

📅 Real update dates Keeps last-modified date, change frequency, and priority when the site provides them.

🗂️ Site structure map See how many pages live under each section, at a glance.

🔔 Change tracking Get only what's new or changed, grouped by section, on every re-run.

🔍 What does URL & Sitemap Link Extractor do?

URL & Sitemap Link Extractor pulls every URL out of a website's sitemap, or the links on a single page, and returns them as clean structured data. Point it at a bare domain and it checks robots.txt for the sitemap itself, following any linked sub-sitemaps automatically, no need to already know the exact sitemap address. Run it on Apify with no setup, schedule it to catch new pages as they go live, and export the results as JSON, CSV, or Excel.

💡 Why use URL & Sitemap Link Extractor?

"I'm feeding a site into my RAG pipeline." You need a real, complete inventory of every page before you can scrape and embed it, not a hand-built list that misses half the site.

"I'm auditing a site's structure and links." You need to see how a site is actually organized, section by section, and catch broken links before your client or your boss does.

"I'm watching what a competitor publishes." You need to know the moment they post something new, not find out a week later when you happen to check.

The other URL and sitemap extractors on Apify Store all need you to already know and paste the exact sitemap link, hand back a flat list with no dates attached, and offer no way to track what changed between runs, even though the most popular one shows people re-running it around 280 times each on average. This Actor is built for that exact pattern: point it at a domain, keep the dates the site reports, and turn re-running it into an actual change feed.

🪜 How to use URL & Sitemap Link Extractor

STEP 1 Point it at a domain Or paste a specific sitemap or page URL if you already have one.

STEP 2 Set your filters Narrow by path or file type, or turn on a link-health check.

STEP 3 Start and export Browse the results, or turn on Monitor Mode to track changes over time.

⚙️ Input

Configure the run from the Apify Console form, or pass the same fields via the API.

Field	Type	Required	Description
domain	string	No	A bare domain to auto-discover its sitemap. Required if links is empty.
links	array of URLs	No	Specific sitemap or page URLs, when you already know them. Required if domain is empty.
urlPattern	string	No	Only keep URLs whose path contains one of these comma-separated patterns.
customExcludePattern	string	No	Drop URLs whose path contains any of these comma-separated patterns.
excludeExtensions	string	No	Drop URLs ending in these comma-separated file extensions.
checkLinkHealth	boolean	No	Send a request to a sample of the returned URLs and record their HTTP status.
maxUrls	integer	No	Maximum number of URLs to return in this run, after filters.
maxSitemapsToFollow	integer	No	Maximum number of sitemap or page URLs to actually fetch.
onlyNewOrChanged	boolean	No	Only return URLs that are new or changed since your last run with the same monitor state name.
stateName	string	No	Name for this run's incremental state, used only when Incremental Monitor Mode is on.
proxyConfiguration	object	No	Proxy settings. Defaults to datacenter, which works for every site checked during development.

📦 Output

Each dataset item is one URL, with whatever date and priority information the site's sitemap actually provides.

{
  "url": "https://example.com/blog/my-post",
  "lastmod": "2026-09-09",
  "sourceSitemap": "https://example.com/sitemap-page-1.xml"
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. Every run also writes a site-structure summary (URL counts grouped by section) to the run's key-value store, and, when Incremental Monitor Mode is on, a change summary grouped the same way.

Data table

Field	Type	Description
url	string	The extracted URL.
lastmod	string	Last-modified date, when the sitemap provides it.
changefreq	string	How often the page is expected to change, when the sitemap provides it.
priority	string	The page's relative priority, when the sitemap provides it.
sourceSitemap	string	The sitemap or page this URL was found on.
linkStatus	number	The URL's HTTP status, only present when Check Link Health is on.
changeType	string	NEW or UPDATED, only present when Incremental Monitor Mode is on.

💰 Pricing

This Actor uses Apify's pay-per-event pricing. You pay a small fee when a run starts plus a fee per URL returned, an empty run costs almost nothing, and there is no subscription.

⭐ Enjoying URL & Sitemap Link Extractor?

⭐ ⭐ ⭐ ⭐ ⭐ Stopped you from hand-checking a sitemap for the hundredth time? A 5-star rating takes 10 seconds and helps other RAG builders, SEO auditors, and site-watchers find this Actor. Your feedback also tells us what to build next.

★  Rate this Actor on Apify

❓ FAQ

Do I need to know the exact sitemap URL? No. Give it a bare domain and it checks the site's robots.txt for the sitemap and follows it automatically, including sites that split their sitemap into several linked files. If you already have a specific sitemap or page URL, you can paste that directly instead.

Why don't some URLs have a last-modified date? Not every site includes that information in its sitemap. When it's missing, the field is left out entirely rather than guessed, so every date you do see is genuine.

Can I track a site for new or changed pages? Yes. Turn on Incremental Monitor Mode and run this Actor on a schedule. It only returns URLs that are new or whose date, priority, or change frequency changed since your last run with the same monitor state name, and it groups those changes by section so you can see where the site actually changed.

Does this check if links are broken? Turn on Check Link Health and it sends a request to a bounded sample of the returned URLs and reports each one's HTTP status, so you can spot broken links without a separate tool.

Is this legal? This Actor only reads publicly available sitemap and page data the same way a search engine would. You are responsible for using the data in line with each site's terms of service and any applicable law in your jurisdiction.

Something looks off or missing. Can you fix it? Yes. Open an issue from the Actor's Issues tab with details, and we'll take a look.

🔗 Other actors

• arXiv scraper for RAG: papers as chunked JSON ↗ - Extracts arXiv papers as clean, chunked JSON ready for embedding.
• PubMed Scraper for RAG: Papers as Chunked JSON ↗ - Extracts PubMed papers as clean, chunked JSON ready for embedding.
• SEC EDGAR Scraper for RAG: 10-K/10-Q/8-K as JSON ↗ - Extracts SEC filings as clean, structured JSON ready for embedding.
• Google Lens OCR API: Image to Text with a Live Standby Endpoint ↗ - Turns any image into text through a live, agent-callable endpoint.
• RapidAPI Hub catalog intelligence ↗ - Extracts API listings, pricing, and developer data from the RapidAPI Hub.