Pricing

from $2.00 / 1,000 website analyzeds

Sitemap & URL Discovery - Find All URLs on Any Site

Discover every URL on any website by parsing sitemap.xml, robots.txt, and sitemap indexes. Extract URLs with last modified dates, change frequency, and priority. Perfect for SEO audits, content analysis, crawling preparation, and site mapping.

Pricing

from $2.00 / 1,000 website analyzeds

Rating

0.0

(0)

Developer

NanoScrape

Actor stats

Bookmarked

Total users

Monthly active users

3 months ago

Last modified

Categories

SEO tools

Developer tools

Websites

websites

Required

List of website URLs to analyze. You can provide bare domains (example.com) or full URLs (https://example.com). The actor will probe robots.txt and the default sitemap locations for each.

Type:array

Follow Sitemap Index

followSitemapIndex

Optional

When a sitemap index () is detected, recursively fetch all child sitemaps. Disable to only inspect the top-level sitemap.

Type:boolean

Default:true

Read robots.txt

respectRobotsTxt

Optional

Read robots.txt and follow any Sitemap: directives found. When disabled, only the default sitemap locations (/sitemap.xml, /sitemap_index.xml, /sitemap.xml.gz) are probed.

Type:boolean

Default:true

Max URLs per Site

maxUrlsPerSite

Optional

Safety cap on URLs returned per website. Large sites (news, e-commerce) can have hundreds of thousands of URLs — increase this value if you need the full set.

Type:integer

Minimum:1

Maximum:1000000

Default:10000

Max Sitemap Depth

maxDepth

Optional

Maximum recursion depth when following nested sitemap indexes. Depth 1 = top-level index + its child sitemaps. Most sites have depth 1-2.

Type:integer

Minimum:1

Maximum:10

Default:3

Include Last Modified

includeLastmod

Optional

Include the date when available. Useful for detecting new or recently updated content.

Type:boolean

Default:true

Include Images

includeImages

Optional

Include image:loc entries from image sitemaps. When enabled, image URLs are emitted as extra rows with is_image=true.

Type:boolean

Default:false

Per-Sitemap Timeout (seconds)

timeoutSeconds

Optional

HTTP timeout applied to every sitemap, robots.txt, and child sitemap fetch.

Type:integer

Minimum:5

Maximum:300

Default:30

Proxy Configuration

proxyConfiguration

Optional

Optional Apify proxy settings. Most sitemaps are served without anti-bot protection, so proxy is usually not needed.

Type:object

Default:

{
  "useApifyProxy": false
}

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

v0iddo/sitemap-url-extractor

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.

vøiddo

Sitemap URL Extractor - List All URLs in a Sitemap

dltik/sitemap-url-extractor

Extract every URL from any XML sitemap, with lastmod, changefreq and priority. Resolves sitemap indexes recursively. Pass a sitemap.xml or just a site root to auto-discover its sitemaps. Pure HTTP, no browser — fast and cheap.

Walid

Sitemap Extractor

automationagents/web-sitemap

Extract all URLs from a website's sitemap (XML, robots.txt, or crawl discovery).

Alex Jordan

Sitemap URL Discovery (sitemap.xml + robots.txt → all URLs)

gochujang/sitemap-url-discovery

Given a domain, finds sitemap.xml / sitemap_index.xml (also via robots.txt), recursively expands sitemap indexes, returns one row per discovered URL with lastmod / changefreq / priority. SEO audits, crawl-target prep, content cataloging. $0.0001/URL + $0.01 site fee.

Hojun Lee

Sitemap to URL Crawler — Extract Sitemap.xml URLs

logiover/sitemap-to-url-crawler

Extract all URLs from any sitemap.xml recursively. Export sitemap URLs to CSV/JSON for RAG pipelines, SEO audits, and LLM training datasets.

Logiover

Sitemap & URL Extractor — Get Every URL of a Website

dataquarry/sitemap-url-extractor

Get every URL of a website: parses sitemap.xml and sitemap-indexes (discovered via robots.txt or the default location), with a same-site crawl fallback when there's no sitemap. Returns each URL + lastmod. No API key.

Daniel Brenner

Sitemap URL Extractor

wiry_kingdom/sitemap-url-extractor

Extract every URL from any website's sitemap.xml with lastmod, changefreq, priority. Recursively expands sitemap index files, reads robots.txt, handles gzipped sitemaps. SEO audits, content migration, site inventory, competitor research.

Mohieldin Mohamed

XML Sitemap Scraper & URL Extractor API - SEO Crawler

pink_comic/sitemap-url-extractor

Extract URLs from XML sitemaps and robots.txt for SEO crawls, audits, content migrations, and RAG indexing. Auto-discovers sitemap files, parses nested sitemap indexes, and exports URL, lastmod, priority, changefreq, and image metadata in bulk.

Ava Torres

Find Sitemap from url

eesti/find-sitemap-from-url

A powerful [Apify Actor] that finds sitemap URLs for any website. This Actor helps you discover XML sitemaps by checking common locations, robots.txt files, and analyzing HTML content for sitemap links.

ando

210

1.0

Find Sitemap From Url

scrapers-hub/find-sitemap-from-url

🔎 Find Sitemap From Url tool instantly locates a website’s sitemap from its URL, helping you discover indexable pages faster. ⚡ Perfect for SEO audits, crawling, indexing analysis & link research. 🚀 Boost efficiency with accurate sitemap detection.