Pricing

$0.30 / 1,000 url extracteds

Try for free

Go to Apify Store

Sitemap URL Extractor – All URLs from sitemap.xml & robots.txt

Try for free

Extract page URLs from a website's sitemaps. Finds sitemaps via robots.txt, common paths and homepage links, then reads XML and .xml.gz sitemaps, sitemap indexes, text sitemaps, RSS and Atom feeds. Returns lastmod, changefreq, priority, images, videos and hreflang.

Pricing

$0.30 / 1,000 url extracteds

Rating

0.0

(0)

Developer

Enes Bulanık

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

Categories

SEO tools

Developer tools

Lead generation

Websites or sitemap URLs

startUrls

Required

Domains (https://example.com or example.com) or direct links to sitemap.xml, sitemap index, .xml.gz, .txt sitemaps, RSS or Atom feeds.

Type:array

Default:

[
  {
    "url": "https://apify.com"
  }
]

Max URLs

maxUrls

Optional

Maximum number of URLs to return in total (0 = unlimited). You only pay for URLs actually returned.

Type:integer

Minimum:0

Default:500

Max URLs per website

maxUrlsPerSite

Optional

Optional cap per start URL so one huge website does not use the whole Max URLs budget when you process several sites (0 = no per-site cap).

Type:integer

Minimum:0

Default:0

Follow sitemap indexes

followSitemapIndex

Optional

Open child sitemaps listed in sitemap index files (recommended).

Type:boolean

Default:true

Discover sitemaps from robots.txt

includeRobotsTxtDiscovery

Optional

Read the site's robots.txt and use every Sitemap: line.

Type:boolean

Default:true

Also try common sitemap paths

tryCommonPaths

Optional

Probe /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /news-sitemap.xml, /sitemap_news.xml and /sitemap.txt. Missing ones are skipped silently.

Type:boolean

Default:true

Discover sitemaps & RSS/Atom feeds from the homepage

discoverFromHomepage

Optional

If robots.txt lists no sitemap, look for and RSS/Atom feed links on the homepage.

Type:boolean

Default:true

Include URLs matching (regex)

urlIncludeRegex

Optional

Only return URLs matching this regular expression (case-insensitive). Example: /blog/ or .html$

Type:string

Exclude URLs matching (regex)

urlExcludeRegex

Optional

Skip URLs matching this regular expression (case-insensitive). Example: /tag/|/author/

Type:string

Only URLs on the same domain

sameDomainOnly

Optional

Drop URLs whose host differs from the start website (subdomains are kept).

Type:boolean

Default:false

Include metadata

includeMeta

Optional

Add lastmod, changefreq, priority, image/video/news extensions, hreflang alternates and feed titles. Turn off for a lean URL-only list.

Type:boolean

Default:true

Remove duplicate URLs

dedupe

Optional

Return each URL only once even if it appears in several sitemaps (duplicates are never charged).

Type:boolean

Default:true

Save run summary (OUTPUT record)

saveSummary

Optional

Store a summary with URL counts per sitemap, discovery source and errors in the key-value store under OUTPUT.

Type:boolean

Default:true

Max parallel downloads

maxConcurrency

Optional

How many sitemaps are downloaded in parallel.

Type:integer

Minimum:1

Maximum:50

Default:8

Max sitemaps per website

maxSitemaps

Optional

Safety cap on the number of sitemap files opened for each start URL (huge sites can have thousands of child sitemaps).

Type:integer

Minimum:1

Default:10000

Request timeout (seconds)

requestTimeoutSecs

Optional

Timeout for the first byte and for idle connections. Large sitemaps keep streaming as long as data flows.

Type:integer

Minimum:5

Maximum:300

Default:30

Max retries

maxRetries

Optional

Retries for network errors, timeouts, 429 and 5xx responses (exponential backoff).

Type:integer

Minimum:0

Maximum:10

Default:3

Proxy configuration

proxyConfiguration

Optional

Most sitemaps are public and need no proxy. Enable Apify Proxy for sites that block data-center IPs.

Type:object

Default:

{
  "useApifyProxy": false
}

XML Sitemap Finder & Extractor API

andok/find-sitemap-from-url

Find and extract all XML sitemaps for any domain. Automatically parses robots.txt, scans HTML tags, and recursively follows indexes. Perfect for SEO & web scraping.

Andok

Website Sitemap Extractor

glassventures/website-sitemap-extractor

Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range.

Glass Ventures

Sitemap Generator - Crawl Website & Create XML Sitemap

scrappy_garden/sitemap-generator

Generate an XML sitemap for any website. Crawls internal pages from start URLs (with depth + page limits), deduplicates URLs, and stores a ready-to-submit sitemap.xml plus a structured dataset and summary for SEO audits.

Bikram Adhikari

SiteVerdict — robots.txt & Sitemap Monitor

siteverdict/siteverdict-robots-sitemap-monitor

Catch SEO disasters before Google does: watches robots.txt, XML sitemaps and key pages for Disallow: /, noindex, canonical and status changes, sitemap URL drops and errors. Diffs vs last run, Slack/webhook alerts, HTML report.

Aman Thakran

Sitemap URL Extractor — recursive, filters, .gz

x402opklaar/sitemap-url-extractor

Extract every URL from any sitemap: recursive sitemap-index expansion, gzipped sitemaps, robots.txt auto-discovery, include/exclude filters, lastmod output. Feed the URL list to any crawler or audit. Pay per sitemap file processed.

Opklaar

Lighthouse Batch Audits & Regression Alerts

peerless_columbine/lighthouse-batch-audit-monitor

Audit public pages with Lighthouse13.5.0, export raw and HTML reports, and compare persistent category baselines.

tingyou333 zhuang

SEO Audit API — On-Page Checks, Meta Tags, Headings & Schema

highbrow_fame/seo-audit-api

On-page SEO audit of any URL over plain HTTP: title, meta, canonical, robots, hreflang, headings, images alt, links, JSON-LD, redirects, a 0-100 score with issues. SEO Audit API.

yestrue

WeChat Article to Clean Markdown & JSON Extractor

knightwwang/wechat-article-to-markdown

Convert WeChat Official Account articles (mp.weixin.qq.com) into clean Markdown with embedded images, author info, publish date, and AI-ready tokens.

knightwwang Wang

E-E-A-T & SEO Compliance Auditor

daslay43/google-eeat-seo-auditor

Automated site quality and E-E-A-T audit crawler. Crawls pages to analyze author signals, JSON-LD Schema markup, sitewide trust links, and content depth—generating an instant compliance score and prioritized fix list to protect rankings against Google core updates.