Pricing

from $1.00 / 1,000 result exporteds

Try for free

Go to Apify Store

Sitemap & robots.txt Extractor - All Site URLs

Try for free

Extract every URL a website publishes. Reads robots.txt, discovers sitemaps, walks sitemap indexes recursively and returns URL, path, last modified date, change frequency, priority and hreflang. Optional robots.txt rule parsing. For SEO audits, content inventory and migration planning.

Pricing

from $1.00 / 1,000 result exporteds

Rating

0.0

(0)

Developer

Ryan Zinburg

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Sitemap & URL Discovery - Find All URLs on Any Site

santamaria-automations/sitemap-url-discovery

Discover every URL on any website by parsing sitemap.xml, robots.txt, and sitemap indexes. Extract URLs with last modified dates, change frequency, and priority. Perfect for SEO audits, content analysis, crawling preparation, and site mapping.

NanoScrape

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

Sitemap URL Extractor

receptional_blender/sitemap-url-extractor

Give it any website or sitemap URL and get back every page URL. It reads robots.txt and sitemap.xml, follows nested sitemap indexes, and returns each URL with its last-modified date, change frequency and priority as clean JSON. No key, login or proxy required.

Assia Fadli

Sitemap URL Extractor — Get All URLs From Any Website

q_services/sitemap-url-extractor

Extract every URL from any website's XML sitemaps. Handles sitemap indexes, robots.txt discovery, URL filtering. Great for SEO audits.

Q Services

Sitemap URL Extractor — every URL, $0.0002/URL

plainapi/sitemap-xml-url-list-extractor

You pay only for URLs actually returned. Nothing found, nothing charged. List every URL a site publishes, from robots.txt and its sitemaps, with last-modified dates. Expands sitemap indexes, honours robots.txt, never crawls or guesses.

Shun Furu

Sitemap URL Extractor

lnlenost/sitemap-url-extractor

Extract page URLs from robots.txt and sitemap.xml files for SEO audits, URL discovery crawl planning, and data pipelines.

Niccolò Salerno

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

v0iddo/sitemap-url-extractor

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.

vøiddo

Sitemap URL Extractor: robots.txt, gzip, Changes

succinct_glider/sitemap-urls

Extract every URL from a website's XML sitemaps: found through robots.txt, sitemap indexes and .gz files followed, lastmod and date filters, URL patterns, and new, changed or removed URLs since your last run. Reads sitemap files only, never the pages.

Alexandre Bobichon

Robots.txt & Sitemap Extractor by Domain

technicaldost/robots-sitemap-extractor

Fetch and parse robots.txt and XML sitemaps for any domain. Extract allowed and disallowed paths, sitemap URLs and every listed page. Great for SEO audits and crawl planning. JSON output.

Technical Dost Solutions

Sitemap Parser

blaspat/sitemap-parser

Parse any XML sitemap — extract all URLs with last modified dates, change frequency, and priority. Handles sitemap indexes recursively. Perfect for SEO auditing, migration planning, and content monitoring

Blasius Patrick