Pricing

from $2.00 / 1,000 successful lookups

Try for free

Go to Apify Store

Sitemap URL Extractor: XML Sitemap & robots.txt

Try for free

Discover every URL a website's own XML sitemap declares, starting from robots.txt's Sitemap: directive and following sitemap indexes automatically. No key, no page crawling: charged only when a working sitemap is found.

Pricing

from $2.00 / 1,000 successful lookups

Rating

0.0

(0)

Developer

Adrian Voss

Actor stats

Bookmarked

Total users

Monthly active users

18 hours ago

Last modified

Categories

Lead generation

Developer tools

You can access the Sitemap URL Extractor: XML Sitemap & robots.txt programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "items": ["bbc.co.uk"] }
9
10# Run the Actor and wait for it to finish
11run = client.actor("accountable_eel/sitemap-url-discovery").call(run_input=run_input)
12
13# Fetch and print Actor results from the run's dataset (if there are any)
14print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
15for item in client.dataset(run.default_dataset_id).iterate_items():
16    print(item)
17
18# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Sitemap URL Extractor | Apify API in Python

The Apify API client for Python is the official library that allows you to use Sitemap URL Extractor: XML Sitemap & robots.txt API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Sitemap URL Extractor: XML Sitemap & robots.txt API in JavaScript

Sitemap URL Extractor: XML Sitemap & robots.txt API through CLI

Sitemap URL Extractor: XML Sitemap & robots.txt OpenAPI definition

Sitemap URL Extractor: XML Sitemap & robots.txt API

Sitemap URL Intelligence

toronto_777/sitemap-url-intelligence

Discover robots.txt sitemap entries and classify public sitemap URLs by page type.

Steven Feng

Sitemap Extractor: Website → All URLs (sitemap.xml parser)

boxbox10/sitemap-extractor

Give it a website. Get every URL from its sitemap — loc, lastmod, changefreq, priority — as one clean record per URL. Auto-discovers sitemap.xml, robots.txt Sitemap: directives, and nested sitemap indexes. Perfect for SEO audits, crawl seeding, and URL discovery.

Marvin Eguilos

Sitemap API

vivid_astronaut/sitemap

BRAINIALL Team

Sitemap URL Extractor

leorochasantos/sitemap-extractor

Extract every URL from any XML sitemap — support sitemap indexes, gzip, robots.txt discovery, and metadata extraction. Returns clean structured output: one URL per row.

Leonardo Santos

Sitemap & URL Extractor — Get Every URL of a Website

dataquarry/sitemap-url-extractor

Get every URL of a website: parses sitemap.xml and sitemap-indexes (discovered via robots.txt or the default location), with a same-site crawl fallback when there's no sitemap. Returns each URL + lastmod. No API key.

Daniel Brenner

XML Sitemap URL Extractor

devilscrapes/xml-sitemap-url-extractor

Turn any site's sitemap.xml, sitemap index, or robots.txt Sitemap directive into a clean URL dataset with loc, lastmod, changefreq and priority per row. Recursively expands sitemap indexes (bounded depth), follows redirects, and decompresses .xml.gz automatically.

DevilScrapes

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

v0iddo/sitemap-url-extractor

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.

vøiddo

Sitemap URL Extractor - XML Sitemap Scraper

benthepythondev/sitemap-url-extractor

Extract URLs from XML sitemaps and sitemap indexes. Get URL, lastmod, changefreq, priority and source sitemap.

Ben

Sitemap URL Extractor

lost.fany/sitemap-url-extractor

Extract every URL from a website's sitemap.xml, sitemap index and robots.txt in seconds — with lastmod, changefreq, priority and hreflang. No crawling, no browser: fast, cheap, reliable.

Natalia Ivanova

Sitemap URL Extractor

receptional_blender/sitemap-url-extractor

Give it any website or sitemap URL and get back every page URL. It reads robots.txt and sitemap.xml, follows nested sitemap indexes, and returns each URL with its last-modified date, change frequency and priority as clean JSON. No key, login or proxy required.