Pricing

from $2.00 / 1,000 results

Robots.txt & Sitemap Extractor by Domain

Fetch and parse robots.txt and XML sitemaps for any domain. Extract allowed and disallowed paths, sitemap URLs and every listed page. Great for SEO audits and crawl planning. JSON output.

Pricing

from $2.00 / 1,000 results

Rating

0.0

(0)

Developer

Technical Dost Solutions

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Robots.txt & Sitemap Extractor - Crawl & SEO Audit Tool API through CLI

The Apify CLI is the official tool that allows you to use Robots.txt & Sitemap Extractor by Domain locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

Robots.txt & Sitemap Extractor by Domain API in Python

Robots.txt & Sitemap Extractor by Domain API in JavaScript

Robots.txt & Sitemap Extractor by Domain OpenAPI definition

Robots.txt & Sitemap Extractor by Domain API

Sitemap URL Extractor - XML Sitemap Scraper

benthepythondev/sitemap-url-extractor

Extract URLs from XML sitemaps and sitemap indexes. Get URL, lastmod, changefreq, priority and source sitemap.

Ben

Find Sitemap From Url

scrapers-hub/find-sitemap-from-url

🔎 Find Sitemap From Url tool instantly locates a website’s sitemap from its URL, helping you discover indexable pages faster. ⚡ Perfect for SEO audits, crawling, indexing analysis & link research. 🚀 Boost efficiency with accurate sitemap detection.

Scrapers Hub

Website Metadata Extractor(sitemap, socialLinks, robotsTxt)

codescraper/website-metadata-extractor

A very fast metadata extractor to get all meta tags, robots.txt, sitemaps, social links, H1s, word count, and JSON-LD data. Also provides technology detection for a full analysis. Get your data fast for just $3/month.

CodeScraper

104

5.0

(2)

Sitemap Scraper

scrapers-hub/sitemap-scraper

Sitemap scraper to crawl and extract URLs, pages, and structure from website sitemaps 🌐📊 Perfect for SEO analysis, website auditing, and data extraction. Fast, reliable, and scalable.

Scrapers Hub

Company Domain Tech, DNS & Security Audit

taroyamada/company-enrichment-intelligence

Turn company domains into structured website, tech stack, DNS, SPF, DMARC, RDAP, TLS certificate, robots.txt, and sitemap audit records. Public, keyless sources only: the company's own website plus public registries. Pay per audited domain.

naoki anzai

Website Metadata Extractor

scrapers-hub/website-metadata-extractor

Website metadata extractor to extract titles, descriptions, keywords, and meta tags from any website 🌐📊 Perfect for SEO analysis, auditing, and research. Fast, accurate, and scalable extraction.

Scrapers Hub

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

v0iddo/sitemap-url-extractor

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.

vøiddo

Sitemap Url Extractor

scrapers-hub/sitemap-url-extractor

Sitemap URL extractor to extract all URLs from XML sitemaps quickly and efficiently 🌐📄 Ideal for SEO audits, site analysis, and indexing workflows. Fast, accurate, and easy to use.

Scrapers Hub