Pricing

$5.00 / 1,000 url auditeds

Robots & Sitemap Inspector

Fetch and validate robots.txt + sitemap.xml for any site. Parses directives, counts sitemap URLs, flags disallow-all mistakes, stale lastmod, missing sitemaps. A-F grade. Use for SEO audits, crawl-budget checks, migration QA. Input: url or urls[]. Output: JSON report.

Pricing

$5.00 / 1,000 url auditeds

Rating

0.0

(0)

Developer

Coleton Patton

Actor stats

Bookmarked

Total users

Monthly active users

7 days ago

Last modified

Sitemap URL Extractor

automationagents/web-sitemap

Extract every URL from a website via sitemap.xml, robots.txt, or crawl discovery. Feed clean URL lists straight into your scrapers.

Alex Jordan

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Stas Persiianenko

Robots.txt & Sitemap Auditor

phoenix2810/robots-sitemap-auditor

Audit a website's robots.txt directives and sitemap discovery in one fast API call. Detect blocked paths, missing sitemaps, and crawlability issues. Built for SEO agencies, technical SEO consultants, and site migration QA teams.

Sanskar Jaiswal

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

v0iddo/sitemap-url-extractor

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.

vøiddo

Sitemap URL Extractor

wiry_kingdom/sitemap-url-extractor

Extract every URL from any website's sitemap.xml with lastmod, changefreq, priority. Recursively expands sitemap index files, reads robots.txt, handles gzipped sitemaps. SEO audits, content migration, site inventory, competitor research.

Mohieldin Mohamed

XML Sitemap Scraper & URL Extractor API - SEO Crawler

pink_comic/sitemap-url-extractor

Extract URLs from XML sitemaps and robots.txt for SEO crawls, audits, content migrations, and RAG indexing. Auto-discovers sitemap files, parses nested sitemap indexes, and exports URL, lastmod, priority, changefreq, and image metadata in bulk.

Ava Torres

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Sitemap URL Discovery (sitemap.xml + robots.txt → all URLs)

gochujang/sitemap-url-discovery

Given a domain, finds sitemap.xml / sitemap_index.xml (also via robots.txt), recursively expands sitemap indexes, returns one row per discovered URL with lastmod / changefreq / priority. SEO audits, crawl-target prep, content cataloging. $0.0001/URL + $0.01 site fee.

Hojun Lee

Sitemap & URL Extractor — Get Every URL of a Website

dataquarry/sitemap-url-extractor

Get every URL of a website: parses sitemap.xml and sitemap-indexes (discovered via robots.txt or the default location), with a same-site crawl fallback when there's no sitemap. Returns each URL + lastmod. No API key.

Daniel Brenner

Sitemap URL Extractor: Every URL, Recursive

thoob/sitemap-extractor

Reads sitemap.xml, sitemap index files, .gz compressed sitemaps, and robots.txt Sitemap directives, and returns one clean row per URL with lastmod, changefreq, and priority. Billed only per delivered URL.

Pono Data