Pricing

from $3.80 / 1,000 successful lookups

Sitemap URL Discovery — XML Sitemap & robots.txt API

Discover every URL a website's own XML sitemap declares, starting from robots.txt's Sitemap: directive and following sitemap indexes automatically. No key, no page crawling — charged only when a working sitemap is found.

Pricing

from $3.80 / 1,000 successful lookups

Rating

0.0

(0)

Developer

Adrian Voss

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Sitemap URL Discovery (sitemap.xml + robots.txt → all URLs)

gochujang/sitemap-url-discovery

Given a domain, finds sitemap.xml / sitemap_index.xml (also via robots.txt), recursively expands sitemap indexes, returns one row per discovered URL with lastmod / changefreq / priority. SEO audits, crawl-target prep, content cataloging. $0.0001/URL + $0.01 site fee.

Hojun Lee

GEO Website Auditor & AI Crawler Checker

frere527/geo-ai-visibility-auditor

Audit one page or discover up to 100 same-origin URLs from XML sitemaps. Check AI crawler access, robots.txt, llms.txt, JSON-LD, indexability, citations, and answer-ready content. Export scored GEO reports with transparent evidence and prioritized fixes for AI search visibility.

Frere

Sitemap URL Extractor

automationagents/web-sitemap

Extract every URL from a website via sitemap.xml, robots.txt, or crawl discovery. Feed clean URL lists straight into your scrapers.

Alex Jordan

Sitemap Extractor - Hardened URL Inventory

8tp/sitemap-url-extractor-coverage-api

Extract URL inventories from public HTTPS sitemap XML and gzip files with robots.txt discovery, bounded index traversal, and machine-readable coverage receipts. No page crawling, login, browser, proxy, or private-network access.

Hunter M.

Sitemap URL Extractor — Get All URLs From Any Website

q_services/sitemap-url-extractor

Extract every URL from any website's XML sitemaps. Handles sitemap indexes, robots.txt discovery, URL filtering. Great for SEO audits.

Q Services

Sitemap Extractor: Website → All URLs (sitemap.xml parser)

boxbox10/sitemap-extractor

Give it a website. Get every URL from its sitemap — loc, lastmod, changefreq, priority — as one clean record per URL. Auto-discovers sitemap.xml, robots.txt Sitemap: directives, and nested sitemap indexes. Perfect for SEO audits, crawl seeding, and URL discovery.

Marvin Eguilos

Robots.txt & Sitemap Discovery API

213x/robots-sitemap-discovery

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

Sitemap URL Extractor

ninhothedev/sitemap-url-extractor

$0.5/1K 🔥 Fast sitemap URL extractor! Every URL from any site's sitemap.xml & robots.txt — with lastmod & priority. JSON, CSV, Excel or API in seconds. Paste sites & pull thousands of URLs for SEO & crawling ⚡

ninhothedev

XML Sitemap Auditor, URL Checker & Inventory

xjenn/sitemap-url-qa-inventory

Discover nested XML sitemaps, deduplicate and validate URLs, flag broken pages and redirects, and export a clean crawl inventory to JSON, CSV, Excel, XML, or RSS. Built for SEO audits, site migrations, QA, monitoring, and automation with conservative limits and transparent error records.

Xjenn tools

TrustRadius $1💰 Reviews, Products, Category & Compare Scraper

memo23/trustradius-scraper

💰 $1 per 1,000 results TrustRadius all-in-one scraper — product, reviews, category & compare URLs in one actor. Flat per-review rows with full Q&A sections, reviewer firmographics & LinkedIn URL, dual 0–10/0–5 ratings, awards & featureList. Sitemap-driven bulk mode. Pure HTTP, no browser.

Muhamed Didovic

GEO / AEO Website Audit — AI Search Readiness

rtworule/ai-search-readiness-auditor

Audit public websites for GEO and AEO readiness: AI crawler access, robots.txt, llms.txt, sitemaps, Schema.org, social metadata, scores, and prioritized fixes.