Pricing

$0.30 / 1,000 url discovereds

Try for free

Go to Apify Store

Sitemap & robots.txt URL Discovery (no crawling)

Try for free

Feed it site roots, get back URLs from their sitemaps plus each site's disallow rules and crawl-delay. Streams sitemaps, follows nested indexes and handles gzip without crawling page bodies. For seeding crawlers, SEO audits and migration checks.

Pricing

$0.30 / 1,000 url discovereds

Rating

0.0

(0)

Developer

Paul Vasquez

Actor stats

Bookmarked

Total users

Monthly active users

25 days ago

Last modified

Sitemap Detector

coder_zoro/sitemap-detector

Find sitemap URLs fast with our free Sitemap Finder tool. Instantly detect sitemaps from any website for SEO audits, indexing checks, and crawl planning. Improve visibility, site structure insights, and search engine performance in just seconds

Zoro

182

5.0

(3)

Sitemap Generator - Crawl Website & Create XML Sitemap

scrappy_garden/sitemap-generator

Generate an XML sitemap for any website. Crawls internal pages from start URLs (with depth + page limits), deduplicates URLs, and stores a ready-to-submit sitemap.xml plus a structured dataset and summary for SEO audits.

Bikram Adhikari

Sitemap URL Harvester

mahogany_songbird/sitemap-url-harvester

Collect URLs from XML sitemaps for SEO and crawling.

Britton Furness

Sitemap URL Extractor — recursive, filters, .gz

x402opklaar/sitemap-url-extractor

Extract every URL from any sitemap: recursive sitemap-index expansion, gzipped sitemaps, robots.txt auto-discovery, include/exclude filters, lastmod output. Feed the URL list to any crawler or audit. Pay per sitemap file processed.

Opklaar

Trademark Search: USPTO, EUIPO & UK IPO in One Call

everyotherfriday/trademark-search

Search US, EU and UK trademarks by mark or owner, with status, Nice classes and goods/services. Sources: TMview (EUIPN), www.tmdn.org/tmview, and the United States Patent and Trademark Office, www.uspto.gov.

Paul Vasquez

Website URL Crawler & Link Extractor

maximedupre/website-url-crawler

Crawl public websites and collect URLs from rendered navigation and sitemaps. Export a link map with source pages, depth, anchor text, HTTP facts, crawl status, and available sitemap metadata.

Maxime Dupré

Sitemap URL Discovery (sitemap.xml + robots.txt → all URLs)

gochujang/sitemap-url-discovery

Given a domain, finds sitemap.xml / sitemap_index.xml (also via robots.txt), recursively expands sitemap indexes, returns one row per discovered URL with lastmod / changefreq / priority. SEO audits, crawl-target prep, content cataloging. $0.0001/URL + $0.01 site fee.

Hojun Lee

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Sitemap URL Extractor

gifted_wagon/sitemap-url-inventory

Traverse public sitemap indexes into a deduplicated URL inventory with source sitemap, lastmod, changefreq, priority, media counts, and partial-failure evidence.

Michael Olmos

App Store Scraper: App Details, Reviews, Search & Top Charts

everyotherfriday/app-store

Everything public about an iOS app in clean rows: ratings, price, version history notes, screenshots, developer and genre data, plus up to 500 recent reviews per country storefront and chart positions. Loop countries for global review coverage. Built for ASO, competitor tracking and app research.

Paul Vasquez