Pricing

$0.30 / 1,000 url discovereds

Sitemap & robots.txt URL Discovery (no crawling)

Feed it site roots, get back URLs from their sitemaps plus each site's disallow rules and crawl-delay. Streams sitemaps, follows nested indexes and handles gzip without crawling page bodies. For seeding crawlers, SEO audits and migration checks.

Pricing

$0.30 / 1,000 url discovereds

Rating

0.0

(0)

Developer

Paul Vasquez

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Sitemap & On-Page Crawl

timbered_oak/sitemap-and-onpage-crawl

Discovers a site's sitemap via robots.txt, crawls up to maxPages pages with CheerioCrawler, and returns per-page SEO/on-page metrics.

Mark

Sitemap URL Extractor — robots.txt, Sitemap Index, gzip

chorelet/sitemap-url-extractor

Every URL of a website from its sitemaps: robots.txt discovery, sitemap indexes, gzip, lastmod, changefreq, priority, hreflang alternates, image/video/news extensions. Include/exclude patterns and date filter, CSV/JSON export and API.

Chorelet

URL & Sitemap Link Extractor: Site Map, Health Checks & Monitor

getascraper/url-link-extractor

Extract every URL from a site's sitemap or a page, no sitemap link needed. Auto-discovers via robots.txt, follows sitemap-index nesting, keeps lastmod, changefreq, and priority. Adds a site-structure summary, link-health checks, and change monitoring. Export as JSON, CSV, or Excel. $0.00001 per URL.

GetAScraper

Sitemap URL Extractor

lost.fany/sitemap-url-extractor

Extract every URL from a website's sitemap.xml, sitemap index and robots.txt in seconds — with lastmod, changefreq, priority and hreflang. No crawling, no browser: fast, cheap, reliable.

Natalia Ivanova

Robots.txt Auditor | Crawl Rules & Sitemap Discovery

sbio-apps/robots-txt-auditor

Read robots.txt for supplied websites and export user-agent groups, allow/disallow rules, sitemap declarations, extension directives and parsing warnings. Identify missing files and compare content hashes in your crawl configuration workflows.

Austin Aryain

Sitemap XML URL Extractor

automation-lab/sitemap-xml-url-extractor

Extract deduplicated URLs and metadata from public XML sitemaps and nested sitemap indexes for crawl planning, migrations, and content audits.

Automation Lab

Sitemap URL Extractor: robots.txt, gzip, Changes

succinct_glider/sitemap-urls

Extract every URL from a website's XML sitemaps: found through robots.txt, sitemap indexes and .gz files followed, lastmod and date filters, URL patterns, and new, changed or removed URLs since your last run. Reads sitemap files only, never the pages.

Alexandre Bobichon

Screenshot & PDF Renderer: Full Page, Mobile & Dark Mode

everyotherfriday/screenshot-renderer

Capture URLs as PNG, JPEG, WebP or PDF with Chromium. Choose full-page or element captures, desktop/mobile/tablet sizes, dark mode, ad blocking and cookie-banner hiding. Export stored files and structured results; failed captures are not charged.

Paul Vasquez

Sitemap URL Extractor

alaudinburki/sitemap-url-extractor

Extract every URL from a site's sitemaps — follows sitemap-index files, reads robots.txt to discover them, decompresses .gz sitemaps, and returns lastmod/changefreq/priority. Feeds any other scraper. Pure logic.

alaudin burki

Bulk URL Status Checker

palgenius/bulk-url-status-checker

Check thousands of URLs for broken links: status code, full redirect chain, response time, content type and size. Chains straight off the Sitemap URL Extractor.