Website Crawl Map & Sitemap Diff

Sitemap Generator — XML Sitemap Crawler

junipr/sitemap-generator

Crawl a website, discover indexable URLs, and generate XML sitemaps with crawl diagnostics, canonical checks, status codes, and exportable URL inventory.

Website Link Graph & Outbound Links Crawler

logiover/website-link-graph-crawler

Extract all links from a website to CSV/JSON. Maps internal & outbound link graph with anchor text + nofollow/rel flags. No API, no login.

Turn any website into a clean list of its URLs. Finds the sitemap from robots.txt, follows nested sitemap indexes, unpacks .xml.gz, reads RSS/Atom feeds, and can return only pages changed since a date — the cheap way to re-crawl just what is new.

Image Scraper - Download All Images From Site

logiover/website-image-media-extractor

Scrape all images from a website without API or login. Bulk image & media URL extractor with alt text; export to CSV/JSON for AI datasets.

SEO & Technical Site Audit

berkaydev/seo-audit-tool

Combined technical SEO audit: broken links, redirect chains, on-page checks, sitemap/robots health. Client-ready HTML report + structured JSON.

Crawl any website and extract clean, LLM-ready Markdown for RAG pipelines, vector databases and AI agents. Boilerplate, navigation and cookie banners removed. No browser, no login.

Sitemap & URL Discovery - Find All URLs on Any Site

santamaria-automations/sitemap-url-discovery

Discover every URL on any website by parsing sitemap.xml, robots.txt, and sitemap indexes. Extract URLs with last modified dates, change frequency, and priority. Perfect for SEO audits, content analysis, crawling preparation, and site mapping.

🗺️ Sitemap Scraper & Analyzer

taroyamada/sitemap-analyzer

Extract thousands of URLs from complex nested XML sitemaps to audit website structure, HTTP status codes, and indexability for technical SEO.

Sitemap to URL Crawler — Extract Sitemap.xml URLs

logiover/sitemap-to-url-crawler

Extract all URLs from any sitemap.xml recursively. Export sitemap URLs to CSV/JSON for RAG pipelines, SEO audits, and LLM training datasets.

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.

Sitemap Url Extractor

scrapers-hub/sitemap-url-extractor

Sitemap URL extractor to extract all URLs from XML sitemaps quickly and efficiently 🌐📄 Ideal for SEO audits, site analysis, and indexing workflows. Fast, accurate, and easy to use.