Robots.txt & Sitemap Analyzer - Crawl Rules and URL Audit

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Generate an XML sitemap for any website. Crawls internal pages from start URLs (with depth + page limits), deduplicates URLs, and stores a ready-to-submit sitemap.xml plus a structured dataset and summary for SEO audits.

Firecrawl Site Mapper

alizarin_refrigerator-owner/firecrawl-site-mapper

Fast URL Discovery for Site Audits & Competitor Analysis Discover all URLs on a website using Firecrawl's Map endpoint. Perfect for competitor analysis, site audits, and content gap discovery.

HTTP Status Codes and URL Checker

antonio_espresso/website-status-code-crawler

A HTTP Status Codes Crawler is a tool that scans a website and retrieves HTTP status codes for each page. This helps in diagnosing errors and optimizing technical SEO.

Answer Engine Optimization & GEO Audit - ChatGPT Fix Plan

tidytools/geo-readiness-audit

AEO with a fix plan, not just a score: for each buyer question, see which page ChatGPT, Perplexity or Google AI Overviews would quote, why competitors win, and what to add. Plus a GEO audit of AI crawler access and firewalls. $0.01/question, $0.03/site.

Check URL redirect chains, final URLs, status codes, loops and too-long redirects in bulk.

llms.txt Generator & Checker - llms-full.txt, AI SEO (GEO)

tidytools/llms-txt-generator

Generate llms.txt and llms-full.txt for any website from its sitemap or a crawl, with sections and AI descriptions, or validate existing llms.txt files in bulk. $2 per 1,000 pages or sites.

Metaworld SEO Meta Auditor

metaworldsystems/metaworld-seo-auditor

Website URLs in — on-page SEO report out: title, meta description, H1s, canonical, robots, Open Graph & Twitter cards, word count, and a plain-English list of issues per page. No API keys.

Sitemap Analyzer — Recursive Parse, Health Check, AI Tags

and_krm/sitemap-analyzer

Parse any sitemap.xml recursively, extract all URLs with metadata, check HTTP health status, and optionally cluster URLs by topic using Claude AI. Perfect for SEO audits and site migration.

Audit a website's robots.txt directives and sitemap discovery in one fast API call. Detect blocked paths, missing sitemaps, and crawlability issues. Built for SEO agencies, technical SEO consultants, and site migration QA teams.