Sitemap Extractor + robots.txt & llms.txt Checker for AI

Transcribe your audio and video files with Whisper: text, timestamped segments, optional word timings, language detection, translation to English, and SRT/VTT subtitle files. MP3, M4A, WAV, MP4, WebM and more from URLs or key-value stores. Priced per audio minute.

Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range.

Firecrawl Site Mapper

alizarin_refrigerator-owner/firecrawl-site-mapper

Fast URL Discovery for Site Audits & Competitor Analysis Discover all URLs on a website using Firecrawl's Map endpoint. Perfect for competitor analysis, site audits, and content gap discovery.

LLMs.txt File Generator

justa/llms-txt-file-generator

Generate an llms.txt file from a website sitemap. Crawls all URLs, extracts titles and meta descriptions, and creates a Markdown-formatted file following the llms.txt specification. Upload then the output of your file directly on your website (Webflow, Wordpress etc.)

Send batches of HTTP requests and get agent-ready results: parsed JSON, clean Markdown from HTML, retries with Retry-After, redirect chain, timing, TLS expiry, SSRF protection and redacted secrets. URL health mode for link checks. Priced per response.

XML Sitemap Finder & Extractor API

andok/find-sitemap-from-url

Find and extract all XML sitemaps for any domain. Automatically parses robots.txt, scans HTML tags, and recursively follows indexes. Perfect for SEO & web scraping.

GEO Site Audit - AI Readiness Checker

dltik/geo-site-audit

Audit your website for AI crawler accessibility: robots.txt (GPTBot, ClaudeBot, Perplexity), llms.txt, sitemap, Schema.org, meta tags, content extractability, TTFB. Get an AI-readiness score 0-100 with prioritized recommendations.

SEO Site Audit — Website SEO Checker & Crawler

saliga/seo-site-audit

Crawl a website and audit every page for technical and on-page SEO: a 0-100 score, issues with fix hints, broken internal links, duplicate titles, orphan pages, canonical and robots problems, plus a site summary. $3 per 1,000 pages.

llms.txt generator and AI search readiness auditor for ChatGPT Search, Perplexity, and Claude. Check /llms.txt, robots.txt AI crawlers, and JSON-LD schema.

Extract still frames from your videos: every N seconds, N evenly spaced, exact timestamps, keyframes or scene changes, plus a best-thumbnail pick and contact sheets. JPEG/PNG/WebP with timestamps and ffprobe metadata. URLs or key-value stores. Priced per frame.