Pricing

from $1.00 / 1,000 urls

Sitemap & llms.txt URL Extractor

Get every URL a website publishes. Reads robots.txt, follows all sitemaps including sitemap indexes and gzipped files, probes the usual sitemap paths when robots.txt names none, and reads llms.txt and llms-full.txt. Filter by pattern or last-modified date.

Pricing

from $1.00 / 1,000 urls

Rating

0.0

(0)

Developer

Martin CONTAL

Actor stats

Bookmarked

Total users

Monthly active users

20 hours ago

Last modified

Store/Crawler Surface Preflight

checksmithcats/store-crawler-surface-preflight

Cross-check public robots.txt, sitemap, noindex, llms.txt, policy links, and contact signals from a storefront URL. Reads only public URLs; no account access.

Checksmith Cats

Technical SEO Portfolio Regression Report

taroyamada/technical-seo-portfolio-regression-report

Try Input: 1 URL + 1 page, reports/exports OFF ≈ USD 0.012/row. From USD 12/1k. Paste a URL and Start.

naoki anzai

AI Crawler Checker & llms.txt Generator — Website AI Audit

dev-hoss/ai-search-readiness-audit

Check if AI search engines can crawl and cite your website: robots.txt rules for GPTBot, ClaudeBot, PerplexityBot and 17 more AI crawlers, llms.txt validation, sitemap, schema.org, meta tags and content extractability. 0-100 score, prioritized fixes, ready llms.txt.

Hossam Mohamed

AI SEO Tracker - Technical & AI Search Audit

khadinakbar/ai-seo-tracker

Audit a public site for technical SEO and AI-search readiness. Returns source-linked checks for robots.txt, sitemap, llms.txt, indexing directives, metadata, canonical URLs and structured data. Use for site audits and repeatable monitoring; not for AI-answer brand visibility.

Khadin Akbar

GEO / AEO Website Audit — AI Search Readiness

rtworule/ai-search-readiness-auditor

Audit public websites for GEO and AEO readiness: AI crawler access, robots.txt, llms.txt, sitemaps, Schema.org, social metadata, scores, and prioritized fixes.

Kunteper Koyu

🌐 Website Content Crawler

data_minds/website-content-crawler

Crawl any website into clean text, Markdown & rich metadata — AI-ready output for RAG, LLMs & vector databases. Streams to your dataset in real time.

Data Minds

AI Crawler Readiness Auditor

nectared_screen/ai-crawler-readiness-auditor

Audit domains for llms.txt, robots.txt AI bot rules, sitemap hints, and public AI crawler readiness signals.

jaehan byun

AI Crawler Policy Auditor

junipr/ai-crawler-policy-auditor

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

junipr

AI Crawler Policy Drift Monitor

checksmithcats/ai-crawler-policy-drift-monitor

Tracks changes in public robots.txt, noindex, sitemap, and llms.txt signals against a stored baseline. Public crawler-facing signals only.

Checksmith Cats

Sitemap URL Extractor - List All URLs in a Sitemap

dltik/sitemap-url-extractor

Extract every URL from any XML sitemap, with lastmod, changefreq and priority. Resolves sitemap indexes recursively. Pass a sitemap.xml or just a site root to auto-discover its sitemaps. Pure HTTP, no browser — fast and cheap.