Robots Rule Tester - Would Google Crawl This URL?

Fetch and parse robots.txt for any site: user-agent rules, crawl-delay, and declared sitemaps. Optionally test whether specific URLs are allowed for a given user-agent, using correct longest-match rules.

Meta Robots Auditor - Is This Page Actually Indexable?

eliai/webpage-meta-robots-auditor

Indexability verdict per URL from ALL the signals: meta robots, meta googlebot, the X-Robots-Tag header (the one checkers miss), canonical + self-canonical check — with the exact reasons. Catches shipped staging noindex and CDN-injected headers. $0.002 per page vs $0.007-$0.011 measured incumbents.

Validate robots.txt for one or more websites: fetches /robots.txt per host, parses directive groups (User-agent/Allow/Disallow/Crawl-delay/Sitemap), reports common errors and warnings, and can test URLs against the chosen User-Agent.

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Parse robots.txt for up to 100 sites into one JSON record each: rule groups, sitemaps and crawl delays, plus an AI crawler policy audit across 17 bots like GPTBot, ClaudeBot and PerplexityBot. Each bot rated allowed, partial or blocked, with an open, selective or closed summary per site.

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

robots.txt Parser & AI Crawler Block Checker

taroyamada/robotstxt-ai-checker

Audit robots.txt files across thousands of websites to detect specific crawl policies, disallowed paths, and user-agents for GPTBot and ClaudeBot.

See exactly which AI crawlers a site allows. Per domain it reads robots.txt for 25 named AI agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot…), plus ai.txt, llms.txt, noai meta, X-Robots-Tag and tdmrep.json, and quotes the rule behind every answer. Alerts on policy changes.

Cron Expression Parser - Validate, Explain, Next Run Times

eliai/cron-expression-parser

Validate up to 50 cron expressions per run: plain-English description + timezone-aware next fire times (DST-correct, any IANA zone), 5/6-field, @shortcuts, names, dom/dow OR rule. $0.0008 per expression, invalid ones never charged — cheaper than measured incumbents ($0.001-$0.03).

Robots.txt Analyzer

sootesting/robots-txt-analyzer

Analyze robots.txt for any list of domains — crawl rules per user-agent, declared sitemaps, crawl-delay, and more. Outputs structured JSON. Pay-per-event.