AI Crawler Access Audit: GPTBot, ClaudeBot, PerplexityBot (GEO)

Scans websites the way AI search crawlers see them. Checks HTTPS, robots.txt for OAI-SearchBot/Claude-SearchBot/PerplexityBot, JSON-LD, llms.txt, content without JS, Shopify readiness, and more. Outputs a score, pass/fail checks, and fix hints.

Audit any list of web pages for on-page SEO problems: title, meta description, headings, images, canonical, noindex, mobile tag, Open Graph, structured data and broken links. Score, grade and fix list per page. $0.005 per page.

Free beta: compare permitted public product pages with supplied SKU, price, currency and availability. Static JSON-LD only; ambiguous evidence stays inconclusive.

SEO audit for any page or website, plus AI search readiness: titles, canonicals, schema, llms.txt, and whether GPTBot, ClaudeBot, and PerplexityBot can read the site.

Compare two supplied sitemap XML snapshots for added or removed URLs and changed lastmod text. Audit duplicates and malformed entries. Handles urlsets or sitemap indexes; no network fetch or page-content claims.

Bulk URL Status, Redirect & SSL Checker

pipwort/url-status-checker

Check thousands of URLs at once: final HTTP status, every redirect hop, response time, canonical and robots tags, and SSL certificate expiry. Built for site migrations, link audits and monitoring.

Broken Link Checker & Bulk URL Status Code Checker

kilndata/broken-link-checker

Check the HTTP status of thousands of URLs, or crawl a site and test every link on it. Status code, full redirect chain, final URL, response time, error type (DNS, SSL, timeout, 4xx, 5xx) and the page each broken link sits on. HEAD with GET fallback, polite per-domain limits.

Crawl any website and get a score and issue list for every page, broken links, duplicate titles, and an AI-search check (llms.txt, AI crawler access in robots.txt, structured data), all ranked into one prioritized fix list.

RAG Web Browser (Anti-Bot) - $1.80 / 1000 Pages

scrapeunblocker/rag-web-browser

Search the web and get the top results as clean Markdown, or read any URL, in seconds. Built as a web tool for LLMs, AI agents and RAG. Pages behind Cloudflare, DataDome, PerimeterX or Akamai work. No Google needed. $1.80 per 1000 pages; failed pages are free.

Audit Arabic and multilingual HTML pages for metadata, language, RTL direction, canonical and hreflang signals. Export actionable Arabic reports plus JSON and CSV. Respects robots.txt.