Pricing

$10.00 / 1,000 domain audits

Go to Apify Store

AI Crawler Readiness Auditor

Try for free

Audit domains for llms.txt, robots.txt AI bot rules, sitemap hints, and public AI crawler readiness signals.

Pricing

$10.00 / 1,000 domain audits

Rating

0.0

(0)

Developer

jaehan byun

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

What it does

This Actor helps SEO, developer relations, documentation, and content teams check whether a site is easy for AI crawlers and agentic tools to discover. For each domain, it fetches public files only and returns a compact readiness score with concrete remediation notes.

It does not log in, bypass access controls, use residential proxies, or crawl private content.

Input

domains: Domains or URLs to audit.
maxDomains: Safety cap for one run.
requestTimeoutSecs: Timeout for each public HTTP request.
checkHomepage: Whether to fetch the homepage and look for discovery hints.

Example:

{
  "domains": ["apify.com", "docs.apify.com", "example.com"],
  "maxDomains": 25,
  "requestTimeoutSecs": 12,
  "checkHomepage": true
}

Output

Each dataset item represents one audited domain.

Important fields:

readinessScore: 0-100 score based on public AI discoverability signals.
llmsTxt: status, URL, line count, byte size, and extracted links for /llms.txt.
llmsFullTxt: status and size for /llms-full.txt.
robots: robots.txt status plus AI bot allow/disallow findings.
sitemap: sitemap hints from robots.txt, homepage HTML, and /sitemap.xml.
issues: concrete problems to fix.
recommendations: actionable next steps.

Example output item:

{
  "domain": "example.com",
  "origin": "https://example.com",
  "readinessScore": 28,
  "grade": "D",
  "llmsTxt": { "exists": false, "statusCode": 404 },
  "robots": { "exists": true, "aiBotRules": [] },
  "sitemap": { "found": false, "urls": [] },
  "issues": ["Missing /llms.txt", "No sitemap hints found"],
  "recommendations": ["Add /llms.txt with key documentation and product URLs"]
}

Pricing

When pay-per-event monetization is enabled, this Actor is intended to charge one domain-audit event per audited domain. It checks the user spending limit before auditing each domain.

Limitations

The Actor checks public files and homepage HTML only; it is not a full-site crawler.
robots.txt interpretation is conservative and focused on AI-related user-agent names.
A high score means good public discovery hygiene, not guaranteed ranking in AI products.

GEO & llms.txt Readiness Auditor

gp005/geo-llms-auditor

Audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot, and 30+ more) your site allows or blocks, validate your llms.txt, and score AI-citability — with an exact prioritized fix list. Non-scraper, reads public well-known files.

Geo

AI Readiness Auditor

rationalistic_counsel/ai-readiness-auditor

Check how AI-ready any website is. Get an AI Readiness Score (0-100) checking llms.txt, robots.txt AI crawler directives, Schema.org structured data, and meta tags. No API key needed.

J N

LLMs.txt Discovery Auditor

phoenix2810/llms-txt-discovery-auditor

Audit a website for /llms.txt and /llms-full.txt AI discovery readiness. Get a readiness score, markdown structure summary, and actionable recommendations. Built for GEO/AI-SEO consultants, documentation teams, and devrel.

Sanskar Jaiswal

GEO Auditor — AI Search Readiness & Citability Audit

foxlabs/geo-auditor

Audit how ready your site is to be found, read & cited by AI search (ChatGPT, Perplexity, Gemini, Claude). Checks AI-crawler access, structured data, content extractability, speed & trust — scored 0-100 with a prioritized fix list. GEO / AEO technical audit.

Berkan Kaplan

Lead Quality & Outreach Readiness Auditor — Decide & Benchmark

ryanclinton/enrichment-quality-auditor

Audit lead lists before outreach. Every lead gets a quality score, trust score, and a use/verify/repair/enrich/reject decision, then rolls up to account health, pipeline risk, and a vendor benchmark of which source delivers usable leads. Works with Clay, Apollo, ZoomInfo, CSV.

Ryan Clinton

Shopify Agentic Commerce Readiness Auditor

automation-lab/shopify-agentic-commerce-readiness-auditor

Audit Shopify catalog, product schema, pricing, inventory, shipping, and returns signals for AI shopping agent readiness scorecards.

Stas Persiianenko

GEO Competitive Benchmark — AI Search Readiness vs Competitors

foxlabs/geo-benchmark

Benchmark your site vs up to 5 competitors for AI-search readiness in one run. See where rivals out-rank you for ChatGPT, Perplexity, Gemini & Claude citation — AI-crawler access, schema, extractability, speed, trust — as a ranked scoreboard + per-signal gap list. Deterministic GEO/AEO.

Berkan Kaplan

AI Website Sales Readiness Auditor

orbitai/ai-website-sales-readiness-auditor

Audits local business websites for conversion readiness, lead-capture gaps, trust signals, and sales opportunity scoring.

Daniel Lozano

Structured Data Validator (JSON-LD / OG)

jungle_synthesizer/structured-data-validator-pro

Extract and validate structured data from any URL: JSON-LD, Open Graph, Twitter Cards, microdata, RDFa, meta tags. Local schema.org validation. Flags Google rich-result eligibility and AI-discovery readiness. Pure HTTP. Built for SEO audits and structured-data debugging at scale.

BowTiedRaccoon

Programmatic SEO Opportunity Mapper

monty15/programmatic-seo-opportunity-mapper

Turn competitor sitemaps into a scored, prioritized programmatic-SEO opportunity matrix — deterministic title/meta/slug suggestions, schema type, and optional AI-citability scoring. No API key required.

Monty

Shopify Scraper

automation-lab/shopify-scraper

Monitor any Shopify store for competitor catalog, price, availability, variants, collections, and review data. Extract DTC storefront intelligence with no API key.

Stas Persiianenko