Pricing

from $250.00 / 1,000 report generateds

AI Visibility Site Audit — robots.txt & llms.txt checks

Evidence-based audit of your own website: which AI crawlers robots.txt lets in (RFC 9309, 21 user-agent and control tokens, checked against the operators' own docs), which pages are discoverable, and each page's technical readiness — a ticket-grade fix plan with honest coverage disclosure.

Pricing

from $250.00 / 1,000 report generateds

Rating

0.0

(0)

Developer

Ken Yao

Actor stats

Bookmarked

Total users

Monthly active users

7 days ago

Last modified

Categories

Developer tools

Automation

Agents

Your website URL

startUrl

Required

The site you own or manage. A bare domain (example.com) is accepted — https is assumed. The audit crawls this site only, stays on the same hostname, and respects robots.txt. No other input is needed.

Type:string

Default:https://www.example.com

Example:

https://www.example.com

Deep-inspection request budget

maxPages

Optional

Budget of crawl REQUESTS for deep inspection. Every request consumes budget — including ones that fail or return non-HTML — so on a rate-limiting site fewer pages than the budget may be inspected; the report states both numbers. Only successfully inspected pages are billed (one Page audited event each, $0.02); failed requests are not billed. At the default 4 GB memory a run totals about $0.79 for 25 inspected pages, $10.29 for 500, $40.29 for 2000 — that is $0.04 to start (Actor Start is charged once per GB of memory, so this part changes if you change the memory setting), $0.25 for the report, and $0.02 per inspected page. Sitemap enumeration and liveness checks run independently of this budget (up to 100 sitemap files / 5,000 URLs, disclosed when capped) and are not billed per page.

Type:integer

Minimum:1

Maximum:2000

Default:25

Faster crawl (it's my own server)

fastCrawl

Optional

Raises crawl concurrency from 5 to 15. Only enable for a server you own and can afford to load. Honest note: if your server rate-limits crawlers, it sets the real pace regardless of this setting. A preliminary report (bot access, site files, early findings) is published within the first minutes either way; deep inspection of large sites can take tens of minutes.

Type:boolean

Default:false

Additional URLs you own (optional)

additionalUrls

Optional

Pages that are not linked from your site and not in your sitemap (e.g. ad landing pages from Search Console or your ad account). Same hostname only. Without this, unlinked pages are invisible to the audit — exactly as they are to AI crawlers.

Type:array

Default:

[]

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers

aiqlabs/seo-ai-visibility-auditor

Crawls a site from one starting URL and reports how visible it is to AI: which AI crawlers robots.txt admits, whether llms.txt exists, and what structured data and metadata each page offers. Give it a domain to crawl - for a fixed list of pages, use SEO Audit Tool instead.

Ai-Q Labs

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

Website Tech Audit Scraper: robots.txt, Sitemaps & Meta

arman-bd/website-tech-audit-scraper

Audit any site's public technical files: robots.txt rules, sitemap discovery and URL counts, ads.txt sellers, security.txt contacts, meta tags, structured data and the detected tech stack with evidence.

Arman Hossain

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Tim Zinin

Agent Readiness Audit

ondrejulehla/agent-readiness-audit

Check how ready a website is for AI agents: robots.txt policy for AI crawlers, llms.txt, sitemap, structured data, MCP server card, and JavaScript dependence.

Ondřej Úlehla

AI Crawler Access Auditor

kingii98/ai-crawler-access-auditor

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

kingii98

AI SEO Tracker - Technical & AI Search Audit

khadinakbar/ai-seo-tracker

Audit a public site for technical SEO and AI-search readiness. Returns source-linked checks for robots.txt, sitemap, llms.txt, indexing directives, metadata, canonical URLs and structured data. Use for site audits and repeatable monitoring; not for AI-answer brand visibility.

Khadin Akbar

llms.txt Generator: Make Any Website AI-Readable

apifmcpfactory/llms-txt-generator

Generates a spec-compliant llms.txt file for any website from the site's own pages and metadata (no AI invention) - the fix when an AI-readiness audit finds llms.txt missing; $0.02 per site.

Howth Technology Factory

Is Your Site AI-Ready? GEO & llms.txt Audit + Fix List

wheaten_steelpan/geo-site-auditor

Audit your website for Generative Engine Optimization (GEO). Crawls pages, tests robots.txt/llms.txt/schema.org, evaluates AI answerability, and generates ranked fixes + visual report.