AI Visibility Site Audit

Find out — in one run — which AI crawlers your robots.txt actually lets in, which of your pages they can discover, and what technical signals each page carries. With evidence for every claim and an acceptance test for every fix.

AI assistants are becoming a real discovery channel, but their crawlers read your site differently than you might expect, and most sites have never checked what those bots are allowed to access. This Actor audits your own site and returns an evidence-based report plus a ticket-grade fix plan.

What it checks

1. AI crawler access — your robots.txt evaluated per RFC 9309 (merged groups, longest match, Allow wins ties, * and $ wildcards) for 21 user-agent and control tokens across OpenAI, Anthropic, Perplexity, Google, Apple, Amazon, Meta, Common Crawl, ByteDance — including search/retrieval crawlers, user-triggered fetchers, training-control tokens, and legacy tokens, each shown with the exact rule text that applies. Every registry entry that has an operator-published documentation page — 20 of the 21 — is checked against that page at run time (token presence check), with source URL and check timestamp printed. The exception is named rather than hidden: ByteDance publishes no crawler documentation, so Bytespider carries no source URL and is labelled "no official page known" in both the report and the public registry. The registry this audit runs against is published, and readable without an account, at https://api.apify.com/v2/key-value-stores/Ac9Ouq5r9eQrcUGXj/records/registry.html — every entry with its type, whether it counts toward the access score, its operator's own documentation page, and the date that entry was last verified.
2. Discoverability — sitemap enumeration (Sitemap: directives in robots.txt honored, nested sitemap indexes followed, up to 100 sitemap files / 5,000 URLs — every cap is disclosed in the report when hit), cross-checked both ways against reality: declared-but-dead URLs (GET-confirmed 404/410 only) and reachable-but-undeclared pages.
3. Technical page readiness — per-page: parseable JSON-LD with a declared @type (a syntax-level check, not schema.org validation; unparseable blocks count as broken, @graph expanded), titles & meta descriptions (heuristic ranges, labeled as such), canonical tags (URL-consolidation signal), noindex via meta and X-Robots-Tag, thin raw-HTML content (CJK-aware count).
4. llms.txt — present or not (optional enhancement, not scored), plus a starter draft built from your actual pages.

What you get

• REPORT (Markdown) and REPORT_HTML (readable by non-developers): Access score, per-check pass rates, coverage & confidence disclosure, findings grouped by URL family, and a ticket-grade fix plan — every ticket has evidence, affected counts, sample URLs, the requested change, an acceptance test, and its risk.
• Progressive delivery: a preliminary report (crawler access + site files + early findings) is published within the first minutes and updated as the crawl runs; the Actor's status line shows phase, progress and ETA. Close the tab — the run finishes in the cloud.
• Dataset: one row per inspected page, streamed live during the run.
• OUTPUT: the full structured result for automation and AI agents.

Honest notes on scoring

There is no single overall score — a composite would hide exactly what matters. The Access score counts search & retrieval crawlers only; allowing or blocking training bots is a business choice we record, not score. Readiness pass rates are transparent per-check percentages; their mean is labeled an uncalibrated heuristic. Coverage and confidence are always disclosed: this audit tells you what it saw, what it could not see, and why.

Input

Field	Description
startUrl	Your website. The crawl starts at exactly this URL (path included) and stays on its canonical hostname only — an apex→www style redirect is followed once at the start and disclosed in the report. URLs whose query string carries a credential-looking parameter (token, key, signature, password, …) are refused at input: audit outputs embed URLs, and a signed URL would end up in your report.
maxPages	Deep-inspection request budget (default 25, max 2000): every crawl request — including ones that fail or return non-HTML — consumes budget, so on a rate-limiting site fewer pages than the budget may be inspected (the report states both numbers). Each successfully inspected page is one billed event; failed requests, enumeration and liveness checks are not billed.
additionalUrls	Optional: pages not linked and not in your sitemap (e.g. ad landing pages) — otherwise invisible to this audit, exactly as they are to crawlers. Entries with credential-looking query parameters are refused and disclosed (values never stored).
fastCrawl	Optional: raise crawl concurrency for a server you own.

More honest notes

• llms.txt is an emerging convention. Google has stated it does not affect ranking; adoption is uneven. We report it as low-cost hygiene, not a growth lever.
• Static analysis. Raw HTML without JavaScript rendering — which is how many crawlers read pages. JS-dependent pages are flagged, not rendered.
• Unanswered is not dead — and blocked is not dead either. Only GET-confirmed 404/410 is reported as confirmed dead. 401/403/429 are reported as access-blocked (often the site's own WAF blocking the audit), 5xx as temporarily unavailable, and silence as unverifiable — none of these ever triggers a deletion prescription. If your server rate-limits the audit, the report says so and tells you how to get a complete re-run.
• Blocking AI bots can be the right choice. The audit makes your current state visible so it is a decision, not an accident.
• Credential screening is name-based, and says so. Inputs with credential-looking query parameters are refused; in everything this audit stores (reports, dataset, OUTPUT) the values of known secret-named query parameters found on discovered URLs are masked — including percent-encoded names and URLs nested inside query values. Values that cannot be fully analyzed — over ~4096 characters, still percent-decoding at the decode cap, malformed percent-encoding, or URL-shaped at the nesting-depth cap — are masked wholesale as unstorable — that is a storage-safety refusal, not an assertion the content was a credential. Only known names are caught — a secret under an unrecognized name passes through, so treat any URL that embeds credentials as exposed wherever it is linked or listed in a sitemap.
• The Actor audits only the site you provide and stays on its canonical hostname: every request it makes — pages, sitemaps, liveness probes, and every redirect hop in between — is checked against your robots.txt and the hostname pin BEFORE it is fetched (two disclosed exceptions: the very first GET of the exact URL you provided, needed to learn which host's robots.txt governs; and the /robots.txt fetch itself, which per RFC 9309 follows redirects even to another host — every hop still passing the public-IP safety guards, with the final robots URL and redirect chain recorded in OUTPUT); anything refused is named in the report, never silently skipped. In addition it fetches only the allowlisted official crawler-documentation pages (following their redirects) for the registry check.

Pricing

Pay per event: a small start fee, a per-page fee for each deep-inspected page, and a report fee. Control cost with maxPages — a 25-page audit costs well under a dollar.