Drishti.Anantasutra: AI Search Visibility Audit (GEO)

Drishti (दृष्टि) is Sanskrit for "sight". This Actor shows you how your website looks to AI search engines and AI agents: which AI crawlers your robots.txt lets in, whether your pages can be read without JavaScript, whether you publish an llms.txt, and what to fix first. It is part of the Anantasutra family of tools.

You give it one website or a list of up to 1,000 URLs. For each one you get a transparent 0-100 score, a per-crawler access matrix for 28 AI crawlers (ChatGPT, Claude, Perplexity, Google, Bing, Apple, Meta, Amazon, Mistral and others), and a prioritised list of fixes with copy-paste snippets.

Who it is for

• Site owners and marketers who want to know whether ChatGPT search, Perplexity or Google AI Overviews can read and cite their site.
• SEO and GEO agencies that audit many client domains and need a consistent, explainable score per domain.
• AI agents and developers that need a machine-readable answer to "may AI crawlers access this page?" (it runs well from Claude, ChatGPT or Cursor through the Apify MCP server, see below).

What it checks

Area	What is checked	Points
AI crawler access	robots.txt verdict at / for 10 AI search crawlers and 7 user-triggered agents (RFC 9309 matching, longest rule wins)	30
robots.txt health	reachable, well formed, no rules outside groups	5
Indexability and snippets	HTTP status, noindex, nosnippet, canonical, redirects, response time	20
Content without JavaScript	main text present in the raw HTML, one H1, text-to-HTML ratio	15
Metadata and structured data	title, meta description, JSON-LD, OpenGraph, lang	15
LLM-specific files	/llms.txt present and well formed, /llms-full.txt, Markdown alternates	10
Sitemap	declared in robots.txt or found at /sitemap.xml, has URLs	5

Blocking AI training crawlers (GPTBot, ClaudeBot, CCBot ...) is a business decision, so it is reported but never lowers the score. Checks that could not run (for example because your robots.txt disallows this auditor) are skipped and excluded from the score; scoreCoverage tells you how many of the 100 points were actually evaluated.

Input

{
  "urls": ["https://example.com", "example.org/pricing"],
  "pathsToCheck": ["/blog/", "/docs/"],
  "checkLlmsFullTxt": true,
  "includeBotDetails": true,
  "maxConcurrency": 3,
  "requestTimeoutSecs": 15
}

• urls: domains or full page URLs. For a domain the home page is audited; robots.txt, llms.txt and the sitemap are always checked at the site root.
• pathsToCheck: extra paths evaluated against robots.txt for every crawler (not fetched).
• requestTimeoutSecs: raise it for slow sites.

Output

One dataset row per input URL. Abridged example:

{
  "url": "https://www.example.com/",
  "status": "ok",
  "score": 100,
  "grade": "A",
  "scoreCoverage": 100,
  "summary": "AI visibility score 100/100 (A). AI search crawlers allowed: 10/10; user-triggered agents allowed: 7/7; training crawlers blocked: 2/10. llms.txt: valid. Main content without JavaScript: ok.",
  "searchBotsAllowed": "10/10",
  "userBotsAllowed": "7/7",
  "trainingBotsBlocked": "2/10",
  "llmsTxtStatus": "valid",
  "topRecommendation": "No critical issues found",
  "blockedAiCrawlers": ["GPTBot", "CCBot"]
}

Every row also contains recommendations (priority, area, title, detail, fix snippet), scoreBreakdown (every check with points and the reason), aiCrawlers (per crawler: matched robots.txt group, deciding rule, link to the operator's documentation), robotsTxt, llmsTxt, llmsFullTxt, sitemap and page. The Recommendations view of the dataset lists one fix per row, highest priority first.

Results without a score

A failure to reach a site is not a verdict on its AI visibility, so these rows have score: null and grade: null instead of a misleading 0/100 "F":

Situation	status	Charged?
No answer at all: timeout (retried once), DNS error, invalid HTTPS certificate, connection refused	error	No
The address is not a public website (private network, localhost, cloud metadata)	error	No
The URL is a PDF, image, JSON API or other non-HTML file	error	No
Invalid input, for example 12345, an e-mail address such as info@example.com, two addresses in one entry, or an object without a url field	error	No
The same address appears again in the input (example.com, EXAMPLE.com/, example.com#top ...)	duplicate	No
robots.txt answers HTTP 5xx or 429, or the page cannot be fetched or answers this audit with HTTP 403 (bot protection), 429 (throttling) or 5xx	ok	Yes

In the last case the site did answer, so you still get the robots.txt result, the AI crawler matrix, llms.txt and sitemap checks. Under RFC 9309 a robots.txt that answers 5xx or 429 means "disallow everything" for compliant crawlers while it lasts, so re-run later and fix /robots.txt if it persists. A page that answers this audit with 403, 429 or 5xx is not graded either: AI crawlers may get a different answer, so check your firewall or bot-protection rules and re-run instead of reading it as a failing page. A page that does not exist (404, 410) is still scored.

Pricing

Pay per event, no subscription. You only pay for audits of sites that answered.

Event	Price	When
URL audited	$0.004	Once per unique address whose site answered (see "Results without a score" for what is never charged)
Actor start	$0.00005	Charged automatically by Apify at the start of each run (per GB of memory)

Auditing 1,000 domains costs about $4. Set "Maximum cost per run" in the run options to cap spending: the Actor audits as many URLs as fit and tells you how many were skipped. Duplicates, unreachable sites, non-HTML URLs and invalid inputs are never charged.

Use it from Claude, ChatGPT or Cursor (MCP)

The Apify MCP server exposes this Actor as a tool to AI assistants.

• Claude Desktop / Claude Code, Cursor and other MCP clients: add the hosted server https://mcp.apify.com/?actors=motivating_bracelet/drishti-anantasutra and sign in with your Apify account, or run it locally with npx -y @apify/actors-mcp-server --actors motivating_bracelet/drishti-anantasutra and your Apify API token in the APIFY_TOKEN environment variable.
• ChatGPT: add https://mcp.apify.com as a custom connector (MCP) where your plan allows it.

Then ask, for example: "Audit example.com with Drishti and list the three most important fixes." The assistant passes urls as input and reads the summary, topRecommendation and recommendations fields.

Limits

• Up to 1,000 URLs per run, 10 extra robots.txt paths, 10 sites in parallel; requests to one site are sequential.
• Raw HTML only: the page is not rendered in a browser, which is how most AI crawlers read it too.
• Page downloads are capped at 3 MB, robots.txt at 500 KiB (as RFC 9309 allows), llms-full.txt at 64 KB.
• Only public websites over http or https. Private, loopback, link-local and other reserved addresses are refused, including through DNS names and redirects.
• The auditor identifies itself honestly in its User-Agent (AIVisibilityAudit) and obeys your robots.txt for itself. If your firewall or bot protection drops or refuses unknown bots (timeout, HTTP 403 or 429), the result says it could not be checked instead of giving a grade.
• robots.txt is a policy, not a lock: the matrix shows what compliant crawlers may do. The crawler catalog was last checked against operator documentation on the date in botsCatalogCheckedAt.

FAQ

Why is my score null instead of a number? The site or page did not answer (timeout, DNS, certificate error), robots.txt answered with a server error, the page answered this audit with a block (403), throttling (429) or a server error (5xx), or the URL is not an HTML page. The summary and topRecommendation say which. Rows without an HTTP answer are not charged.

Does blocking GPTBot hurt my score? No. Training crawlers are reported for information only. Blocking AI search crawlers such as OAI-SearchBot or PerplexityBot does lower the score, because it removes you from AI answers.

I entered the same domain several times. Am I charged several times? No. Entries that normalise to the same address are audited and charged once; the repeats get a free duplicate row pointing to the first one. Note that http:// and https://, or www. and bare-domain forms (www.example.com and example.com), are different sites with their own robots.txt, so each form you enter is audited and charged. The run log lists such pairs before the audit starts, so keep only the forms you need.

Can I audit a PDF or an API endpoint? The Actor tells you it is not a web page and does not charge for it. Audit the HTML page that links to it instead.

Does it log in, submit forms or bypass bot protection? No. It reads public pages the way a polite crawler does.

How do I fix the issues? Each recommendation has a fix field with a snippet (robots.txt lines, meta tags, an llms.txt template). To generate an llms.txt for your site, see the sibling Actor Suchi.Anantasutra.