GEO Website Auditor & AI Crawler Checker

LICENSE

Turn any public web page into a scored, actionable Generative Engine Optimization (GEO) report. This Actor checks whether major AI crawlers can retrieve the site, whether it publishes useful AI discovery files, and whether each page is structured so answer engines can understand and cite it. It produces explainable JSON results for SEO and content teams—without browser automation, logins, or proxy configuration.

Common use cases

• Check whether GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Applebot-Extended are allowed by robots.txt
• Audit a landing page before an AI-search or GEO campaign
• Find missing llms.txt, JSON-LD, canonical, sitemap, and indexability signals
• Audit exact URLs or automatically discover up to 100 same-origin pages from XML sitemaps and robots.txt directives
• Export GEO scores and recommendations to a spreadsheet, dashboard, API, or MCP workflow
• Compare the answer readiness of product, documentation, and editorial pages

What you get

• A 0–100 GEO readiness score and A–F grade for every URL
• robots.txt analysis for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Applebot-Extended
• Detection of llms.txt, llms-full.txt, and sitemap.xml
• Optional sitemap and sitemap-index discovery with a buyer-controlled total-page cap
• JSON-LD validation and discovered Schema.org types
• Indexability, canonical, metadata, heading, content-depth, freshness, FAQ, list, and table checks
• Prioritized recommendations with a plain-language reason for every action
• Structured JSON results ready for dashboards, spreadsheets, APIs, and MCP clients

Quick start

1. Open the Actor in Apify Store.
2. Add one or more public HTTP(S) pages in URLs to audit. To expand them automatically, enable Discover pages from sitemaps.
3. Click Start. Most single-page audits finish in seconds.
4. Open the Dataset tab to review scores, crawler access, evidence, and prioritized fixes.

The default input audits https://example.com, so you can run a safe demonstration before changing any settings.

Input

{
  "startUrls": [
    { "url": "https://example.com" },
    { "url": "https://example.com/blog/guide" }
  ],
  "discoverFromSitemaps": false,
  "sitemapUrls": [],
  "maxUrls": 25,
  "timeoutSecs": 20,
  "maxConcurrency": 3
}

Field	Type	Default	Description
startUrls	array	https://example.com	One to 100 public HTTP(S) pages to audit.
discoverFromSitemaps	boolean	false	Read each seed origin's /sitemap.xml and Sitemap: directives, including same-host sitemap indexes.
sitemapUrls	array	empty	Up to 10 custom public XML sitemap URLs. Providing one enables discovery automatically.
maxUrls	integer	25	Maximum total seed plus discovered pages, capped at 100.
timeoutSecs	integer	20	Per-request timeout from 5 to 60 seconds.
maxConcurrency	integer	3	Concurrent audits from 1 to 10. Keep this low for polite crawling.
userAgent	string	Actor user agent	User-Agent sent with audit requests.

Authentication and private pages are intentionally out of scope.

To audit a managed site's sitemap, keep one page from the site as the seed and enable discovery:

{
  "startUrls": [{ "url": "https://your-site.example" }],
  "discoverFromSitemaps": true,
  "maxUrls": 25
}

Only pages with the same origin as a seed URL are added. Custom sitemap files may be hosted on another public origin, but URLs inside them still cannot expand the audit to unrelated sites.

Run through the API

Replace <APIFY_TOKEN> with an Apify API token. The synchronous endpoint returns dataset items directly:

curl -X POST "https://api.apify.com/v2/acts/frere527~geo-ai-visibility-auditor/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com"}],"maxConcurrency":1}'

With the Apify CLI:

apify call frere527/geo-ai-visibility-auditor \
  --input '{"startUrls":[{"url":"https://example.com"}]}'

Output example

{
  "url": "https://example.com/blog/guide",
  "score": 74,
  "grade": "C",
  "indexable": true,
  "robotsTxt": {
    "present": true,
    "allowedCrawlers": ["GPTBot", "ClaudeBot", "PerplexityBot"],
    "blockedCrawlers": []
  },
  "structuredDataTypes": ["Article", "Organization"],
  "topRecommendations": [
    "Publish /llms.txt with a concise site summary and links to canonical, useful pages.",
    "Add concise question-answer sections, lists, or comparison tables."
  ]
}

Each successful URL creates one dataset item. The overview exposes the requested and final URL, HTTP status, response time, 0–100 score, grade, indexability, AI crawler access, discovery files, structured data types, and top actions. The full item also includes every weighted check and its evidence, so the score is auditable instead of opaque.

Scoring

The weighted score covers reachability, indexability, titles and descriptions, canonical URLs, headings, substantive content, semantic HTML, structured data, social metadata, crawler policy, AI discovery files, sitemap availability, answer-shaped content, freshness, and response time. The full checks array makes every point auditable.

Scores are diagnostic signals, not a promise of ranking or inclusion in an AI answer. Search and answer engines use independent systems that can change over time.

Pricing

The live price is $0.025 per successfully audited page, including each page found through a sitemap. The maxUrls input gives the buyer a predictable report-count cap before starting. Failed audits return an error item and do not emit the billable audit-report event. There is no separate subscription or minimum run charge; normal Apify platform usage rules still apply to the person running the Actor.

What this Actor does not do

• It does not guarantee citations, rankings, traffic, or model inclusion.
• It does not log in, bypass access controls, solve CAPTCHAs, or audit private networks.
• It follows declared XML sitemaps only; it does not spider arbitrary links or expand beyond the seed origins.
• It does not modify the audited website.

Troubleshooting

A URL is rejected before the audit starts. The Actor blocks credentials in URLs and private, loopback, link-local, reserved, or local-network destinations, including unsafe redirects. Submit a public HTTP(S) URL.

The result says robots.txt is missing. This is a valid audit finding, not a run failure. Review the recommendation in the output before publishing a crawler policy.

A page times out or returns an error item. Increase timeoutSecs up to 60, lower maxConcurrency, and confirm the page is publicly reachable. Failed audits are not charged as reports.

The reported crawler policy differs from a browser visit. The site may vary responses by network, region, or user agent. Check the returned evidence and verify the live robots.txt from the same environment.

Responsible use

This Actor only requests URLs supplied by the user, declared sitemaps, and four conventional discovery paths on each seed origin. Discovered pages are restricted to the seed origins, sitemap indexes are capped at 10 documents, total pages are capped at 100, and concurrency is capped at 10. No login, evasion, fingerprint spoofing, or access-control bypass is attempted. Private, loopback, link-local, reserved, credential-bearing, and local-network URLs are rejected, including after redirects. Only audit sites you are authorized to assess and follow applicable laws and site terms.

Support and source

Report reproducible problems through the GitHub issue tracker. Include the public URL, run ID, and the unexpected field, but never post API tokens or private data.

The source is available on GitHub under the LICENSE. Focused bug fixes, test cases, and documentation improvements are welcome.