AI Search Readiness Audit (GEO/AEO) & llms.txt Generator

Audit how ready any website is for AI search engines — ChatGPT, Perplexity, Claude, Gemini, Google AI Overviews and Copilot. Get a 0–100 AI-readiness score, a prioritized fix list, a per-domain AI-crawler access matrix, and a ready-to-upload llms.txt draft.

More search traffic is shifting to AI answers every quarter. But AI engines can only cite your content if their crawlers can access it, understand its structure, and extract it cleanly. This Actor audits exactly that — deterministically, with no LLM calls and no guesswork.

What it checks

Category	Checks
Crawlability	robots.txt presence; access rules for 20 AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot, Bytespider, CCBot, Amazonbot, meta-externalagent, and more); sitemap discovery (robots.txt directives, /sitemap.xml, sitemap indexes); HTTP status of every audited page
Discoverability	llms.txt presence and validity (llmstxt.org format), meta descriptions, canonical URLs, Open Graph tags
Content structure	H1 presence and heading hierarchy, subheading coverage, content depth (word count), schema.org JSON-LD coverage
Machine readability	text-to-HTML ratio (extractability), JSON-LD parse errors, entity schema types (Organization, Article, Product, FAQPage…), llms.txt

Quick start

{
    "startUrls": ["https://apify.com"],
    "maxPages": 10
}

That's it. Each distinct domain in startUrls gets its own full report.

Input

Field	Type	Default	Description
startUrls	array (required)	—	1–10 URLs. Each distinct domain is audited separately. Any page on the domain works; the homepage is recommended.
maxPages	integer	10	Pages crawled per domain (1–200). 10 for a quick check, 25–50 for a real audit.
maxConcurrency	integer	5	Parallel requests. Lower to 1–2 for rate-limiting sites.
respectRobotsTxt	boolean	true	Whether the audit crawler itself obeys robots.txt. Does not affect AI-crawler checks, which are computed analytically.
proxyConfiguration	object	Apify proxy off	Optional proxy for sites that rate-limit datacenter IPs.

Output

Results land in two datasets, so each stays clean:

1. Default dataset — summary, one per domain (the headline result):

{
    "type": "summary",
    "domain": "example.com",
    "overall": 62,
    "scoreInterpretation": "Fair — several important fixes needed",
    "categories": {
        "crawlability": 18,
        "discoverability": 8,
        "contentStructure": 22,
        "machineReadability": 14
    },
    "aiCrawlerMatrix": {
        "GPTBot": { "label": "OpenAI GPTBot (model training)", "verdict": "allowed" },
        "PerplexityBot": { "label": "PerplexityBot (Perplexity index)", "verdict": "disallowed" }
    },
    "robotsTxt": { "exists": true, "sitemapDirectives": ["https://example.com/sitemap.xml"] },
    "llmsTxt": { "present": false, "valid": false, "url": "https://example.com/llms.txt" },
    "sitemap": { "found": true, "url": "https://example.com/sitemap.xml", "urlCount": 412 },
    "topIssues": [
        {
            "code": "AI_BOTS_BLOCKED",
            "severity": "critical",
            "category": "crawlability",
            "count": 2,
            "detail": "Fully blocked in robots.txt: PerplexityBot, Bytespider.",
            "fix": "Remove the Disallow rules for the AI crawlers you want to be visible to.",
            "exampleUrls": ["https://example.com/robots.txt"]
        }
    ],
    "pagesAudited": 10,
    "llmsTxtDraft": "# Example Company\n\n> We help teams...\n\n## Pages\n\n- [Pricing](https://example.com/pricing): Plans for teams of every size...",
    "stats": { "avgScore": 58, "minScore": 31, "maxScore": 88, "avgWordCount": 743 }
}

2. Named dataset pages — one page record per audited URL (per-page detail): title, meta description, canonical, meta robots, headings, JSON-LD types and parse errors, Open Graph, word count, text-to-HTML ratio, per-crawler access verdict, per-page score (0–100) and issues. Pages that can't be fetched after retries produce a page-error record here. In Apify Console, open the run's Storage → Datasets → pages to browse them; via API:

GET https://api.apify.com/v2/datasets/{pagesDatasetId}/items?clean=true

If no page on a domain is reachable, the default dataset gets an error record explaining why.

The generated llms.txt draft

When llms.txt is missing, the summary includes a complete draft built from your best pages (title + description + URL each), in llmstxt.org format. Save it as llms.txt, review, upload it to your domain root — instant fix for the most common GEO gap.

Scoring

Category scores are weighted: crawlability 25, content structure 30, machine readability 25, discoverability 20 → overall 0–100. Issues subtract points by severity (critical −25, high −12, medium −6, low −2). Scoring is fully deterministic — the same site always gets the same score, so re-runs measure your real progress.

How much does it cost?

Pay per event: you pay only for pages actually audited (plus a small summary event per domain). A 10-page audit of one domain costs a few cents — compare that to GEO SaaS platforms at $99–$500+/month or agency audits at $1,500+.

Who is this for?

• SEO/GEO agencies & freelancers — add AI-readiness audits to your service offering
• Marketing teams — check whether ChatGPT/Perplexity can even see your site
• Developers — CI/monitoring integration via Apify API; deterministic scores for regression tracking
• AI agents — structured JSON output, predictable records, machine-readable verdicts

Use with AI agents

This Actor is designed for agent consumption: documented JSON input schema, stable output records, no auth beyond the caller's own Apify token. The simplest integration is the run-sync endpoint, which starts the run and returns the results in one call:

POST https://api.apify.com/acts/dev-hoss~ai-search-readiness-audit/run-sync-get-dataset-items?token=YOUR_API_TOKEN
Content-Type: application/json
{"startUrls": ["https://example.com"], "maxPages": 10}

It also works with Apify's hosted MCP server (mcp.apify.com) — add the Actor to your MCP client and the input schema drives the tool call. Record shapes never change silently: summary records carry the score and fixes, page records live in the named pages dataset.