AI Crawler Checker & llms.txt Generator — Website AI Audit

Audit how ready any website is for AI search engines — ChatGPT, Perplexity, Claude, Gemini, Google AI Overviews and Copilot. Get a 0–100 AI-readiness score, a prioritized fix list, a per-domain AI-crawler access matrix, and a ready-to-upload llms.txt draft.

More search traffic is shifting to AI answers every quarter. But AI engines can only cite your content if their crawlers can access it, understand its structure, and extract it cleanly. This Actor audits exactly that — deterministically, with no LLM calls and no guesswork.

No LLM calls. No guesswork. Deterministic checks for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 16 more AI crawlers, llms.txt, sitemap, schema.org and content extractability — the same site always gets the same score, so re-runs measure real progress.

AI agent quick start

Actor tool ID: dev-hoss/ai-search-readiness-audit

• Send startUrls (1–10 domains) and optionally maxPages (default 10). One summary record per domain: filter dataset items by type === "summary".
• Per-page detail (type === "page") lives in the named pages dataset: GET /v2/datasets/{pagesDatasetId}/items.
• Scores are deterministic 0–100 — safe to use as a regression metric in CI or scheduled re-audits.
• An info/error record with a plain-English message is emitted when a domain is unreachable; it is not a failed run.

What it checks

Category	Checks
Crawlability	robots.txt presence; access rules for 20 AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot, Bytespider, CCBot, Amazonbot, meta-externalagent, and more); sitemap discovery (robots.txt directives, /sitemap.xml, sitemap indexes); HTTP status of every audited page
Discoverability	llms.txt presence and validity (llmstxt.org format), meta descriptions, canonical URLs, Open Graph tags
Content structure	H1 presence and heading hierarchy, subheading coverage, content depth (word count), schema.org JSON-LD coverage
Machine readability	text-to-HTML ratio (extractability), JSON-LD parse errors, entity schema types (Organization, Article, Product, FAQPage…), llms.txt

Quick start

{
    "startUrls": ["https://apify.com"],
    "maxPages": 10
}

That's it. Each distinct domain in startUrls gets its own full report.

Input

Field	Type	Default	Description
startUrls	array (required)	—	1–10 URLs. Each distinct domain is audited separately. Any page on the domain works; the homepage is recommended.
maxPages	integer	10	Pages crawled per domain (1–200). 10 for a quick check, 25–50 for a real audit.
maxConcurrency	integer	5	Parallel requests. Lower to 1–2 for rate-limiting sites.
respectRobotsTxt	boolean	true	Whether the audit crawler itself obeys robots.txt. Does not affect AI-crawler checks, which are computed analytically.
proxyConfiguration	object	Apify proxy off	Optional proxy for sites that rate-limit datacenter IPs.

Output

Results land in the default dataset, distinguished by type — filter in Console or via API:

1. type=summary, one per domain (the headline result):

{
    "type": "summary",
    "domain": "example.com",
    "overall": 62,
    "scoreInterpretation": "Fair — several important fixes needed",
    "categories": {
        "crawlability": 18,
        "discoverability": 8,
        "contentStructure": 22,
        "machineReadability": 14
    },
    "aiCrawlerMatrix": {
        "GPTBot": { "label": "OpenAI GPTBot (model training)", "verdict": "allowed" },
        "PerplexityBot": { "label": "PerplexityBot (Perplexity index)", "verdict": "disallowed" }
    },
    "robotsTxt": { "exists": true, "sitemapDirectives": ["https://example.com/sitemap.xml"] },
    "llmsTxt": { "present": false, "valid": false, "url": "https://example.com/llms.txt" },
    "sitemap": { "found": true, "url": "https://example.com/sitemap.xml", "urlCount": 412 },
    "topIssues": [
        {
            "code": "AI_BOTS_BLOCKED",
            "severity": "critical",
            "category": "crawlability",
            "count": 2,
            "detail": "Fully blocked in robots.txt: PerplexityBot, Bytespider.",
            "fix": "Remove the Disallow rules for the AI crawlers you want to be visible to.",
            "exampleUrls": ["https://example.com/robots.txt"]
        }
    ],
    "pagesAudited": 10,
    "llmsTxtDraft": "# Example Company\n\n> We help teams...\n\n## Pages\n\n- [Pricing](https://example.com/pricing): Plans for teams of every size...",
    "stats": { "avgScore": 58, "minScore": 31, "maxScore": 88, "avgWordCount": 743 }
}

2. type=page — one record per audited URL (per-page detail): title, meta description, canonical, meta robots, headings, JSON-LD types and parse errors, Open Graph, word count, text-to-HTML ratio, per-crawler access verdict, per-page score (0–100) and issues. Pages that can't be fetched after retries produce a type=page-error record. Filter the default dataset by type=page in Console, or via API:

GET https://api.apify.com/v2/datasets/{defaultDatasetId}/items?clean=true

Use the Domain summary / Pages tabs in Console to browse each record kind (type=summary vs type=page).

If no page on a domain is reachable, the default dataset gets an error record explaining why.

The generated llms.txt draft

When llms.txt is missing, the summary includes a complete draft built from your best pages (title + description + URL each), in llmstxt.org format. Save it as llms.txt, review, upload it to your domain root — instant fix for the most common GEO gap.

Scoring

Category scores are weighted: crawlability 25, content structure 30, machine readability 25, discoverability 20 → overall 0–100. Issues subtract points by severity (critical −25, high −12, medium −6, low −2). Scoring is fully deterministic — the same site always gets the same score, so re-runs measure your real progress.

How much does it cost?

Pay per event: you pay only for pages actually audited (plus a small summary event per domain). A 10-page audit of one domain costs a few cents — compare that to GEO SaaS platforms at $99–$500+/month or agency audits at $1,500+.

Who is this for?

• SEO/GEO agencies & freelancers — add AI-readiness audits to your service offering
• Marketing teams — check whether ChatGPT/Perplexity can even see your site
• Developers — CI/monitoring integration via Apify API; deterministic scores for regression tracking
• AI agents — structured JSON output, predictable records, machine-readable verdicts

Use with AI agents

This Actor is designed for agent consumption: documented JSON input schema, stable output records, no auth beyond the caller's own Apify token. The simplest integration is the run-sync endpoint, which starts the run and returns the results in one call:

POST https://api.apify.com/acts/dev-hoss~ai-search-readiness-audit/run-sync-get-dataset-items?token=YOUR_API_TOKEN
Content-Type: application/json
{"startUrls": ["https://example.com"], "maxPages": 10}

It also works with Apify's hosted MCP server (mcp.apify.com) — add the Actor to your MCP client and the input schema drives the tool call. Record shapes never change silently: summary records carry the score and fixes, page / page-error records carry per-URL detail (all in the default dataset, filter by type).