Website AI-Readiness Audit

Give it a domain. Get a 0–100 AI-readiness score, a letter grade and the specific fixes, from robots.txt, llms.txt, the sitemap and sampled pages.

No API key, no login. $8 per 1,000 audits ($0.008 each), rising to $20 per 1,000 on 17 Nov 2026; sites that are unreachable, blocked or invalid get a free item that says why.

What you get

• What robots.txt tells 16 known AI crawlers, plus checks for /llms.txt, a sitemap and HTTPS. Training bots are reported but don't affect the score.
• Sampled pages checked for text in the raw HTML (no JavaScript), JSON-LD, title and meta description, headings, noindex/noai directives, dates and canonical links.
• Structured JSON plus a ready-to-share Markdown report (reportMarkdown). The score is our own heuristic checklist, not a prediction of AI rankings or citations.

Quick start

1. Open this Actor in Apify Console.
2. On the Input tab, switch to the JSON view and paste the input below.
3. Click Start. When the run finishes, open the Output tab to view or export the results.

Input: one audit of crawlee.dev with 3 sampled pages, so $0.008.

{
  "urls": [
    "crawlee.dev"
  ],
  "maxPagesPerSite": 3
}

Sample output

From cloud run etWPp2rSYEZEyg4Vh (2 Nov 2026), the single item this exact input returned. findings is cut to its 1 failing check (of 14), and input, domain, redirectedTo, aiCrawlers, llmsTxt, llmsFullTxt, pages, reportMarkdown, error and auditedAt are left out. That cloud run took about 8.5 seconds.

{
  "url": "https://crawlee.dev/",
  "score": 90,
  "grade": "A",
  "summary": "crawlee.dev scores 90/100 (A). 1 failing and 2 warning checks.",
  "findings": [
    {
      "id": "freshness",
      "category": "Content",
      "weight": 3,
      "status": "fail",
      "message": "33% of 3 sampled page(s) expose a published or modified date.",
      "fix": "Expose dateModified in JSON-LD or article:modified_time."
    }
  ],
  "robotsTxtFound": true,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "pagesAudited": 3,
  "errorCode": null
}

API

Same input as above. Every example reads your token from the APIFY_TOKEN environment variable; never paste a real token into code you share.

curl

curl -X POST "https://api.apify.com/v2/acts/madpai~ai-readiness-audit/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls": ["crawlee.dev"], "maxPagesPerSite": 3}'

JavaScript (apify-client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const input = {
  "urls": [
    "crawlee.dev"
  ],
  "maxPagesPerSite": 3
};

const run = await client.actor('madpai/ai-readiness-audit').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const item of items) {
    console.log(item.url, item.score, item.grade);
}

Python (apify-client)

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run_input = {
  "urls": [
    "crawlee.dev"
  ],
  "maxPagesPerSite": 3
}

run = client.actor('madpai/ai-readiness-audit').call(run_input=run_input)
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item.get('url'), item.get('score'), item.get('grade'))

Run summary

The run also writes an OUTPUT_SUMMARY record to its default key-value store: await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT_SUMMARY') (Python: client.key_value_store(run['defaultKeyValueStoreId']).get_record('OUTPUT_SUMMARY')).

With curl, run-sync-get-dataset-items returns only the items, not the run ID, so it can't read OUTPUT_SUMMARY. To get the summary from curl, start the run and wait for it to finish, then fetch the record from the run's key-value store (needs jq):

RUN=$(curl -s -X POST "https://api.apify.com/v2/acts/madpai~ai-readiness-audit/runs?waitForFinish=300" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls": ["crawlee.dev"], "maxPagesPerSite": 3}')
KVS=$(echo "$RUN" | jq -r '.data.defaultKeyValueStoreId')
DS=$(echo "$RUN" | jq -r '.data.defaultDatasetId')
curl -s "https://api.apify.com/v2/key-value-stores/$KVS/records/OUTPUT_SUMMARY" -H "Authorization: Bearer $APIFY_TOKEN"
curl -s "https://api.apify.com/v2/datasets/$DS/items?format=json" -H "Authorization: Bearer $APIFY_TOKEN"

How it works

It checks how easy it is for AI search and answer engines to reach and read a website, then gives a 0–100 score, a letter grade, and a list of specific fixes. Per site it checks:

• Crawler access: what robots.txt says to 16 known AI crawlers. Search and user-fetch bots count toward the score; training bots are reported but don't affect it, since blocking them is a legitimate choice.
• AI guidance: whether /llms.txt (and the optional /llms-full.txt) exists.
• Discoverability and basics: sitemap, HTTPS.
• Content (sampled pages): whether main text is in the raw HTML without JavaScript, JSON-LD structured data, title and meta description, heading structure, noindex/noai directives, dates and canonical links.

What the score is and isn't

The score is our heuristic: our own weighted checklist of technical signals. It shows what you can fix; it does not measure or guarantee whether any AI product will crawl, cite or rank you.

Who is it for?

SEO and content teams preparing for AI search, agencies running quick audits for clients, and AI agents that need a structured readiness check.

Input

Field	Default	What it does
urls (required)	prefill crawlee.dev	Domains or homepages to audit. One audit, one charge, per site. Different spellings of the same site (crawlee.dev, https://crawlee.dev/, www.crawlee.dev) are audited and charged once.
maxPagesPerSite	5 (max 25)	Homepage plus pages spread across the sitemap.
includePageDetails	true	Add per-page checks to each result.
respectRobotsTxt	true	Don't sample pages robots.txt disallows. Can only be turned off together with ownSiteConfirmed.
requestDelayMs	1000	Delay between requests to one site (about 1 request per second, longer if robots.txt sets a Crawl-delay). Shorter delays need ownSiteConfirmed.
maxConcurrency	3 (max 10)	Sites audited in parallel. Each site still gets about 1 request per second.
ownSiteConfirmed	false	Tick only for sites you own or are authorised to crawl. Unlocks turning off robots.txt and delays under 1 second.

Minimal input:

{ "urls": ["crawlee.dev"] }

Output

One item per site: score, grade, summary, findings (each with category, weight, status, message and fix), aiCrawlers, robotsTxtFound, llmsTxt, llmsFullTxt, sitemapUrl, pagesAudited, pages, reportMarkdown, redirectedTo, error, errorCode. Real excerpt from a local test run (1 Nov 2026):

{
  "domain": "example.com",
  "score": 38,
  "grade": "F",
  "summary": "example.com scores 38/100 (F). 8 failing and 2 warning checks.",
  "findings": [
    { "id": "sitemap", "category": "Discoverability", "weight": 10, "status": "fail",
      "message": "No sitemap found via robots.txt or common paths.",
      "fix": "Publish /sitemap.xml and reference it in robots.txt." }
  ]
}

reportMarkdown is a ready-to-share report you can paste into a doc or hand to an LLM.

If the homepage redirects to another host, that host is audited instead: redirectedTo shows where it went and a redirected finding says so.

Sites that aren't audited still get an item with an error message, and aren't charged. Every one also has an errorCode. From the same local run:

{ "input": "nytimes.com", "url": "https://nytimes.com/", "error": "Homepage unreachable: HTTP 403", "errorCode": "http-error" }

errorCode	Meaning
robots-disallowed	robots.txt disallows the homepage (or a redirect hop to it) for our crawler.
robots-unavailable	robots.txt couldn't be fetched (server error, 429 or network error). Under RFC 9309 that means "disallow everything", so the site isn't audited.
redirect-signin	The homepage redirects to a sign-in page. We never log in.
blocked-private	The address is private, loopback, link-local or otherwise reserved. It's never requested.
http-error / unreachable	The homepage returned an error status (for example 403) or couldn't be reached. A domain that doesn't resolve is unreachable with "DNS lookup failed (ENOTFOUND)", and nothing is requested, not even robots.txt.
invalid-input	The input isn't a valid URL or domain, so nothing was requested.
internal-error	The audit stopped on an unexpected error. Please report it.

How it treats websites

• robots.txt is read before any other request to a host, following RFC 9309. llms.txt, llms-full.txt, sitemap probes and sampled pages are all checked against it.
• About 1 request per second per host, or the robots.txt Crawl-delay if that's longer (up to 30 s), counted from the robots.txt request itself. Retry-After is honoured, and after 3 HTTP 429 responses in a row the host gets no further requests.
• Redirects are followed one hop at a time, and each hop is checked against its own host's robots.txt.
• llms-full.txt is never downloaded in full: only the first 256 KB is read, and its size comes from the response headers.
• It reads public pages and files only. No logins, no CAPTCHA solving, no proxies.

How long does it take?

Local test numbers (17 fixture runs, 1 Nov 2026): a run took 6.7 s at the median and 30.7 s at the slowest, with a peak of 165 MB against the 256 MB default memory. Because requests to one host are spaced about 1 second apart, more sampled pages or a robots.txt Crawl-delay make an audit slower.

How much does it cost?

Pay per event: one site-audit event per completed audit, at $8 per 1,000 audits ($0.008 per audit). Example: auditing 50 sites costs $0.40. Sites that are unreachable, blocked (by robots.txt, a 403 or a sign-in redirect) or on a private address are saved with an error and not charged. Duplicate spellings of a site are audited and charged once. The price rises to $20 per 1,000 audits ($0.02 per audit) on 17 November 2026; until then it's $8 per 1,000.

Use with AI agents

Designed to run with limited permissions. Output is structured JSON plus a Markdown report, so an agent can read the score and the fixes directly. Example prompt: "Audit example.com for AI readiness and list the top three fixes."

Limits

• It samples a few pages, not the whole site, so the content score is an estimate for the site.
• Sites that block automated requests (for example with a 403) return an error, which is never bypassed.
• It reads raw HTML only and doesn't run JavaScript.
• The list of AI crawlers reflects what's publicly documented as of the release date; new bots appear often.

FAQ

Does a high score mean AI products will crawl, cite or rank my site? No. The score is our own weighted checklist of technical signals. It shows what you can fix; it doesn't measure or guarantee what any AI product does. Does blocking AI training bots lower my score? No. Training bots are reported, but only search and user-fetch bots count toward the score, since blocking training bots is a legitimate choice. Am I charged for sites that can't be audited? No. Unreachable sites, sites blocked by robots.txt, a 403 or a sign-in redirect, private addresses and invalid inputs are saved with an error and an errorCode, and aren't charged. Different spellings of the same site are audited and charged once. Does it run JavaScript? No. It reads raw HTML only, which is why it checks whether a page's main text is there without JavaScript. What if my homepage redirects to another domain? That host is audited instead. redirectedTo shows where it went, and a redirected finding says so.

Run it on a schedule

To track the score after you make fixes, create a schedule for this Actor in Apify Console (Schedules), for example weekly, with the same input.

Related Actors

• Sitemap Extractor & URL Discovery: list the URLs a site publishes in its sitemaps.
• Broken Link & Redirect Checker: find broken links and redirect chains on the site.

Legal

Reads public pages and files only, respects robots.txt by default, and collects no personal data. You're responsible for using it in line with each site's terms. Not affiliated with any AI company or crawler operator named in the results.

Changelog

See CHANGELOG.md.

Support

Report problems in the Issues tab. We aim to reply within 24 hours.