AI-Readiness Site Report: Sitemap, llms.txt, robots.txt & Broken Links

Enter a list of websites and get one clear report per site: whether AI crawlers and search engines can find and read it, what's broken, and what to fix first, with a score out of 100.

What it checks

• robots.txt: whether it exists, whether it blocks all crawlers, which sitemaps it declares, and whether 12 known AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, Google-Extended, PerplexityBot, CCBot, Applebot-Extended, Bytespider, meta-externalagent) are allowed or blocked.
• llms.txt: whether it exists and follows the llmstxt.org format (H1 title, summary, sections of links), whether its links work, and whether llms-full.txt exists.
• Sitemaps: finds them via robots.txt or the usual locations, follows sitemap indexes, counts URLs, checks <lastmod> coverage and off-domain URLs, and tests an even sample of URLs for errors.
• Broken links: crawls up to your page limit on the site, then checks every link it found. Reports each broken link with its status and the page it was found on. Logins and rate limits (401, 403, 429) are counted as "blocked", not broken.
• Structured data: see below.

Structured data (schema.org)

The Actor reads the homepage it already fetches and looks for JSON-LD (<script type="application/ld+json">). It reports each block that isn't valid JSON and lists every @type it finds, including types inside @graph and nested objects. Microdata and RDFa are only reported as present or not.

For a few common types it checks that basic properties are there:

• Organization, LocalBusiness, WebSite: name, url
• Article, BlogPosting, NewsArticle: headline, datePublished
• Product: name, plus offers, review or aggregateRating
• BreadcrumbList: itemListElement
• FAQPage: mainEntity

These are basic checks only. They are not Google Rich Results validation, and passing them doesn't mean a page qualifies for rich results. Only the homepage is checked.

Input

Field	Default	What it does
sites	(required)	Domains or URLs, one per line. Duplicates are removed.
maxPages	25	Pages to crawl per site.
maxLinksToCheck	300	Links to check per site. 0 skips the link check.
checkExternalLinks	false	Also check links to other domains (checked once, never crawled).
sitemapSampleSize	50	Sitemap URLs to test for errors.
maxSitemapUrls	10,000	Stops reading sitemaps after this many URLs.
maxConcurrency	5	Parallel requests per site.

{ "sites": ["example.com", "https://yourshop.com.au"], "maxPages": 50 }

Output

One dataset item per site:

{
  "site": "https://example.com",
  "finalUrl": "https://example.com/",
  "score": 72,
  "schemaScore": 10,
  "scoreBreakdown": { "robots": 25, "sitemap": 17, "llmsTxt": 0, "links": 30 },
  "issueCount": 3,
  "issues": [{ "severity": "high", "area": "sitemap", "message": "4 of 50 sampled sitemap URLs are broken." }],
  "robotsTxt": { "found": true, "sitemaps": ["..."], "blocksAllCrawlers": false, "aiBots": { "GPTBot": "allowed" } },
  "sitemap": { "found": true, "urlCount": 1240, "withLastmodPct": 98, "sampleBroken": [{ "url": "...", "status": 404 }] },
  "llmsTxt": { "found": false, "issues": ["No llms.txt file."] },
  "links": { "pagesCrawled": 25, "linksChecked": 300, "broken": [], "blocked": 2 },
  "schema": { "jsonLdBlocks": 2, "invalidJsonLdBlocks": 0, "types": ["Organization", "WebSite"], "microdataPresent": false, "rdfaPresent": false, "missingProperties": [] },
  "auditedAt": "2026-11-01T04:00:00.000Z"
}

How the score works

robots.txt is worth 25 points, sitemaps 25, llms.txt 20 and links 30, for a total of 100. Structured data is scored separately in schemaScore (0 to 10) and does not change score. Structured data findings are listed in schema.issues. schemaScore is 10 for valid JSON-LD with no missing basic properties, minus 2 per type with missing properties (at most 4) and 3 if any block is invalid. Invalid JSON-LD only scores 2, microdata or RDFa only scores 5, and none scores 0. Points come off for each issue found, and every deduction to score appears in issues, while structured data deductions are listed in schema.issues, not issues, so you can see exactly why a site scored what it did. Blocking AI crawlers is reported but doesn't cost points, because that can be a deliberate choice.

Example output

A real local run with { "sites": ["https://example.com"] } on 4 Nov 2026. example.com has no robots.txt, sitemap or llms.txt, so it scores 45. It has no structured data either, so schemaScore is 0. Some fields are trimmed.

{
  "site": "https://example.com",
  "finalUrl": "https://example.com/",
  "score": 45,
  "schemaScore": 0,
  "scoreBreakdown": {
    "robots": 15,
    "sitemap": 0,
    "llmsTxt": 0,
    "links": 30
  },
  "issueCount": 3,
  "issues": [
    {
      "severity": "high",
      "area": "sitemap",
      "message": "No sitemap found."
    },
    {
      "severity": "medium",
      "area": "robots",
      "message": "No valid robots.txt (HTTP 404)."
    },
    {
      "severity": "medium",
      "area": "llms.txt",
      "message": "No llms.txt file."
    }
  ],
  "robotsTxt": {
    "url": "https://example.com/robots.txt",
    "found": false,
    "status": 404,
    "sitemaps": [],
    "blocksAllCrawlers": false,
    "aiBots": {
      "GPTBot": "allowed",
      "ClaudeBot": "allowed",
      "Google-Extended": "allowed",
      "...": "(trimmed, 12 bots in full output)"
    },
    "issues": [
      "No valid robots.txt (HTTP 404)."
    ]
  },
  "sitemap": {
    "found": false,
    "urlCount": 0,
    "sampleBroken": [],
    "issues": [
      "No sitemap found."
    ],
    "...": "(trimmed)"
  },
  "llmsTxt": {
    "found": false,
    "status": 404,
    "issues": [
      "No llms.txt file."
    ],
    "...": "(trimmed)"
  },
  "links": {
    "pagesCrawled": 1,
    "linksChecked": 0,
    "linksCapped": false,
    "broken": [],
    "blocked": 0
  },
  "schema": {
    "pagesChecked": ["https://example.com/"],
    "jsonLdBlocks": 0,
    "invalidJsonLdBlocks": 0,
    "types": [],
    "microdataPresent": false,
    "rdfaPresent": false,
    "missingProperties": [],
    "issues": ["No schema.org structured data found on the homepage."],
    "note": "Basic properties checks only, not Google Rich Results validation."
  },
  "auditedAt": "2026-11-04T05:27:55.375Z"
}

Pricing

Pay per event: one charge per delivered site report. Unreachable sites and failed checks aren't charged, and the run stops before starting a site your maximum spend can't cover.

Limits and good use

• This Actor only reads publicly available pages or files you provide. It doesn't log in, get past paywalls or captchas, or work around blocks.
• It doesn't return personal information about individuals. If a page happens to contain names or contact details, they're not extracted into the results.
• It sends requests at a polite rate and follows each site's robots.txt rules. Please don't raise concurrency to work around this.
• Only run it on sites and files you own or are allowed to check. You're responsible for using the results in line with the laws that apply to you and the terms of any site you check.
• You're only charged for results the Actor actually delivers. Failed or empty checks aren't billed.
• This Actor isn't affiliated with, endorsed by or sponsored by any site, search engine or AI company it mentions.
• The broken-link check follows links on the site you enter, stays within that domain unless you turn on external-link checks, and stops at your page limit.
• The robots.txt and llms.txt results report what the files say. They're not legal advice about what crawlers are allowed to do.

Changelog

• 0.2: New schema section with basic schema.org structured-data checks on the homepage, and a separate schemaScore from 0 to 10. The main score is unchanged.
• 0.1: First release.