Broken Link Checker, Sitemap and AI SEO (llms.txt) Auditor

Audit any website for broken internal and external links (with automatic GET fallback so WAF/CDN endpoints never cause HEAD false positives), XML & gzipped sitemaps, technical on-page SEO, AI search crawler permissions (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended), and spec-compliant llms.txt generation in one lightweight run.

Why AI-SEO + Technical Link Auditing in One Actor?

Most websites silently block AI search engines in robots.txt or ship broken links inside /llms.txt and XML sitemaps. For every domain you pass, this Actor checks:

1. AI Search Readiness (aiSeo & aiReadinessScore): Parses robots.txt rules across 11 AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, cohere-ai), validates /llms.txt and /llms-full.txt (rejecting SPA HTML soft-404s), verifies links inside llms.txt, and builds a ready-to-publish generatedLlmsTxt Markdown file.
2. Zero-False-Positive Broken Link Checker: Probes links with HEAD first, retries non-2xx responses with a byte-capped GET (Range: bytes=0-4095), and cleanly separates real dead links (404, 410, 5xx, DNS/TLS failures) from bot-protected walls (Cloudflare, AWS WAF, LinkedIn 999, DataDome).
3. Sitemap & On-Page SEO Audit: Expands /sitemap.xml, nested <sitemapindex>, and .xml.gz archives while auditing titles, meta descriptions, canonicals, H1 tags, HTTPS/HSTS, OpenGraph, JSON-LD, and missing image alt attributes (plus optional Google PageSpeed Core Web Vitals).

Who Is This For?

• SEO & GEO (Generative Engine Optimization) Agencies: Run bulk prospect audits showing which clients block GPTBot / ClaudeBot, lack /llms.txt, or leak link equity to 404 URLs — and hand them a ready-to-upload llms.txt file.
• SaaS Documentation & Content Teams: Schedule weekly n8n or Apify cron checks across marketing sites and /docs portals before broken links hurt rankings or break LLM retrieval.
• B2B Lead Generation (Clay / Make): Score agency prospects by brokenLinksCount, seoScore, and aiReadinessScore to trigger hyper-specific cold outreach.

Input

Pass bare domains, root URLs, or documentation subfolder URLs:

{
  "urls": ["anthropic.com", "https://linear.app"],
  "maxPagesPerSite": 5,
  "maxLinksPerSite": 150,
  "checkExternalLinks": true,
  "extractSitemap": true,
  "generateLlmsTxt": true
}

Parameter	Type	Default	Description
urls (required)	string[]	["https://linear.app", "https://cal.com"]	Domains ("anthropic.com") or full URLs ("https://linear.app").
maxPagesPerSite	integer	5	Max internal HTML pages fetched and audited per website (1–200).
maxLinksPerSite	integer	150	Max unique links checked per website (0–2000). Internal links are prioritized first.
checkExternalLinks	boolean	true	Verify external outbound links in addition to internal links.
extractSitemap	boolean	true	Discover and expand XML, .xml.gz, and text sitemaps from robots.txt and standard paths.
maxSitemapUrls	integer	100	Max sitemap URLs returned in sitemapUrls (0–5000; full total is always in sitemapUrlsCount).
generateLlmsTxt	boolean	true	Validate existing /llms.txt and generate a clean llms.txt Markdown file from discovered pages.
respectRobotsTxt	boolean	true	Respect User-agent: * Disallow directives when crawling internal subpages.
concurrency	integer	15	Parallel HTTP worker concurrency (1–50).
googlePageSpeedApiKey	string	""	Optional Google PageSpeed Insights v5 API key (BYOK) to include live Lighthouse & Core Web Vitals (LCP, CLS, TBT).
pageSpeedStrategy	string	"mobile"	"mobile" or "desktop" when googlePageSpeedApiKey is set.

Output

Each item is written to the default Dataset (JSON, CSV, Excel, Parquet), and run totals are saved to SUMMARY in the default Key-Value Store. Below is an untouched live output record for "anthropic.com" (maxPagesPerSite: 1, maxSitemapUrls: 8):

{
  "input": "anthropic.com",
  "domain": "anthropic.com",
  "rootUrl": "https://www.anthropic.com",
  "finalUrl": "https://www.anthropic.com/",
  "status": "audited",
  "overallScore": 86,
  "seoScore": 97,
  "aiReadinessScore": 60,
  "discoveryMethod": "robots_sitemap",
  "sitemapFound": true,
  "sitemapUrlsCount": 589,
  "sitemapLocations": [
    "https://www.anthropic.com/sitemap.xml"
  ],
  "sitemapUrls": [
    "https://www.anthropic.com/",
    "https://www.anthropic.com/about-anthropic-interviewer",
    "https://www.anthropic.com/ai-for-science-program-rules",
    "https://www.anthropic.com/beneficial-deployments",
    "https://www.anthropic.com/candidate-ai-guidance",
    "https://www.anthropic.com/careers",
    "https://www.anthropic.com/careers/jobs",
    "https://www.anthropic.com/claude-corps"
  ],
  "pagesAudited": 1,
  "linksChecked": 27,
  "brokenLinksCount": 0,
  "botProtectedLinksCount": 0,
  "redirectedLinksCount": 7,
  "brokenLinks": [],
  "aiSeo": {
    "hasRobotsTxt": true,
    "hasLlmsTxt": false,
    "hasLlmsFullTxt": false,
    "llmsTxtUrl": null,
    "llmsTxtValidFormat": false,
    "llmsTxtLinksCount": 0,
    "llmsTxtBrokenLinksCount": 0,
    "aiCrawlersAllowed": [
      "GPTBot",
      "OAI-SearchBot",
      "ChatGPT-User",
      "ClaudeBot",
      "anthropic-ai",
      "PerplexityBot",
      "Google-Extended",
      "Applebot-Extended",
      "CCBot",
      "Bytespider",
      "cohere-ai"
    ],
    "aiCrawlersBlocked": [],
    "aiSearchReady": true
  },
  "rootPage": {
    "url": "https://anthropic.com/",
    "finalUrl": "https://www.anthropic.com/",
    "statusCode": 200,
    "responseTimeMs": 356,
    "htmlBytes": 173179,
    "botProtected": false,
    "title": "Home \\ Anthropic",
    "metaDescription": "Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.",
    "canonicalUrl": "https://www.anthropic.com/",
    "h1": "AI research and products that put safety at the frontier. AI will have a vast impact on the world. Anthropic is a public benefit corporation dedicated to securing its benefits and mitigating its risks.",
    "h1Count": 1,
    "lang": "en",
    "indexable": true,
    "hasHttps": true,
    "hasHsts": true,
    "hasOpenGraph": true,
    "jsonLdTypes": [],
    "wordCount": 269,
    "imagesTotal": 4,
    "imagesMissingAlt": 4,
    "internalLinksCount": 28,
    "externalLinksCount": 5,
    "brokenLinksOnPage": 0,
    "issues": [
      "images_missing_alt"
    ]
  },
  "issues": [
    {
      "code": "images_missing_alt",
      "severity": "low",
      "message": "One or more content images are missing alt attributes.",
      "affectedCount": 1,
      "sampleUrls": [
        "https://www.anthropic.com/"
      ]
    },
    {
      "code": "missing_llms_txt",
      "severity": "low",
      "message": "No /llms.txt file found to guide LLM search engines and coding/research agents.",
      "affectedCount": 1,
      "sampleUrls": [
        "https://www.anthropic.com/llms.txt"
      ]
    }
  ],
  "auditedPages": [
    {
      "url": "https://anthropic.com/",
      "finalUrl": "https://www.anthropic.com/",
      "statusCode": 200,
      "responseTimeMs": 356,
      "htmlBytes": 173179,
      "botProtected": false,
      "title": "Home \\ Anthropic",
      "metaDescription": "Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.",
      "canonicalUrl": "https://www.anthropic.com/",
      "h1": "AI research and products that put safety at the frontier. AI will have a vast impact on the world. Anthropic is a public benefit corporation dedicated to securing its benefits and mitigating its risks.",
      "h1Count": 1,
      "lang": "en",
      "indexable": true,
      "hasHttps": true,
      "hasHsts": true,
      "hasOpenGraph": true,
      "jsonLdTypes": [],
      "wordCount": 269,
      "imagesTotal": 4,
      "imagesMissingAlt": 4,
      "internalLinksCount": 28,
      "externalLinksCount": 5,
      "brokenLinksOnPage": 0,
      "issues": [
        "images_missing_alt"
      ]
    }
  ],
  "generatedLlmsTxt": "# Home \\ Anthropic\n\n> Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.\n\n## Company & Content\n\n- [Careers](https://www.anthropic.com/careers)\n- [Jobs](https://www.anthropic.com/careers/jobs)\n- [Company](https://www.anthropic.com/company)\n- [Leadership](https://www.anthropic.com/company/leadership)\n\n## Key Pages\n\n- [Home \\ Anthropic](https://www.anthropic.com/): Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.\n- [About Anthropic Interviewer](https://www.anthropic.com/about-anthropic-interviewer)\n- [Ai For Science Program Rules](https://www.anthropic.com/ai-for-science-program-rules)\n- [Beneficial Deployments](https://www.anthropic.com/beneficial-deployments)\n- [Candidate Ai Guidance](https://www.anthropic.com/candidate-ai-guidance)\n- [Claude Corps](https://www.anthropic.com/claude-corps)\n- [Fellow](https://www.anthropic.com/claude-corps/fellow)\n- [Host](https://www.anthropic.com/claude-corps/host)\n- [Claude Fable And Mythos 5 1](https://www.anthropic.com/claude-fable-and-mythos-5-1)\n- [Claude Haiku 5 5](https://www.anthropic.com/claude-haiku-5-5)\n- [Claude Opus 5 5](https://www.anthropic.com/claude-opus-5-5)\n- [Claude Sonnet 5 5](https://www.anthropic.com/claude-sonnet-5-5)\n- [Fable](https://www.anthropic.com/claude/fable)\n- [Haiku](https://www.anthropic.com/claude/haiku)\n- [Mythos](https://www.anthropic.com/claude/mythos)\n- [Opus](https://www.anthropic.com/claude/opus)\n- [Sonnet](https://www.anthropic.com/claude/sonnet)\n- [Constitution](https://www.anthropic.com/constitution)\n- [Coordinated Vulnerability Disclosure](https://www.anthropic.com/coordinated-vulnerability-disclosure)\n- [Economic Futures](https://www.anthropic.com/economic-futures)\n- [Program](https://www.anthropic.com/economic-futures/program)\n",
  "lighthouse": null,
  "error": null,
  "processedAt": "2026-11-04T09:25:00.589Z"
}

Pricing (Pay Per Event)

Invalid URLs and unreachable domains cost $0. Link checks (maxLinksPerSite) are included in the page audit price at $0 extra.

Event	Title	Price	Charged When
actor-start	Actor start	$0.0005	Once per run
site-audited	Audited website	$0.0030 ($3.00 / 1,000)	Target website responds and its robots.txt, sitemaps, AI crawler rules, and llms.txt are audited
page-audited	Audited web page	$0.0010 ($1.00 / 1,000)	Each reachable HTML page audited (includes up to maxLinksPerSite link checks)

• 100 websites (5 pages + up to 150 link checks + sitemap + llms.txt each): ~$0.80 total (~625 multi-page site audits/month covered by Apify's $5 Free plan credit).
• 1,000 single-page AI-SEO & homepage link audits (maxPagesPerSite: 1): $4.00 total.
• Respects Apify's Maximum total charge cap and stops cleanly when reached.

API, Clay, n8n & MCP Integration

curl -X POST "https://api.apify.com/v2/acts/xenikii~broken-link-sitemap-llms-txt-auditor/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://linear.app", "anthropic.com"], "maxPagesPerSite": 5}'

• n8n / Make Scheduled Monitoring: Trigger weekly site health checks and alert Slack whenever brokenLinksCount > 0 or aiSearchReady === false.
• Clay Agency Prospecting: Pass ["{{Website}}"] with maxPagesPerSite: 1 to personalize outreach around missing /llms.txt, blocked AI bots, or broken homepage links.
• MCP (mcp.apify.com): Works in Claude Desktop, Cursor, and AI coding agents as xenikii/broken-link-sitemap-llms-txt-auditor.

FAQ & Data Compliance

• Why do other link checkers flag LinkedIn, Cloudflare, or X links as 403/999 broken? Many sites reject HEAD requests or challenge automated probes. This Actor retries non-2xx HEAD responses with a ranged GET and classifies WAF challenges separately under botProtectedLinksCount so brokenLinks stays clean.
• Does it respect robots.txt? Yes (respectRobotsTxt: true by default) and includes built-in SSRF protection blocking private/internal IP ranges.
• Data & Privacy Compliance: Audits publicly accessible web pages, robots.txt, sitemaps, and HTTP response headers only — zero personal data or cookies are stored.

Other B2B Intelligence Actors

Combine these Actors in Clay, n8n, Make, or MCP to build a complete B2B signal and SEO stack:

Actor	Best For	Pricing
Company Domain, Tech Stack & MX/DNS Enricher	Resolve company names to domains, 12-category tech stacks, MX/SPF/DMARC & role contacts	$2.50 / 1k enriched domains
B2B Hiring Signals & Multi-ATS Job Scraper	Extract hiring velocity, salary transparency & open roles across 7 ATS platforms	$1.50 / 1k job postings
Google Ads Transparency & Ad Library Scraper	Spy on active Google Search, Display & YouTube creatives, days active & EU reach	$1.80 / 1k ad creatives
Broken Link, Sitemap & AI SEO (llms.txt) Auditor (This Actor)	Audit broken links, XML sitemaps, GPTBot/ClaudeBot rules & generate llms.txt	$3.00 / 1k audited websites
Google Trends Breakout & Comparison Scraper	Track +5,000% breakout queries, multi-keyword share of search & live news trends	$2.50 / 1k keyword trend reports