Bulk llms.txt & Technical SEO Checker

A bulk llms.txt checker and technical SEO audit: check hundreds of websites in one run for AI-search readiness (llms.txt, AI crawlers in robots.txt) and technical SEO. Each site gets a 0-100 score, a letter grade and a plain-English fix for every problem found.

Built for agencies, freelancers and site owners who need a fast first look before a proposal, a migration or a monthly report.

Try it on 10 websites for $0.05 (the free credit on Apify's free plan covers it).

How to run it

1. Paste domains or URLs into websites.
2. Optional: set onlyBelowScore to keep only sites with problems, and maxLinksToCheck for the broken-link test.
3. Press Start and download the results as Excel, CSV or JSON.

What it checks (homepage of each site)

Area	Checks
AI search	Which AI crawlers robots.txt blocks (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and more), and whether an llms.txt file exists
Indexing	robots.txt present, Googlebot blocked, noindex on the homepage, XML sitemap found and URL count, sitemap listed in robots.txt
On-page	Title and length, meta description and length, H1 count, canonical tag, viewport, lang, Open Graph tags, hreflang count
Structured data	Every JSON-LD @type on the page (Organization, Product, FAQPage, LocalBusiness...)
Security & speed	HTTPS redirect, redirect count, response time, SSL certificate days left
Broken links	Internal links on the homepage that return 404, 410 or 5xx (login and cart links are skipped)

Real numbers: new Shopify stores

We ran it on 56 randomly picked Shopify stores launched in September–October 2026:

• 58% had no meta description on the homepage.
• 58% had a homepage title shorter than 15 characters, usually just the brand name.
• 19% used more than one H1.
• Average score 93/100: the platform handles the basics (sitemap, robots.txt, llms.txt, HTTPS), but the content fields are left empty.

Input

{
  "websites": ["apify.com", "https://www.example-store.com"],
  "maxLinksToCheck": 20,
  "onlyBelowScore": 80
}

• websites: domains or URLs. Duplicates are removed.
• maxLinksToCheck: homepage links to test for broken pages (0 turns it off).
• onlyBelowScore (optional): return only sites below this score. Sites filtered out are not charged.

Output (one row per website)

{
  "domain": "theverge.com",
  "score": 85,
  "grade": "B",
  "https": true,
  "responseMs": 1394,
  "title": "The Verge",
  "titleLength": 9,
  "metaDescriptionLength": 221,
  "h1Count": 0,
  "schemaTypes": ["EntryPoint", "ImageObject", "NewsMediaOrganization", "Person", "SearchAction", "Thing", "WebSite"],
  "robotsTxt": true,
  "googlebotBlocked": false,
  "aiBotsBlocked": ["ChatGPT-User", "ClaudeBot", "Claude-SearchBot", "anthropic-ai", "PerplexityBot", "Google-Extended", "Applebot-Extended", "CCBot", "Bytespider", "Meta-ExternalAgent"],
  "aiSearchBotsBlocked": ["ChatGPT-User", "Claude-SearchBot", "PerplexityBot"],
  "llmsTxt": false,
  "sitemapUrls": 69,
  "sslDaysLeft": 87,
  "linksChecked": 20,
  "brokenLinks": [],
  "issues": ["Title is 9 characters.", "Meta description is 221 characters.", "No H1 heading on the homepage.", "robots.txt blocks AI search crawlers: ChatGPT-User, Claude-SearchBot, PerplexityBot."],
  "fixes": ["Keep the title between 30 and 60 characters.", "Aim for 120-160 characters.", "Add one H1 that says what the business does.", "Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot if you want to be cited in AI answers."],
  "status": "ok"
}

aiBotsBlocked lists every AI crawler the site blocks; aiSearchBotsBlocked is the subset that sends visitors from AI answers. Blocking training-only crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) is a business choice and is not penalised; blocking AI search crawlers is.

Scoring

Starts at 100. Biggest deductions: Googlebot blocked −30, homepage noindex −25, no HTTPS −15, no title −10, SSL expiring within 14 days −10, broken links up to −10, no viewport −8, no meta description −6, no JSON-LD −6, AI search crawlers blocked −5, no sitemap −5, no H1 −5. Grades: A ≥ 90, B ≥ 75, C ≥ 60, D ≥ 40, F below.

Pricing

$5 per 1,000 websites audited (pay per event site-audited). Sites that are unreachable, return an error or show a bot-protection page are free.

Good to know

• Only the homepage is audited; the sitemap count follows up to 3 child sitemaps of a sitemap index.
• Sites behind bot protection are reported with their status and are not charged.

FAQ

What is llms.txt and does it matter? llms.txt is a proposed file that tells AI tools what a site is about. It is optional and Google doesn't use it; the actor simply reports whether it exists. Allowing AI search crawlers in robots.txt matters more if you want to be cited in AI answers.

Does it crawl the whole site? No. It audits the homepage, robots.txt, llms.txt and the sitemap, and tests up to maxLinksToCheck homepage links for broken pages.

Why was a site not charged? Sites that are unreachable, return an error or show a bot-protection page are listed with their status and are free.

Do I need an API key? No. Only public pages are read.

Related actors

• Shopify Store Audit - CRO & SEO Score: 27-check conversion and SEO audit built for Shopify stores.
• Bulk SPF, DKIM & DMARC Checker: SPF, DKIM, DMARC, MX and blocklists with a score and fixes.
• Website Business Contact Finder: role-based business emails, contact page and socials.
• App Store & Google Play Scraper · Shopify Store Leads Scraper · Shopify Theme & Apps Detector