llms.txt Generator & Checker – AI Crawler Checker & GEO Audit
Pricing
from $17.50 / 1,000 ai search geo domain audits
llms.txt Generator & Checker – AI Crawler Checker & GEO Audit
llms.txt generator and checker plus AI crawler checker and GEO audit. Input: URLs or domains. One row per domain: llms.txt / llms-full.txt status, robots.txt allow/block for GPTBot, ClaudeBot, PerplexityBot and more, 0-100 AI search score, ready llms.txt, llms-full.txt and JSON-LD. $0.025/domain.
Pricing
from $17.50 / 1,000 ai search geo domain audits
Rating
0.0
(0)
Developer
AusBizStack Intelligence
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
an hour ago
Last modified
Features
Share
Turn a list of websites into one row per domain with an AI search (GEO) readiness score, llms.txt / llms-full.txt checks, which AI crawlers robots.txt allows or blocks, plus a ready-to-publish llms.txt.
Try it now: click Try for free. The input is already filled with 3 sample domains (atlassian.com, shopify.com, xero.com), so you can press Start straight away. A run like this usually finishes in under a minute and costs about $0.075 ($0.025 per domain).
Sample output
Real rows from a run with the prefilled input on 2 November 2026 (a few of the main columns shown; long lists shortened):
| Domain | AI readiness score | llms.txt | llms-full.txt | sitemap.xml | AI crawlers blocked |
|---|---|---|---|---|---|
| atlassian.com | 95 | Yes | Yes | Yes | none |
| shopify.com | 80 | Yes | Yes | Yes | none |
| xero.com | 70 | Yes | No | Yes | none |
Every row has many more fields (see the Output tab and the full example below). Export as CSV, Excel, JSON or Google Sheets, or read it over the Apify API.
Cost example
You pay $0.025 per domain returned. Failed or unreachable domains cost $0.
| You run | You pay |
|---|---|
| 3 domains (the prefilled sample) | $0.075 |
| 100 | $2.50 |
| 1,000 | $25 |
| 10,000 | $250 |
Apify's free plan includes $5 of monthly credit, enough for about 200 domains. Questions? See the FAQ at the bottom of this page.
What does llms.txt Generator & GEO Audit for AI Search do?
Bulk llms.txt generator and GEO (generative engine optimization) audit. Checks robots.txt access for GPTBot, ClaudeBot, PerplexityBot and 6 more AI crawlers, scores AI search readiness, and outputs ready-to-publish llms.txt, llms-full.txt and JSON-LD per domain.
- One row per site, ready to act on: robots.txt verdicts for 9 AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended, CCBot, Applebot-Extended), llms.txt / llms-full.txt / sitemap presence, and a 0–100
aiGeoReadinessScorewith a plain-EnglishidentifiedGapslist. - Generates the fix, not just the finding: returns a draft
llms.txt,llms-full.txtand Organization / WebSite / FAQPage JSON-LD built from the site's own pages, so you can hand a client the files. - Built for bulk and automation: paste hundreds of URLs, or call it from Make, n8n, Clay or an MCP client. $25 / 1,000 audits (Free $25 · Bronze $22.50 · Silver $20 · Gold+ $17.50). Failed or unreachable domains cost $0.
What data can I get with llms.txt Generator & GEO Audit for AI Search?
What This Actor Does in One Run
- Audits
/robots.txtAcross 9 Major AI Crawlers:- Inspects explicit and wildcard rules for
GPTBot,OAI-SearchBot,ChatGPT-User,ClaudeBot,Claude-Web,PerplexityBot,Google-Extended,CCBot, andApplebot-Extended. - Identifies whether a prospect is accidentally blocking ChatGPT Search, Claude, or Perplexity from citing their business.
- Inspects explicit and wildcard rules for
- Verifies Live
/llms.txt,/llms-full.txt&/sitemap.xml:- Checks HTTP 200 status, validates Markdown structure (filtering out soft-404 HTML pages), and counts sections and curated links.
- Extracts Homepage & Subpage Metadata + Existing Schema.org JSON-LD:
- Extracts
<title>, meta descriptions, canonical URLs, H1/H2 hierarchy, top internal navigation/documentation links, social profiles, strictly role-based corporate emails (info@,sales@,support@,bookings@— zero personal emails), phone numbers, and Australian 11-digit ABNs (validated via the ATO modulo-89 checksum). - Inspects existing
application/ld+jsonblocks forOrganization,LocalBusiness,WebSite,FAQPage,Product, andService.
- Extracts
- Scores Every Domain (
aiGeoReadinessScore&agencyPitchScore):- Computes a 0–100
aiGeoReadinessScore, a 0–100agencyPitchScore, and a structuredidentifiedGapschecklist.
- Computes a 0–100
- Auto-Generates Ready-to-Deploy Client Deliverables:
generatedLlmsTxt: Draft/llms.txtMarkdown file following the llms.txt proposal, tailored to the business.generatedLlmsFullTxt: Expanded/llms-full.txtreference corpus for LLM RAG ingestion.generatedJsonLdSchema: Copy-paste<script type="application/ld+json">containing validOrganization+WebSite+FAQPageSchema.org markup.agencyDeliverableSummary: a short audit summary you can share with the site owner.
How to use llms.txt Generator & GEO Audit for AI Search
- Click Try for free (or Start) on this page and sign in to Apify. The free plan's monthly credit covers a first test.
- Paste your domains into Target Websites or Domains to Audit (
targets), one per line. - Leave the other options on their defaults, or switch checks on or off.
- Click Start. Each domain usually finishes in a few seconds.
- Open the Output tab and export to CSV, Excel or JSON, or connect the dataset to Google Sheets, Make, n8n or Clay.
How much does llms.txt Generator & GEO Audit for AI Search cost?
llms.txt Generator & GEO Audit for AI Search uses pay-per-event pricing: $25 per 1,000 domains (Free $25 · Bronze $22.50 · Silver $20 · Gold+ $17.50, by Apify plan), plus a tiny start fee per run. Failed, timed-out or unreachable domains still appear in the dataset with a status and reason, but cost $0. Apify's free plan includes monthly platform credit, which is enough to test llms.txt Generator & GEO Audit for AI Search on a short list.
What counts as a billed result
You are charged one domain-geo-audited event ($0.025) per domain when the homepage loads and the audit completes (status: "ok"). That is the only per-result charge. Apify also adds its standard tiny start fee per run (apify-actor-start).
A domain is still billed when the audit finds no llms.txt, no schema markup or blocked AI crawlers. Those findings are the audit result.
| Row in your dataset | Charged? |
|---|---|
Domain audited successfully (status as above) | Yes, $0.025 |
Duplicate input (status: "duplicate_input") | No, $0 |
Invalid input (status: "invalid_input" / invalid_target) | No, $0 |
Website unreachable after retries, DNS failure, or homepage blocked / timed out / returned 403, 429 or 5xx (status: "fetch_failed" or "unreachable"), or homepage 404 / 410 (status: "http_404") | No, $0 |
Timed out (status: "timeout_error") | No, $0 |
Every input gets a row, so you can always see why something was not charged.
Accuracy: failed fetches (timeouts, network errors, 403 / 429 / 5xx and bot-challenge pages) are retried and then reported as fetch_failed, never as missing; only a real 404 / 410 counts as missing. Unverified checks are left out of the score (aiGeoScoreIsPartial: true), and a domain that is unreachable after retries is not charged.
Accepted input formats
Paste your list as-is. Each line can be any of these, and each resolves to a site to audit:
| You enter | Audited as |
|---|---|
stripe.com (bare domain) | stripe.com |
Stripe.COM, stripe.com/ | stripe.com (case, spaces and trailing slashes are ignored) |
www.stripe.com | www.stripe.com (same domain as stripe.com for de-duplication) |
https://www.stripe.com/pricing/ (full URL) | www.stripe.com (the path is dropped) |
sales@stripe.com or mailto:sales@stripe.com (email) | stripe.com (the part after @) |
Inputs are cleaned and de-duplicated before anything runs or is charged. If two lines point at the same domain, it is audited once, and the extra line comes back as a free row with status: "duplicate_input" and duplicateOf. Anything that is not a domain, URL or email (for example not a domain, localhost or an IP address) comes back as a free status: "invalid_input" row. The existing input fields (targets, domains and urls) keep working exactly as before.
For this Actor the audit runs on the host exactly as you gave it (lower-cased), so https://WWW.Stripe.com/pricing/ audits www.stripe.com and https://blog.example.com/post audits blog.example.com. De-duplication still ignores www., so www.stripe.com and stripe.com count as one domain.
Input example
{"targets": ["https://www.atlassian.com","https://www.shopify.com","https://www.xero.com"],"crawlInternalPages": true,"maxInternalPages": 2,"generateLlmsTxt": true,"generateLlmsFullTxt": true,"generateJsonLdSchema": true,"concurrency": 10,"requestTimeoutSecs": 5}
Output example
One full row from the sample run above (atlassian.com; lists cut to 5 items and long text shortened):
{"inputTarget": "https://www.atlassian.com","domain": "atlassian.com","websiteUrl": "https://www.atlassian.com/","companyName": "Atlassian","countryDetected": "US","status": "ok","pagesInspected": 3,"aiGeoReadinessScore": 95,"agencyPitchScore": 65,"hasLlmsTxt": true,"llmsTxtCharLength": 26567,"llmsTxtLinkCount": 105,"hasLlmsFullTxt": true,"llmsFullTxtCharLength": 35280,"hasRobotsTxt": true,"hasSitemapXml": true,"allowedAiCrawlers": ["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-Web"],"blockedAiCrawlers": [],"aiCrawlerRules": {"GPTBot": "allowed_default","OAI-SearchBot": "allowed_default","ChatGPT-User": "allowed_default","ClaudeBot": "allowed_default","Claude-Web": "allowed_default","PerplexityBot": "allowed_default","Google-Extended": "allowed_default","CCBot": "allowed_default","Applebot-Extended": "allowed_default"},"homepageMetadata": {"title": "Collaboration software for software, IT and business teams | Atlassian","metaDescription": "Atlassian's team collaboration software like Jira, Confluence and Trello help teams organize, discuss, and complete shared work.","canonicalUrl": "https://www.atlassian.com","h1Headings": ["Discover the power of knowing"],"h2Headings": ["Featured apps","Atlassian Collections","Powered by","AI platform","AI ecosystem"],"keyNavigationLinksCount": 15,"keyNavigationLinks": [{"title": "See all apps","url": "https://www.atlassian.com/software","description": "See all apps page on atlassian.com"},{"title": "JiraFlexible project management","url": "https://www.atlassian.com/software/jira","description": "JiraFlexible project management page on atlassian.com"},{"title": "ConfluenceKnowledge, all in one place","url": "https://www.atlassian.com/software/confluence","description": "ConfluenceKnowledge, all in one place page on atlassian.com"},{"title": "Jira Service ManagementDeliver service at high velocity","url": "https://www.atlassian.com/software/jira/service-management","description": "Jira Service ManagementDeliver service at high velocity page on atlassian.com"},{"title": "Teamwork CollectionSupercharge teamwork seamlesslyJiraConfluenceLoom","url": "https://www.atlassian.com/collections/teamwork","description": "Teamwork CollectionSupercharge teamwork seamlesslyJiraConfluenceLoom page on atlassian.com"}]},"existingJsonLdTypes": ["Organization","WebSite","WebPage","SoftwareApplication"],"missingRecommendedJsonLdTypes": ["LocalBusiness","FAQPage","Product","Service"],"jsonLdBlocksCount": 1,"roleEmails": ["press@atlassian.com"],"phones": [],"abns": [],"socialLinks": {},"identifiedGaps": ["Missing FAQPage Schema.org JSON-LD markup for direct Q&A citation in AI Overviews."],"agencyDeliverableSummary": "AI Search (GEO) Readiness Audit for Atlassian (atlassian.com): Current Score 95/100 (Projected 100/100 after deploying attached deliverables). Live /llms.txt detected (26567 chars, 105 links). CRAWLER STATUS: All 9/9 major AI search crawlers are permitted i...","generatedLlmsTxt": "# Atlassian\n\n> Atlassian's team collaboration software like Jira, Confluence and Trello help teams organize, discuss, and complete shared work.\n\n## Core Offerings & Key Highlights\n- Discover the power of knowing\n- Featured apps\n- Atlassian Collections\n- Pow...","generatedLlmsFullTxt": "# Atlassian\n\n> Atlassian's team collaboration software like Jira, Confluence and Trello help teams organize, discuss, and complete shared work.\n\n## Core Offerings & Key Highlights\n- Discover the power of knowing\n- Featured apps\n- Atlassian Collections\n- Pow...","generatedJsonLdSchema": "<script type=\"application/ld+json\">\n{\n \"@context\": \"https://schema.org\",\n \"@graph\": [\n {\n \"@type\": \"Organization\",\n \"@id\": \"https://www.atlassian.com/#organization\",\n \"name\": \"Atlassian\",\n \"url\": \"https://www.atlassian.com/\",\n ...","auditedAt": "2026-11-02T09:56:57.123Z"}
Use cases
- SEO and GEO agencies: audit client and prospect sites for AI search visibility in bulk.
- Marketing teams: generate llms.txt and JSON-LD for many brand sites at once.
- Lead generation: find businesses that block AI crawlers or lack llms.txt and offer a fix.
How do I create an llms.txt file?
Run the Actor on your domain. It reads your pages and returns a ready-to-publish llms.txt (and llms-full.txt) you can upload to your site root.
Is my website blocking ChatGPT or other AI crawlers?
The Actor parses robots.txt for GPTBot, ClaudeBot, PerplexityBot and other AI crawlers and lists which ones are blocked.
What is a GEO audit?
Generative engine optimization (GEO) checks how ready a site is to be read and cited by AI search tools: crawler access, llms.txt, structured data (JSON-LD) and sitemaps. This Actor scores each of these 0–100.
Use from an AI agent (Apify MCP)
Connect Claude, ChatGPT, Cursor or any MCP client to the Apify MCP server at https://mcp.apify.com?tools=ausbizstack/ai-geo-llms-txt-auditor (or add ausbizstack/ai-geo-llms-txt-auditor to an existing Apify MCP connection), then ask:
"Use ausbizstack/ai-geo-llms-txt-auditor to check whether atlassian.com and shopify.com have an llms.txt file and allow GPTBot and ClaudeBot, then give me a draft llms.txt for each."
- Input:
{"targets": ["https://www.atlassian.com", "https://www.shopify.com", "https://www.xero.com"]}(a list; other fields are optional). - Output: one dataset row per domain with a
statusfield, so the agent can tell successes from failures. - Cost: $0.025 per domain on the Free plan (less on paid plans); failed or unreachable rows cost $0.
Integrations (API, Make, n8n, Clay, MCP)
Call llms.txt Generator & GEO Audit for AI Search from the Apify API, schedule it in Apify, or plug it into Make, n8n, Clay, Zapier or an AI agent through the Apify MCP server (mcp.apify.com). Ready-to-import Make and n8n templates that send results to Google Sheets are free to download at https://ausbizstack.github.io/#templates.
Pass your token as a header, never in the URL:
curl -X POST "https://api.apify.com/v2/acts/ausbizstack~ai-geo-llms-txt-auditor/run-sync-get-dataset-items" \-H "Authorization: Bearer YOUR_APIFY_TOKEN" -H "Content-Type: application/json" -d '{...}'
FAQ
What does llms.txt Generator & GEO Audit for AI Search do?
For each website it reads robots.txt, /llms.txt, /llms-full.txt, /sitemap.xml, the homepage and up to a few internal pages, then returns an AI search (GEO) readiness score, the gaps it found, and draft llms.txt / llms-full.txt / JSON-LD files.
Does it change my robots.txt or website? No. It only reads public pages with plain HTTP requests. The generated files are drafts for you to review and publish yourself.
Is llms.txt an official standard? No. llms.txt is a community proposal (llmstxt.org), not a W3C or IETF standard, and AI providers support it unevenly. Treat it as a low-cost extra signal, alongside robots.txt rules and structured data.
How much does it cost, and what counts as a billable audit?
$25 per 1,000 audits (Free $25 · Bronze $22.50 · Silver $20 · Gold+ $17.50). A domain is billed only when it is reached and audited (status: "ok"). Invalid, timed-out or unreachable domains still appear in the dataset but cost $0.
What inputs and output formats are supported?
Pass website URLs or bare domains in targets; everything else is optional. You get one JSON row per site (see the Output tab), exportable as JSON, CSV or Excel or readable via the API.
Are there limits? Who do I contact? It does not run a headless browser, so content rendered only by JavaScript may be missed, and each domain has a time cap of about 20 seconds. For help or a custom version, email ausbizstack@outlook.com.
Acceptable use
This Actor returns publicly available business information only. You are responsible for how you use it. If you send commercial electronic messages to Australian recipients you must comply with the Spam Act 2003 (Cth): have consent (express or inferred), identify yourself accurately, and include a functional unsubscribe. Respect any website notice stating that unsolicited messages are not wanted, and comply with the Privacy Act 1988 (Cth), GDPR and other applicable laws. Do not use this Actor for bulk unsolicited messaging.
Other AusBizStack Actors
- Tech Stack Detector: tech stack detector and website technology lookup with role-based contact enrichment.
- DMARC, SPF & DKIM Checker: bulk DMARC checker, SPF checker and DKIM lookup.
- ABN Lookup Australia: bulk ABN lookup, ACN and GST check for Australian businesses.
- Shopify Apps Detector: Shopify apps detector and store tech stack checker.
- ATS Detector & Careers Page Finder: ATS detector and careers page finder with hiring signals.
- SaaS Pricing Page Scraper: SaaS pricing page scraper for plans, tiers and prices.
- Store Locator Scraper: store locator scraper and franchise location counter.
Questions or a custom version? Email ausbizstack@outlook.com.