Pricing

$50.00 / 1,000 generate llms.txts

📄 llms.txt Generator & Validator — AI Search Visibility

Generate llms.txt files for AI search visibility. Fetches sitemap, validates structure, checks AI bot access (OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended). 2 blue ocean competitors only.

Pricing

$50.00 / 1,000 generate llms.txts

Rating

0.0

(0)

Developer

Alexander Maksimchuk

Actor stats

Bookmarked

Total users

Monthly active users

13 days ago

Last modified

Dockerfile

FROM apify/actor-node-playwright-chrome:20-1.60.0

ENV PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1
ENV PLAYWRIGHT_BROWSERS_PATH=/pw-browsers
ENV NODE_ENV=development

COPY --chown=myuser:myuser package*.json ./

RUN npm install --quiet \
 && echo "Installed npm packages:" \
 && npm list 2>/dev/null | tail -n +2

COPY --chown=myuser:myuser . ./

RUN ./node_modules/.bin/tsc \
 && npm prune --omit=dev \
 && echo "Production packages:" \
 && npm list --omit=dev 2>/dev/null | tail -n +2

ENV NODE_ENV=production
CMD ["node", "dist/main.js"]

package.json

{
  "name": "llms-txt-generator",
  "version": "1.0.0",
  "description": "Generate and validate llms.txt files for AI search engine visibility",
  "main": "dist/main.js",
  "type": "commonjs",
  "scripts": {
    "start": "node dist/main.js"
  },
  "dependencies": {
    "apify": "^3.2.6",
    "playwright": "1.60.0"
  },
  "devDependencies": {
    "typescript": "^5.7.0",
    "@types/node": "^22.0.0"
  }
}

tsconfig.json

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "strict": true,
    "skipLibCheck": true,
    "outDir": "dist",
    "rootDir": "src",
    "declaration": true,
    "resolveJsonModule": true,
    "ignoreDeprecations": "5.0"
  },
  "include": ["src/**/*"]
}

.actor/actor.json

{
    "actorSpecification": 1,
    "name": "llms-txt-generator",
    "title": "\ud83d\udcc4 llms.txt Generator & Validator",
    "description": "Generate llms.txt files for AI search visibility. Fetches sitemap, validates structure, checks AI bot access (OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended). 2 blue ocean competitors only.",
    "version": "1.0",
    "buildTag": "latest",
    "input": "./input_schema.json",
    "dockerfile": "../Dockerfile",
    "output": "./output_schema.json"
}

.actor/dataset_schema.json

{"title":"Dataset","description":"Results dataset","fields":[]}

.actor/input_schema.json

{
    "title": "llms.txt Generator Input",
    "description": "Generate or validate llms.txt for AI search engine visibility.",
    "type": "object",
    "schemaVersion": 1,
    "properties": {
        "url": {
            "title": "Website URL",
            "type": "string",
            "description": "Root URL of the website (e.g., https://example.com). Sitemap will be auto-discovered.",
            "editor": "textfield",
            "prefill": "https://example.com"
        },
        "siteName": {
            "title": "Site name",
            "type": "string",
            "description": "Human-readable site name for llms.txt header.",
            "editor": "textfield",
            "prefill": "Example Site"
        },
        "siteDescription": {
            "title": "Site description",
            "type": "string",
            "description": "One-sentence description of what the site does.",
            "editor": "textfield",
            "prefill": "A website about..."
        },
        "maxUrls": {
            "title": "Max URLs in llms.txt",
            "type": "integer",
            "description": "Maximum number of URLs to include. Keep under 200 for best results.",
            "default": 50,
            "minimum": 5,
            "maximum": 500,
            "editor": "number"
        },
        "validateOnly": {
            "title": "Validate only (don't generate)",
            "type": "boolean",
            "description": "If true, only check for existing llms.txt and AI bot access. Don't generate new file.",
            "default": false
        }
    },
    "required": ["url"]
}

.actor/output_schema.json

{
    "$schema": "https://apify.com/schemas/v1/output.ide.json",
    "actorOutputSchemaVersion": 1,
    "title": "llms.txt Generation Results",
    "properties": {
        "results": {
            "type": "string",
            "title": "Generated llms.txt Files",
            "template": "{{links.apiDefaultDatasetUrl}}/items"
        }
    }
}

src/main.ts

1import { Actor, log } from 'apify';
2import { chromium, type Browser } from 'playwright';
3
4// --- Types ---
5
6interface ActorInput {
7    url: string;
8    siteName?: string;
9    siteDescription?: string;
10    maxUrls?: number;
11    validateOnly?: boolean;
12}
13
14interface SitemapUrl {
15    loc: string;
16    lastmod?: string;
17    priority?: string;
18    changefreq?: string;
19}
20
21interface RobotsAnalysis {
22    raw: string;
23    aiBots: {
24        'OAI-SearchBot': boolean;
25        'PerplexityBot': boolean;
26        'ClaudeBot': boolean;
27        'Google-Extended': boolean;
28        'GPTBot': boolean;
29        'anthropic-ai': boolean;
30    };
31    blockingTraining: string[];
32    allowingCitation: string[];
33    hasWildcard: boolean;
34}
35
36interface LlmsTxtAnalysis {
37    exists: boolean;
38    content: string;
39    lineCount: number;
40    hasH1: boolean;
41    hasBlockquote: boolean;
42    sectionCount: number;
43    linkCount: number;
44    issues: string[];
45}
46
47interface AiDiscoveryCheck {
48    hasAiTxt: boolean;
49    aiTxtContent: string;
50    hasSummaryJson: boolean;
51    hasFaqJson: boolean;
52    summaryJsonContent: any;
53}
54
55interface PageMeta {
56    title: string;
57    description: string;
58    ogTitle: string;
59    ogDescription: string;
60    canonical: string;
61    lang: string;
62}
63
64interface SitemapEntry {
65    url: string;
66    title: string;
67    description: string;
68    section: string;
69}
70
71interface ActorOutput {
72    websiteUrl: string;
73    generatedAt: string;
74    llmsTxt: string;
75    llmsTxtUrl: string;
76    sitemapUrlsFound: number;
77    sitemapUrlsIncluded: number;
78    robotsAnalysis: RobotsAnalysis;
79    existingLlmsTxt: LlmsTxtAnalysis;
80    aiDiscovery: AiDiscoveryCheck;
81    sitemapEntries: SitemapEntry[];
82    score: number;
83    grade: string;
84    recommendations: string[];
85    llmsFullTxt: string;
86}
87
88// --- Robots.txt parsing ---
89
90async function fetchRobotsTxt(baseUrl: string): Promise<string> {
91    const robotsUrl = `${baseUrl}/robots.txt`;
92    try {
93        const resp = await fetch(robotsUrl, { redirect: 'follow' });
94        if (!resp.ok) {
95            log.warning(`robots.txt returned ${resp.status}`);
96            return '';
97        }
98        return await resp.text();
99    } catch (error) {
100        log.warning(`Failed to fetch robots.txt: ${(error as Error).message}`);
101        return '';
102    }
103}
104
105function parseRobotsForAiBots(raw: string): RobotsAnalysis {
106    const bots: RobotsAnalysis['aiBots'] = {
107        'OAI-SearchBot': false,
108        'PerplexityBot': false,
109        'ClaudeBot': false,
110        'Google-Extended': false,
111        'GPTBot': false,
112        'anthropic-ai': false,
113    };
114
115    const blockingTraining: string[] = [];
116    const allowingCitation: string[] = [];
117
118    // Parse robots.txt: look for User-agent + Disallow/Allow rules
119    const lines = raw.split('\n');
120    let currentUserAgent = '';
121
122    // Track which bots have explicit rules
123    const botRules: Record<string, { disallowAll: boolean; allowAll: boolean; hasRules: boolean }> = {};
124
125    for (const line of lines) {
126        const trimmed = line.trim();
127        const lower = trimmed.toLowerCase();
128
129        if (lower.startsWith('user-agent:')) {
130            currentUserAgent = trimmed.substring(11).trim();
131            if (!botRules[currentUserAgent]) {
132                botRules[currentUserAgent] = { disallowAll: false, allowAll: false, hasRules: true };
133            }
134        } else if (lower.startsWith('disallow:')) {
135            const path = trimmed.substring(9).trim();
136            if (path === '/' || path === '/*') {
137                if (botRules[currentUserAgent]) {
138                    botRules[currentUserAgent].disallowAll = true;
139                }
140            }
141        } else if (lower.startsWith('allow:')) {
142            const path = trimmed.substring(6).trim();
143            if (path === '/' || path === '/*') {
144                if (botRules[currentUserAgent]) {
145                    botRules[currentUserAgent].allowAll = true;
146                }
147            }
148        }
149    }
150
151    // Check each AI bot
152    const citationBots = ['OAI-SearchBot', 'PerplexityBot', 'ClaudeBot', 'Google-Extended'];
153    const trainingBots = ['GPTBot', 'anthropic-ai'];
154
155    for (const bot of citationBots) {
156        const rules = botRules[bot];
157        if (rules) {
158            bots[bot as keyof typeof bots] = !rules.disallowAll || rules.allowAll;
159            if (!rules.disallowAll || rules.allowAll) {
160                allowingCitation.push(bot);
161            }
162        } else {
163            // No explicit rule — check wildcard
164            const wildcard = botRules['*'];
165            if (wildcard && wildcard.disallowAll && !wildcard.allowAll) {
166                bots[bot as keyof typeof bots] = false;
167            } else {
168                // Default: allowed (no rule = allowed)
169                bots[bot as keyof typeof bots] = true;
170                allowingCitation.push(bot);
171            }
172        }
173    }
174
175    for (const bot of trainingBots) {
176        const rules = botRules[bot];
177        if (rules && rules.disallowAll) {
178            blockingTraining.push(bot);
179        }
180    }
181
182    const hasWildcard = botRules['*']?.hasRules || false;
183
184    return { raw, aiBots: bots, blockingTraining, allowingCitation, hasWildcard };
185}
186
187// --- llms.txt analysis ---
188
189async function fetchLlmsTxt(baseUrl: string): Promise<LlmsTxtAnalysis> {
190    const llmsUrl = `${baseUrl}/llms.txt`;
191    try {
192        const resp = await fetch(llmsUrl, { redirect: 'follow' });
193        if (!resp.ok) {
194            return {
195                exists: false,
196                content: '',
197                lineCount: 0,
198                hasH1: false,
199                hasBlockquote: false,
200                sectionCount: 0,
201                linkCount: 0,
202                issues: ['llms.txt not found'],
203            };
204        }
205        const content = await resp.text();
206        const lines = content.split('\n');
207
208        const hasH1 = lines.some(l => l.startsWith('# ') && !l.startsWith('## '));
209        const hasBlockquote = lines.some(l => l.startsWith('> '));
210        const sectionCount = lines.filter(l => l.startsWith('## ')).length;
211        const linkCount = lines.filter(l => l.match(/^\s*\[.+?\]:\s*\S+/) || l.match(/^\s*-\s*\[.+?\]\(.+?\)/)).length;
212
213        const issues: string[] = [];
214        if (!hasH1) issues.push('Missing H1 site name header');
215        if (!hasBlockquote) issues.push('Missing blockquote description');
216        if (lines.length > 200) issues.push('File exceeds 200 line recommendation');
217        if (linkCount < 5) issues.push(`Only ${linkCount} links — add more key pages`);
218
219        return {
220            exists: true,
221            content,
222            lineCount: lines.length,
223            hasH1,
224            hasBlockquote,
225            sectionCount,
226            linkCount,
227            issues,
228        };
229    } catch (error) {
230        return {
231            exists: false,
232            content: '',
233            lineCount: 0,
234            hasH1: false,
235            hasBlockquote: false,
236            sectionCount: 0,
237            linkCount: 0,
238            issues: [`Failed to fetch: ${(error as Error).message}`],
239        };
240    }
241}
242
243// --- AI discovery endpoints ---
244
245async function checkAiDiscovery(baseUrl: string): Promise<AiDiscoveryCheck> {
246    const result: AiDiscoveryCheck = {
247        hasAiTxt: false,
248        aiTxtContent: '',
249        hasSummaryJson: false,
250        hasFaqJson: false,
251        summaryJsonContent: null,
252    };
253
254    // Check /.well-known/ai.txt
255    try {
256        const resp = await fetch(`${baseUrl}/.well-known/ai.txt`, { redirect: 'follow' });
257        if (resp.ok) {
258            result.hasAiTxt = true;
259            result.aiTxtContent = await resp.text();
260        }
261    } catch { /* not mandatory */ }
262
263    // Check /ai/summary.json
264    try {
265        const resp = await fetch(`${baseUrl}/ai/summary.json`, { redirect: 'follow' });
266        if (resp.ok) {
267            result.hasSummaryJson = true;
268            result.summaryJsonContent = await resp.json();
269        }
270    } catch { /* not mandatory */ }
271
272    // Check /ai/faq.json
273    try {
274        const resp = await fetch(`${baseUrl}/ai/faq.json`, { redirect: 'follow' });
275        if (resp.ok) {
276            result.hasFaqJson = true;
277        }
278    } catch { /* not mandatory */ }
279
280    return result;
281}
282
283// --- Sitemap fetching ---
284
285async function fetchSitemapUrls(baseUrl: string, maxUrls: number): Promise<SitemapUrl[]> {
286    const sitemapUrls: SitemapUrl[] = [];
287
288    // Try common sitemap locations
289    const sitemapCandidates = [
290        `${baseUrl}/sitemap.xml`,
291        `${baseUrl}/sitemap_index.xml`,
292        `${baseUrl}/sitemaps.xml`,
293    ];
294
295    // Also try robots.txt for sitemap reference
296    const robotsRaw = await fetchRobotsTxt(baseUrl);
297    const sitemapLine = robotsRaw.split('\n').find(l => l.toLowerCase().startsWith('sitemap:'));
298    if (sitemapLine) {
299        const sitemapUrl = sitemapLine.substring(8).trim();
300        sitemapCandidates.unshift(sitemapUrl);
301    }
302
303    for (const sitemapUrl of sitemapCandidates) {
304        try {
305            const resp = await fetch(sitemapUrl, { redirect: 'follow' });
306            if (!resp.ok) continue;
307
308            const xml = await resp.text();
309
310            // Parse XML sitemap
311            const urlMatches = xml.matchAll(/<url>\s*<loc>([^<]+)<\/loc>(?:\s*<lastmod>([^<]*)<\/lastmod>)?(?:\s*<priority>([^<]*)<\/priority>)?(?:\s*<changefreq>([^<]*)<\/changefreq>)?\s*<\/url>/g);
312
313            for (const match of urlMatches) {
314                if (sitemapUrls.length >= maxUrls * 2) break; // Over-fetch for dedup
315                sitemapUrls.push({
316                    loc: match[1].trim(),
317                    lastmod: match[2]?.trim(),
318                    priority: match[3]?.trim(),
319                    changefreq: match[4]?.trim(),
320                });
321            }
322
323            // Check for sitemap index (nested sitemaps)
324            const subMatches = xml.matchAll(/<sitemap>\s*<loc>([^<]+)<\/loc>/g);
325            for (const match of subMatches) {
326                if (sitemapUrls.length >= maxUrls * 2) break;
327                // Fetch sub-sitemap
328                try {
329                    const subResp = await fetch(match[1].trim(), { redirect: 'follow' });
330                    if (subResp.ok) {
331                        const subXml = await subResp.text();
332                        const subUrlMatches = subXml.matchAll(/<url>\s*<loc>([^<]+)<\/loc>/g);
333                        for (const subMatch of subUrlMatches) {
334                            if (sitemapUrls.length >= maxUrls * 2) break;
335                            sitemapUrls.push({ loc: subMatch[1].trim() });
336                        }
337                    }
338                } catch { /* skip failed sub-sitemap */ }
339            }
340
341            if (sitemapUrls.length > 0) break; // Found URLs, stop trying candidates
342        } catch {
343            continue;
344        }
345    }
346
347    return sitemapUrls;
348}
349
350// --- Page metadata extraction ---
351
352async function extractPageMeta(browser: Browser, url: string): Promise<PageMeta> {
353    const context = await browser.newContext({ viewport: { width: 1280, height: 720 } });
354    const page = await context.newPage();
355
356    try {
357        await page.goto(url, { timeout: 15000, waitUntil: 'domcontentloaded' });
358
359        const meta = await page.evaluate(() => {
360            const getMeta = (name: string): string => {
361                const el = document.querySelector(`meta[name="${name}"]`) || document.querySelector(`meta[property="${name}"]`);
362                return el?.getAttribute('content') || '';
363            };
364
365            return {
366                title: document.title || '',
367                description: getMeta('description'),
368                ogTitle: getMeta('og:title'),
369                ogDescription: getMeta('og:description'),
370                canonical: (document.querySelector('link[rel="canonical"]') as HTMLLinkElement)?.href || '',
371                lang: document.documentElement.lang || '',
372            };
373        });
374
375        return meta;
376    } catch {
377        return { title: '', description: '', ogTitle: '', ogDescription: '', canonical: '', lang: '' };
378    } finally {
379        await context.close();
380    }
381}
382
383// --- llms.txt generation ---
384
385function categorizeUrl(url: string, baseUrl: string): string {
386    const path = url.replace(baseUrl, '').toLowerCase();
387
388    if (path === '/' || path === '') return 'Home';
389    if (path.includes('/about')) return 'About';
390    if (path.includes('/pricing') || path.includes('/plans')) return 'Pricing';
391    if (path.includes('/blog') || path.includes('/news') || path.includes('/article')) return 'Blog';
392    if (path.includes('/product')) return 'Products';
393    if (path.includes('/service')) return 'Services';
394    if (path.includes('/contact')) return 'Contact';
395    if (path.includes('/faq') || path.includes('/help')) return 'Help & FAQ';
396    if (path.includes('/doc') || path.includes('/api') || path.includes('/guide')) return 'Documentation';
397    if (path.includes('/case') || path.includes('/portfolio')) return 'Case Studies';
398    if (path.includes('/team')) return 'Team';
399    if (path.includes('/privacy') || path.includes('/terms') || path.includes('/legal')) return 'Legal';
400    if (path.includes('/feature') || path.includes('/how-to') || path.includes('/tutorial')) return 'Guides';
401    return 'Pages';
402}
403
404function generateLlmsTxt(
405    siteName: string,
406    siteDescription: string,
407    baseUrl: string,
408    entries: SitemapEntry[]
409): string {
410    const lines: string[] = [];
411
412    // H1: site name
413    lines.push(`# ${siteName}`);
414    lines.push('');
415
416    // Blockquote: description
417    lines.push(`> ${siteDescription}`);
418    lines.push('');
419
420    // Group by section
421    const sections: Record<string, SitemapEntry[]> = {};
422    for (const entry of entries) {
423        if (!sections[entry.section]) sections[entry.section] = [];
424        sections[entry.section].push(entry);
425    }
426
427    // Output sections
428    for (const [sectionName, sectionEntries] of Object.entries(sections)) {
429        lines.push(`## ${sectionName}`);
430        lines.push('');
431        for (const entry of sectionEntries) {
432            const title = entry.title || entry.url;
433            const desc = entry.description ? `: ${entry.description}` : '';
434            lines.push(`- [${title}](${entry.url})${desc}`);
435        }
436        lines.push('');
437    }
438
439    lines.push('## Links');
440    lines.push('');
441    lines.push(`- [XML Sitemap](${baseUrl}/sitemap.xml)`);
442    lines.push(`- [robots.txt](${baseUrl}/robots.txt)`);
443
444    return lines.join('\n');
445}
446
447// --- Scoring ---
448
449function calculateScore(
450    robots: RobotsAnalysis,
451    llmsTxt: LlmsTxtAnalysis,
452    aiDiscovery: AiDiscoveryCheck
453): { score: number; grade: string; recommendations: string[] } {
454    let score = 0;
455    const recommendations: string[] = [];
456
457    // Robots.txt AI bot access (40 points)
458    const citationBotsAllowed = Object.entries(robots.aiBots).filter(([bot, allowed]) =>
459        ['OAI-SearchBot', 'PerplexityBot', 'ClaudeBot', 'Google-Extended'].includes(bot) && allowed
460    ).length;
461    score += (citationBotsAllowed / 4) * 25;
462
463    if (!robots.aiBots['OAI-SearchBot']) recommendations.push('Allow OAI-SearchBot in robots.txt for ChatGPT Search citations');
464    if (!robots.aiBots['PerplexityBot']) recommendations.push('Allow PerplexityBot in robots.txt for Perplexity citations');
465    if (!robots.aiBots['ClaudeBot']) recommendations.push('Allow ClaudeBot in robots.txt for Claude web citations');
466    if (!robots.aiBots['Google-Extended']) recommendations.push('Allow Google-Extended in robots.txt for Gemini AI Overviews');
467
468    // Block training bots while allowing citation (5 points)
469    if (robots.blockingTraining.length > 0) {
470        score += 5;
471    } else {
472        recommendations.push('Consider blocking GPTBot and anthropic-ai (training) while allowing citation bots');
473    }
474
475    // llms.txt (40 points)
476    if (llmsTxt.exists) {
477        score += 10;
478        if (llmsTxt.hasH1) score += 5;
479        if (llmsTxt.hasBlockquote) score += 5;
480        score += Math.min(llmsTxt.sectionCount * 3, 12);
481        score += Math.min(llmsTxt.linkCount * 0.5, 8);
482
483        for (const issue of llmsTxt.issues) {
484            if (!issue.includes('not found')) recommendations.push(`Fix llms.txt: ${issue}`);
485        }
486    } else {
487        score += 0;
488        recommendations.push('Create llms.txt file — this Actor can generate one for you');
489    }
490
491    // AI discovery endpoints (15 points)
492    if (aiDiscovery.hasAiTxt) score += 5;
493    if (aiDiscovery.hasSummaryJson) score += 5;
494    if (aiDiscovery.hasFaqJson) score += 5;
495
496    if (!aiDiscovery.hasAiTxt) recommendations.push('Add /.well-known/ai.txt for AI agent discovery');
497    if (!aiDiscovery.hasSummaryJson) recommendations.push('Add /ai/summary.json for AI summary access');
498
499    score = Math.min(100, Math.round(score));
500
501    let grade: string;
502    if (score >= 86) grade = 'Excellent';
503    else if (score >= 68) grade = 'Good';
504    else if (score >= 36) grade = 'Foundation';
505    else grade = 'Critical';
506
507    return { score, grade, recommendations };
508}
509
510// --- Main ---
511
512async function main() {
513    await Actor.init();
514
515    const input = (await Actor.getInput()) as ActorInput;
516
517    if (!input?.url) {
518        log.error('No URL provided');
519        await Actor.exit('No URL provided', { exitCode: 1 });
520        return;
521    }
522
523    const baseUrl = input.url.replace(/\/$/, '');
524    const siteName = input.siteName || new URL(baseUrl).hostname;
525    const siteDescription = input.siteDescription || `Website at ${baseUrl}`;
526    const maxUrls = input.maxUrls || 50;
527    const validateOnly = input.validateOnly || false;
528
529    log.info(`Processing: ${baseUrl} (validateOnly=${validateOnly})`);
530
531    // 1. Fetch + analyze robots.txt
532    log.info('Fetching robots.txt...');
533    const robotsRaw = await fetchRobotsTxt(baseUrl);
534    const robotsAnalysis = parseRobotsForAiBots(robotsRaw);
535
536    log.info(`robots.txt: ${Object.entries(robotsAnalysis.aiBots).filter(([_, v]) => v).length}/6 AI bots allowed`);
537    log.info(`Training bots blocked: ${robotsAnalysis.blockingTraining.join(', ') || 'none'}`);
538
539    // 2. Check existing llms.txt
540    log.info('Checking existing llms.txt...');
541    const existingLlmsTxt = await fetchLlmsTxt(baseUrl);
542
543    if (existingLlmsTxt.exists) {
544        log.info(`llms.txt found: ${existingLlmsTxt.lineCount} lines, ${existingLlmsTxt.linkCount} links, ${existingLlmsTxt.sectionCount} sections`);
545    } else {
546        log.info('llms.txt not found');
547    }
548
549    // 3. Check AI discovery endpoints
550    log.info('Checking AI discovery endpoints...');
551    const aiDiscovery = await checkAiDiscovery(baseUrl);
552    log.info(`AI discovery: ai.txt=${aiDiscovery.hasAiTxt}, summary.json=${aiDiscovery.hasSummaryJson}, faq.json=${aiDiscovery.hasFaqJson}`);
553
554    // 4. Fetch sitemap
555    log.info('Fetching sitemap...');
556    const sitemapUrls = await fetchSitemapUrls(baseUrl, maxUrls);
557    log.info(`Found ${sitemapUrls.length} URLs in sitemap`);
558
559    // 5. Generate llms.txt (if not validate-only)
560    let generatedLlmsTxt = '';
561    let sitemapEntries: SitemapEntry[] = [];
562
563    if (!validateOnly) {
564        log.info('Extracting page metadata for llms.txt generation...');
565
566        const browser = await chromium.launch({
567            headless: true,
568            args: ['--disable-gpu', '--no-sandbox', '--disable-setuid-sandbox'],
569        });
570
571        try {
572            // Dedup + limit URLs
573            const seen = new Set<string>();
574            const uniqueUrls = sitemapUrls
575                .map(s => s.loc)
576                .filter(url => {
577                    if (seen.has(url)) return false;
578                    seen.add(url);
579                    return true;
580                })
581                .slice(0, maxUrls);
582
583            for (const url of uniqueUrls) {
584                log.info(`  Extracting: ${url}`);
585                const meta = await extractPageMeta(browser, url);
586                sitemapEntries.push({
587                    url,
588                    title: meta.ogTitle || meta.title || url,
589                    description: meta.ogDescription || meta.description || '',
590                    section: categorizeUrl(url, baseUrl),
591                });
592            }
593        } finally {
594            await browser.close();
595        }
596
597        generatedLlmsTxt = generateLlmsTxt(siteName, siteDescription, baseUrl, sitemapEntries);
598        log.info(`Generated llms.txt: ${generatedLlmsTxt.split('\n').length} lines, ${sitemapEntries.length} entries`);
599    }
600
601    // 6. Calculate score
602    const { score, grade, recommendations } = calculateScore(robotsAnalysis, existingLlmsTxt, aiDiscovery);
603
604    // 7. Build output
605    const output: ActorOutput = {
606        websiteUrl: baseUrl,
607        generatedAt: new Date().toISOString(),
608        llmsTxt: generatedLlmsTxt,
609        llmsTxtUrl: `${baseUrl}/llms.txt`,
610        sitemapUrlsFound: sitemapUrls.length,
611        sitemapUrlsIncluded: sitemapEntries.length,
612        robotsAnalysis,
613        existingLlmsTxt,
614        aiDiscovery,
615        sitemapEntries,
616        score,
617        grade,
618        recommendations,
619        llmsFullTxt: generatedLlmsTxt,
620    };
621
622    // Push to dataset
623    await Actor.pushData({
624        url: baseUrl,
625        score,
626        grade,
627        aiBotsAllowed: Object.entries(robotsAnalysis.aiBots).filter(([_, v]) => v).map(([k]) => k),
628        trainingBotsBlocked: robotsAnalysis.blockingTraining,
629        llmsTxtExists: existingLlmsTxt.exists,
630        llmsTxtIssues: existingLlmsTxt.issues,
631        sitemapUrlsFound: sitemapUrls.length,
632        sitemapUrlsIncluded: sitemapEntries.length,
633        recommendations,
634    });
635
636    // Charge for the run
637    await Actor.charge({ eventName: 'GENERATE', count: 1 });
638
639    // Set OUTPUT
640    const kvStore = await Actor.openKeyValueStore();
641    await kvStore.setValue('OUTPUT', output);
642
643    // Also save llms.txt as downloadable file
644    if (generatedLlmsTxt) {
645        await kvStore.setValue('llms.txt', generatedLlmsTxt, { contentType: 'text/plain' });
646    }
647
648    log.info(`Complete: Score ${score}/100 (${grade}), ${recommendations.length} recommendations`);
649
650    await Actor.exit();
651}
652
653main().catch(async (error) => {
654    console.error('Fatal error:', error);
655    await Actor.exit('Fatal error', { exitCode: 1 });
656});

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.

Cathal Judge

LLMs.txt AI Discovery Auditor

glowing_glove/llms-txt-ai-discovery-auditor

Check websites for llms.txt, llms-full.txt, robots AI crawler rules, sitemap hints, and AI search readiness gaps.

Ushba Khan

llms.txt Generator — Make Any Website AI-Readable

darknezz/llms-txt-generator

Crawl any website and generate llms.txt and llms-full.txt files following the llms.txt standard. BFS crawler with URL filtering and markdown/plaintext output for AI readiness, SEO for AI, docs portals and archiving. Agent-ready via Apify API and MCP.

Oaida Adrian

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit

The Howlers

LLMs.txt Discovery

toronto_777/llms-txt-discovery

Find public llms.txt, llms-full.txt, and AI-readable documentation links for company domains.

Steven Feng

LLMs.txt Sitemap Generator

silicon1235/llms-txt-sitemap-generator

Generate an llms.txt draft and priority AI-readable link inventory from public sitemaps.

X L

LLMs.txt Generator

onescales/the-llms-txt-generator

The most powerful LLMs.txt Generator tool online. Generates LLMs.txt , llms-full.txt and markdown .md files within seconds! Get your website discovered, and recommended by ChatGPT, Claude, Google Gemini, Perplexity, Grok, and every AI. (Great for AEO, AIO, GEO, AI SEO) Made by Hi LLMs

One Scales

123

5.0

LLMs.txt & llms-full.txt Generator for Any Website

fayoussef/llms-txt-generator

Generate a spec-compliant llms.txt and llms-full.txt for any website in one click. Crawls your sitemap or site, extracts titles, descriptions and full Markdown content, and outputs ready-to-upload files that make your site visible to ChatGPT, Claude, Perplexity and other AI search engines.