Site to llms.txt Generator

llms.txt Generator - Make Your Site AI-Discoverable

dltik/llms-txt-generator

Generate a standards-compliant llms.txt file for any website. Crawls your sitemap, extracts page titles & descriptions, groups them into sections, and outputs a clean llms.txt that helps ChatGPT, Claude, Gemini & Perplexity understand your site. Optional AI enrichment.

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit

LLMs.txt Generator

onescales/the-llms-txt-generator

The most powerful LLMs.txt Generator tool online. Generates LLMs.txt , llms-full.txt and markdown .md files within seconds! Get your website discovered, and recommended by ChatGPT, Claude, Google Gemini, Perplexity, Grok, and every AI. (Great for AEO, AIO, GEO, AI SEO) Made by Hi LLMs

llms-txt-converter is an Apify Actor that crawls any documentation site and outputs the two GEO-standard files (llms.txt and llms-full.txt) used by ChatGPT, Claude, Perplexity, Gemini, and Bing Copilot to find and cite your docs.

Sitemap URL Extractor - List All URLs in a Sitemap

dltik/sitemap-url-extractor

Extract every URL from any XML sitemap, with lastmod, changefreq and priority. Resolves sitemap indexes recursively. Pass a sitemap.xml or just a site root to auto-discover its sitemaps. Pure HTTP, no browser — fast and cheap.

LLMs.txt File Generator

justa/llms-txt-file-generator

Generate an llms.txt file from a website sitemap. Crawls all URLs, extracts titles and meta descriptions, and creates a Markdown-formatted file following the llms.txt specification. Upload then the output of your file directly on your website (Webflow, Wordpress etc.)

Documentation Crawler for RAG

liquid_bark/docs-crawler-for-rag

Specialized crawler for developer documentation sites. Detects frameworks (Docusaurus, GitBook, ReadTheDocs, MkDocs, Sphinx), extracts clean content, and outputs semantically chunked Markdown optimized for RAG pipelines.

Website Metadata Extractor(sitemap, socialLinks, robotsTxt)

codescraper/website-metadata-extractor

A very fast metadata extractor to get all meta tags, robots.txt, sitemaps, social links, H1s, word count, and JSON-LD data. Also provides technology detection for a full analysis. Get your data fast for just $3/month.

AI agents that need web data without anti-bot headaches. 20 tools for API-based web scraping, crawl, search, and extract — no proxy rotation, no stealth needed.

AI Technical SEO MCP Server

alizarin_refrigerator-owner/ai-technical-seo-mcp-server

AI-ready technical SEO auditing with 7 specialized tools. Check robots.txt, llms.txt, generate sitemaps, run Lighthouse & GTmetrix audits, test rich results, and perform comprehensive technical SEO audits - all through a unified MCP interface.