llms.txt Generator & Checker (plus llms-full.txt)

LLMs.txt File Generator

justa/llms-txt-file-generator

Generate an llms.txt file from a website sitemap. Crawls all URLs, extracts titles and meta descriptions, and creates a Markdown-formatted file following the llms.txt specification. Upload then the output of your file directly on your website (Webflow, Wordpress etc.)

Website & Page to Markdown for RAG and LLMs

humble-echidna/page-to-markdown

Turn web pages or whole sites into clean Markdown for RAG and LLMs: main content only, with title, links and word count. Whole-site mode reads the sitemap first, can return only changed pages, and splits pages into heading-based chunks. Respects robots.txt and AI opt-outs. No browser.

LLMs.txt Checker - AI Readiness Scanner

alizarin_refrigerator-owner/llms-txt-checker

Batch check websites for llms.txt files & identify AI optimization opportunities. A emerging standard that websites use to communicate with AI systems. It helps LLMs understand, What the company does, How to present information about the business, Contact information & key pages Brand guidelines

llms.txt Generator: llms-full.txt for Any Website (AI SEO)

fayoussef/llms-txt-generator

Generate a spec-compliant llms.txt and llms-full.txt for any website in one run. Crawls your sitemap or internal links, extracts every page's title, meta description and full Markdown content, and outputs two ready-to-upload files for ChatGPT, Claude, Perplexity and Gemini.

Website to Markdown for LLM — Crawl Website to Markdown

nexgendata/website-content-crawler

Website to markdown for LLM use: crawl website to markdown, text or HTML — entire sites, JS-rendered pages included. A website to markdown crawler for RAG pipelines, LLM context and AI training. Firecrawl & Jina alternative. Pay per page.

Firecrawl Search - LLM-ready content

alizarin_refrigerator-owner/firecrawl-search---llm-ready-content

Search the web and get clean, LLM-ready content in one API call. Powered by Firecrawl's /v1/search endpoint. Returns markdown, HTML, or extracted data. Perfect for SEO research, competitor analysis, and AI training data collection

Real-time knowledge for LLMs

ai-developer/real-time-knowledge-for-llms

Unlock the Full Potential of Your LLM with Real-Time Web Knowledge! Say goodbye to outdated responses, misinformation, and hallucinations. Now you can ground your Language Model with the freshest information from the web. NO API NEEDED!

LLM Web Scraper

incredible_moment/llm-scraper

Turn any website into structured JSON using AI. Supports OpenAI GPT-4 and Anthropic Claude. Built in Rust to minimize compute costs while waiting for LLM responses. Extract data without selectors.

Website Content Crawler — Markdown for LLM & RAG

brilliant_gum/web-content-crawler

Deep-crawl websites and extract LLM-ready Markdown with OG tags, JSON-LD, author, dates, token estimates, native RAG chunking, language filtering, content-hash dedup, and per-page error reporting. Enforced timeouts. Zero silent failures.

LLMs.txt AI Discovery Auditor

glowing_glove/llms-txt-ai-discovery-auditor

Check websites for llms.txt, llms-full.txt, robots AI crawler rules, sitemap hints, and AI search readiness gaps.