Website to Markdown: Sitemap Crawler for RAG + llms.txt

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

Website to Markdown Scraper for AI & LLMs

quaking_pail/ai-website-content-markdown-scraper

Crawl a website and get clean Markdown of each page's main content, with URL and title. Ready for LLMs, RAG, vector databases and AI agents. Headless Chrome, same-domain crawl, API access.

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.

Web-to-Markdown Generator for AI & RAG Pipelines

profitstack/web-to-markdown-generator-for-ai-rag-pipelines

Convert any website into clean, heading-based chunking, LLM-ready Markdown for RAG and AI agents.

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Greenhouse Jobs Scraper: Workday, Lever + Visa Flags

chimey_forearm/ats-jobs-scraper

Greenhouse jobs and Workday jobs scraper, plus Lever, Ashby and 4 more ATSs in one schema. Visa sponsorship, US citizenship, ITAR and clearance flags, salary, remote, job alerts. No login. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Universal Markdown Scraper for LLMs

botflowtech/universal-markdown-scraper-for-llms

Universal Markdown Scraper for LLMs

Paste a homepage or sitemap and get clean Markdown, metadata, JSONL chunks, and source URLs for RAG at a low per-page price.

Website to Markdown API & Crawler – RAG/LLM

signal_lab/website-to-markdown-crawler

Website-to-Markdown API and crawler for RAG and LLM datasets. Extract clean Markdown, SEO metadata, links, JSON-LD, sitemaps, and robots.txt-compliant pages.

Website Content Extractor — Pages to Markdown for LLM/RAG

taroyamada/website-content-extractor

Turn docs, help centers and pricing pages into clean Markdown for LLM and RAG: one row per URL with main text, title, word count and a content-quality score. No API key or login.