Website Content Crawler — Clean Markdown for AI

Crawl any website and get clean markdown, text, or HTML per page, ready for RAG pipelines, chatbots, and LLM fine tuning. Plain HTTP, no browser, no API key. Pay per page.

Website Content Crawler Pro — Markdown for AI & RAG

hipersoft/web-content-crawler-pro

Crawl any website into clean Markdown or text for LLMs, RAG and AI. HTTP + JS-browser modes, main-content (readability) extraction, sitemaps, URL glob filters, cookie-banner removal and CSS selectors. No setup.

Download a public, server-rendered website as a ZIP with HTML, CSS, JavaScript, images, fonts, and other static assets. Crawl same-host links with configurable depth and limits. Best for landing pages, portfolios, blogs, and documentation; JavaScript-only apps are not rendered.

Website Content Crawler - Markdown, 45 Fields

parseforge/website-content-crawler

Crawls websites from start URLs with depth, domain and URL-pattern limits. 45 fields per page: Markdown and plain text, word count, title and meta tags, headings, Open Graph and JSON-LD, author and dates, emails, phones and social links, detected technologies, links and HTTP status. Ready for LLMs.

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Cloudflare & Anti-Bot Fetcher – Gap-Filler for Crawlers

vietaro/cloudflare-bypass-scraper

Fast HTTP by default, automatic Cloudflare/anti-bot bypass when blocked. The gap-filler for Website Content Crawler, RAG Web Browser, or any crawler that got blocked — retries just their failed pages and returns clean, LLM-ready Markdown.

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

Crawl any website from a seed URL and extract clean Markdown content, ready for LLM training data, RAG pipelines, and vector databases. Set crawl depth, page limits, and domain scope.

Extract clean, readable markdown content from any web page. Removes navigation, ads, footers, and boilerplate — outputs structured markdown optimized for LLM training, RAG pipelines, and AI agents. Pay-per-event pricing. $0.002/page.

Website Content Crawler: Markdown for AI and RAG

automa-flow/website-content-crawler

Crawl sites you own or are authorized to process into clean Markdown for AI and RAG. Keep headings, tables and code, emit source-linked chunks, and cap pages with HTTP-first limits.

Website Contact & Email Extractor

code-node-tools/website-contact-extractor

Crawl a domain or list of URLs and extract emails, phone numbers, and social media handles. Cheerio-based crawling with configurable depth, proxies, and selectable extraction targets.