AgentShot

Web Crawler & Semantic Schema-Enhanced Extractor

devil_port369-owner/web-crawler

Depth-controlled web crawler that transforms websites into structured analytics-ready data. Starting from one or more URLs, it crawls internal links up to a configurable depth and outputs detailed JSON records per page

Fetch any URL and return clean markdown/text plus links. No headless browser for most pages. Free demo: https://vaytric.com/page-to-markdown/ — bulk URL lists here.

Website to Markdown for RAG: Crawl & Changed Pages

cliqtomedia/website-to-markdown

Turn a website or docs section into clean Markdown, one row per page, for RAG and LLM pipelines. Stays inside your start path, respects robots.txt and maxPages exactly. Repeat runs can return only new and changed pages. No login.

URL to Markdown: Convert Any Webpage to Clean Markdown

hydrafetch/website-to-markdown

Give it URLs and get each page as clean markdown for LLMs, RAG and AI agents, with navigation, cookie banners and ads removed and JavaScript pages rendered. Input: full URLs. Each row has the markdown, title, language and word count; pages that return nothing are not charged.

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

A powerful web crawler that extracts text content from websites, optimized for AI models, Large Language Models (LLMs), vector databases, and Retrieval-Augmented Generation (RAG) pipelines.

Crawl any website and get each page's main content as clean Markdown and text for LLMs, RAG and search. Sitemap-seeded, de-duplicated, no nav or cookie banners. Respects robots.txt and AI-crawler opt-outs. $1.50 per 1,000 pages.

RAG Web Browser (Anti-Bot) - $1.80 / 1000 Pages

scrapeunblocker/rag-web-browser

Search the web and get the top results as clean Markdown, or read any URL, in seconds. Built as a web tool for LLMs, AI agents and RAG. Pages behind Cloudflare, DataDome, PerimeterX or Akamai work. No Google needed. $1.80 per 1000 pages; failed pages are free.

Turn web pages into clean, LLM-ready Markdown: main content only, without menus, ads or cookie banners. Keeps headings, lists, links, code blocks and tables. Pay only per converted page.

Article Extractor API — Title, Author, Date & Full Text

highbrow_fame/article-extractor-api

Extracts the article from any news or blog URL: title, author, published date, full text, Markdown, lead image and language. Works on paywall-free pages. Unofficial Article Extractor API.