Sitemap Content Extractor — Website to Clean Markdown

Website Content Extractor - Markdown for LLM, RAG, Vector DB

snow_leo_data/website-to-markdown

Readable text from 73.1% of 52 live pages over plain HTTP, including 10 of 10 documentation sites. Website to LLM text and a website to markdown converter in one run: clean Markdown, chunks with overlap for a vector database, 46 flat fields, and what changed since the last run.

Turns a list of article URLs into normalized records: title, author, dates, canonical, heading outline, main text, word count, language, an extraction-confidence score, and per-field provenance.

Turn any article into clean markdown for LLMs. Renders JS-heavy pages, strips ads and navigation, pay only for successful extractions. Best for blog posts, news articles, and documentation pages — not homepage or listing pages.

Sitemap URL Extractor

lost.fany/sitemap-url-extractor

Extract every URL from a website's sitemap.xml, sitemap index and robots.txt in seconds — with lastmod, changefreq, priority and hreflang. No crawling, no browser: fast, cheap, reliable.

Scrape any podcast RSS feed for episodes with metadata, durations and chapters. Get FREE publisher transcripts (Podcasting 2.0) or fall back to Whisper with your own key. Bulk-import subscriptions via OPML, filter by date/duration/title, and monitor shows for new episodes.

Sitemap URL Extractor — robots.txt, Sitemap Index, gzip

chorelet/sitemap-url-extractor

Every URL of a website from its sitemaps: robots.txt discovery, sitemap indexes, gzip, lastmod, changefreq, priority, hreflang alternates, image/video/news extensions. Include/exclude patterns and date filter, CSV/JSON export and API.

Crawl any website and extract the main readable content of each page as clean markdown - nav, footers, scripts and ads stripped. Built for RAG pipelines and LLM ingestion. One record per page {url, title, markdown, word_count} plus a final site inventory.

Extract the exact sitemap file you name, or discover a domain's sitemaps. Deduplicated URL, lastmod and source rows with explicit partial/error reports. No page crawl or proxy; empty runs are free.

Scrape SEC EDGAR filings with zero anti-bot friction: 10-K, 10-Q, 8-K, Form 3/4/5 insider trades, Form D funding, 13F holdings, 13D/G, S-1, proxy statements. XBRL financials, full-text search, monitor mode. Pure HTTP, no browser, no proxies needed.