Pricing

from $1.20 / 1,000 pages

Website Content Crawler (Anti-Bot) - $1.20 / 1000 Pages

Crawl any website, even behind Cloudflare, DataDome, PerimeterX or Akamai, and get clean Markdown and text of every page for LLMs, RAG and AI agents. Main-content extraction, URL globs, sitemaps, depth and page limits. $1.20 per 1000 pages; failed pages are free.

Pricing

from $1.20 / 1,000 pages

Rating

0.0

(0)

Developer

Scrapeunblocker

Actor stats

Bookmarked

Total users

Monthly active users

7 days ago

Last modified

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

Website Content Crawler - Clean Text, Markdown & RAG Chunks

topapi/website-content-crawler

Crawl any website or URL list for $1.20 per 1,000 pages: one row per page with clean text, Markdown, HTML, metadata, links, headings, tables and RAG chunks. Plain HTTP first, headless Chrome only for JavaScript pages. Sitemaps, llms.txt, PDF/DOCX text, Bing/DuckDuckGo search, change monitor.

Top Api

Universal Web Scraper & Data Extractor

motivational_nickel/universal-scraper

Fast AI-ready webpage extraction with structured JSON output for automation workflows, AI agents, RAG pipelines, SEO analysis, and website intelligence.

Leoncio Coronado Jr

LLMs.txt File Generator

justa/llms-txt-file-generator

Generate an llms.txt file from a website sitemap. Crawls all URLs, extracts titles and meta descriptions, and creates a Markdown-formatted file following the llms.txt specification. Upload then the output of your file directly on your website (Webflow, Wordpress etc.)

Benoit Eveillard

Blog / Dated Content Crawler

diarmuidr/blog-content-crawler

Crawl an entire blog / knowledge base or filter to just the new content. Supporting relevant AI queries by filtering pages by date

Diarmuid

221

5.0

(1)

Web Crawler & Semantic Schema-Enhanced Extractor

devil_port369-owner/web-crawler

Depth-controlled web crawler that transforms websites into structured analytics-ready data. Starting from one or more URLs, it crawls internal links up to a configurable depth and outputs detailed JSON records per page

DataFusionX

5.0

(8)

Website to Markdown Converter for LLM Prompts

bepolabs/web-to-markdown

Paste URLs, get Markdown rows without menus, footers or cookie banners. Optional ready-made chunks for vector search and a depth setting to follow internal hyperlinks.

BEPO Labs

News Scraper - Keyword News Search & Monitoring API

tidytools/news-scraper

Search 250+ news publishers by keyword in one run: headline, link, time, publisher, description, keywords. Country, language and topic filters, your own sites and feeds, duplicate stories collapsed, only-new alerts. $1 per 1,000 articles.

Yukai Lin

Web Search Reader: results + clean page text (FR/CA by default)

tolca/web-search-reader

One search, done: 10 web results plus the clean text of the top pages (title, text, sha256, fetch time). French/Canada by default, official-sources mode (.gc.ca, .gouv.qc.ca, .gov, CanLII). No charge when a query returns nothing.