Pricing

from $1.50 / 1,000 results

Website Content Crawler for AI & LLM Data

Crawl any website from a seed URL and extract clean Markdown content, ready for LLM training data, RAG pipelines, and vector databases. Set crawl depth, page limits, and domain scope.

Pricing

from $1.50 / 1,000 results

Rating

0.0

(0)

Developer

Code With Aqib

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Website Content Crawler (LLM/RAG ready)

Crawl any website starting from one or more seed URLs, follow links up to a set depth, and extract clean, readable main content, stripped of navigation, ads, and footers, ready to feed into an LLM, RAG pipeline, or dataset.

What it does

Point it at a URL. It follows internal links up to maxDepth hops, up to maxPagesPerRun total pages, and for each page:

Extracts the main readable content using the same engine behind Firefox's Reader Mode
Outputs it as Markdown (default), plain text, or raw HTML
Grabs the title and meta description
Respects robots.txt by default

Input

Field	Type	Description
`startUrls`	array	Seed URLs to start crawling from
`maxDepth`	integer	How many link-hops to follow from the seed (0 = only the seed pages)
`maxPagesPerRun`	integer	Total page cap for the run
`crawlScope`	string	`same-domain`, `same-hostname`, or `all`
`includeUrlGlobs`	array	Optional allowlist patterns (e.g. `https://example.com/blog/*`)
`excludeUrlGlobs`	array	Optional blocklist patterns (e.g. `/login`)
`respectRobotsTxt`	boolean	Skip pages disallowed by robots.txt
`outputFormat`	string	`markdown`, `text`, or `html`
`maxContentChars`	integer	Truncates content per page to keep the dataset light
`proxyConfiguration`	object	Optional, only needed for sites that block plain requests

Example input

{
    "startUrls": ["https://example.com/blog"],
    "maxDepth": 2,
    "maxPagesPerRun": 100,
    "crawlScope": "same-domain",
    "outputFormat": "markdown"
}

Output

{
    "url": "https://example.com/blog/some-post",
    "depth": 1,
    "title": "Some Post Title",
    "description": "A short meta description of the page.",
    "content": "# Some Post Title\n\nCleaned markdown content of the article...",
    "contentFormat": "markdown",
    "wordCount": 842,
    "truncated": false,
    "crawledAt": "2026-07-13T10:00:00.000Z"
}

Notes

Works on static/server-rendered pages out of the box. Heavy JavaScript single-page apps may need a browser-based crawler instead.
Use crawlScope: "same-domain" to stay within a site, or "all" for broader research crawls.
Turn on proxy configuration only if a specific site blocks default requests, most sites don't require it.

Wikipedia Scraper - Articles, Summaries, Metadata

santamaria-automations/wikipedia-scraper

Extract Wikipedia articles including full content, summary, thumbnails, categories, external links, coordinates, and Wikidata IDs. Multi-language support for 12+ languages. Export data, run via API, schedule and monitor runs, or integrate with other tools.

NanoScrape

Website to Text & Markdown — AI / RAG Content Crawler

inexhaustible_glass/rag-website-crawler

Scrape any website into clean text & Markdown with RAG-ready chunks and token counts for LLMs, vector databases (Pinecone, Qdrant) and AI chatbots. Also extracts linked PDF/Word/Excel. Anti-block, robots.txt-aware. Website-to-text for beginners, full RAG pipeline for pros. CPU only, no API key.

Hitman studio

Website Main Content Extractor

sync-network/website-main-content-extractor

Alam

Website Content Scraper: Clean Markdown for AI and RAG

scrapemint/website-content-scraper

Crawl any website and get clean markdown, text, or HTML per page, ready for RAG pipelines, chatbots, and LLM fine tuning. Plain HTTP, no browser, no API key. Pay per page.

Ken M

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

Stealth Website Crawler

nocturne/stealth-website-crawler

Crawl websites protected by Cloudflare, DataDome, and other anti-bot systems. Extract clean text or markdown for AI/LLM pipelines. Like Website Content Crawler, but for sites that block you.

Nocturne

Youtube

canadesk/youtube

Collect channels, videos (with comments), shorts, streams, playlists, podcasts and posts from Youtube! Filters included. It's fast and costs little!

Canadesk Support

2.0

(1)

Youtube Bulk

canadesk/youtube-bulk

No limits and Bulk mode! Collect channels, videos (with comments), shorts, streams, playlists, podcasts and posts from Youtube! Supports a lot of filters. It's fast and costs little!

Canadesk Support

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

558

3.7

(3)

Quick Website Content Scraper ( Extract Text for RAG & LLMs )

automateitplease/ai-web-content-scraper-extract-text-for-rag-llms

Extract clean text from any website for AI/LLM applications. Supports both static and JavaScript-rendered sites (React, Vue, Angular). Perfect for RAG systems, chatbot training, and content analysis.