Pricing

from $1.00 / 1,000 results

Website Content Crawler

Crawls websites starting from your given URLs, follows internal links up to a set depth, and extracts clean page content — titles, text, and metadata — from every page. Built with Python and Crawlee, it outputs structured, ready-to-use data for search indexing, RAG pipelines, or content analysis.

Pricing

from $1.00 / 1,000 results

Rating

0.0

(0)

Developer

Biddut Hossain

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

RAG Web Browser

RAG Web Browser is a lightweight Python Actor that fetches any URL or search query, strips ads and clutter, and extracts clean, readable text from web pages. It returns structured, LLM-ready content optimized for retrieval-augmented pipelines and AI chat apps needing accurate real-time context.

✨ Features

Accepts a direct URL or a plain text search query
Strips scripts, nav bars, footers, ads, and other noise
Returns clean, structured JSON output (title + text + char count)
Configurable character limit and CSS selectors to remove
Built on httpx + BeautifulSoup for fast, reliable scraping

📥 Input

Field	Type	Description	Default
`query`	string	A URL or search query	(required)
`maxChars`	integer	Max characters of text to return	`5000`
`removeSelectors`	array	CSS selectors to strip before extraction	`["script","style","nav","footer","header","aside","iframe","noscript"]`

Example input

```json { "query": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation", "maxChars": 3000 } ```

📤 Output

Each run pushes one item to the dataset:

```json { "query": "https://example.com", "url": "https://example.com", "title": "Example Domain", "text": "This domain is for use in illustrative examples...", "charCount": 214 } ```

🚀 Usage

Set your query input (URL or search term).
Run the Actor.
Retrieve extracted text from the Dataset tab, or via the Apify API.

🧠 Use Cases

Feeding live web content into LLM prompts (RAG pipelines)
Building AI chatbots with up-to-date web knowledge
Content extraction and summarization pipelines
Lightweight alternative to full-page scrapers when only text is needed

🛠 Built With

Web Crawler & Semantic Schema-Enhanced Extractor

devil_port369-owner/web-crawler

Depth-controlled web crawler that transforms websites into structured analytics-ready data. Starting from one or more URLs, it crawls internal links up to a configurable depth and outputs detailed JSON records per page

DataFusionX

5.0

Website Content Crawler

crawlerbros/website-content-crawler

Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.

Crawler Bros

Website to Markdown Crawler - Full-Site Text for LLMs & RAG

entranced_gelato/website-to-markdown-crawler

Crawl any website from a start URL and get every page as clean text + Markdown for LLMs, RAG, and AI agents. Follows internal links with depth and page limits, strips nav and ads, and returns one structured record per page. A fast, no-config site-to-Markdown crawler.

AIDevs

Website Crawler

elcon/website-crawler

Crawls a website starting from one or more URLs and extracts the title, meta description, headings and text from each page.

elcon software

Website Content Crawler for AI & LLM Data

your_scraper_guy/website-content-crawler-lite

Crawl any website from a seed URL and extract clean Markdown content, ready for LLM training data, RAG pipelines, and vector databases. Set crawl depth, page limits, and domain scope.

Code With Aqib

AI-Powered Web Content & Link Extractor

scrapercoder/ai-powered-web-content-link-extractor

Crawls websites to extract clean, structured content for AI/LLM use, ideal for training datasets, knowledge bases, and RAG systems. Json output includes: * text: Normalized page content * links: Extracted sub-URLs

wallnut.ai

177

Website Content Crawler

parseforge/website-content-crawler

Crawl any website and pull clean Markdown content ready for AI! Follow links across a whole domain and extract page text, titles, headings, images, and metadata. Perfect for building RAG pipelines, training datasets, knowledge bases, and vector databases. Start crawling content in minutes!