Pricing

from $1.00 / 1,000 results

Website Content Crawler

Crawls websites starting from your given URLs, follows internal links up to a set depth, and extracts clean page content — titles, text, and metadata — from every page. Built with Python and Crawlee, it outputs structured, ready-to-use data for search indexing, RAG pipelines, or content analysis.

Pricing

from $1.00 / 1,000 results

Rating

0.0

(0)

Developer

Biddut Hossain

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

Configure MCP server with Website Content Crawler

The configuration above connects to the hosted Apify MCP server with the Website Content Crawler Actor already loaded. It uses OAuth, so your client opens a browser window to sign you in to Apify on first use. No API token is needed. Clients without OAuth support can authenticate with an Authorization: Bearer header carrying an API token from API & Integrations in Apify Console.

Get a ready-to-use configuration for your MCP client with the Website Content Crawler Actor preconfigured at https://mcp.apify.com/?tools=fetch-actor-details,bornoo/website-content-crawler.

You can connect to the Apify MCP Server using clients like Tester MCP Client, or any other MCP client of your choice.

If you want to learn more about our Apify MCP implementation, check out our MCP documentation. To learn more about the Model Context Protocol in general, refer to the official MCP documentation or read our blog post.

Web Crawler & Semantic Schema-Enhanced Extractor

devil_port369-owner/web-crawler

Depth-controlled web crawler that transforms websites into structured analytics-ready data. Starting from one or more URLs, it crawls internal links up to a configurable depth and outputs detailed JSON records per page

DataFusionX

5.0

Website Content Crawler

crawlerbros/website-content-crawler

Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.

Crawler Bros

Website to Markdown Crawler - Full-Site Text for LLMs & RAG

entranced_gelato/website-to-markdown-crawler

Crawl any website from a start URL and get every page as clean text + Markdown for LLMs, RAG, and AI agents. Follows internal links with depth and page limits, strips nav and ads, and returns one structured record per page. A fast, no-config site-to-Markdown crawler.

AIDevs

Website Crawler

elcon/website-crawler

Crawls a website starting from one or more URLs and extracts the title, meta description, headings and text from each page.

elcon software

Website Content Crawler for AI & LLM Data

your_scraper_guy/website-content-crawler-lite

Crawl any website from a seed URL and extract clean Markdown content, ready for LLM training data, RAG pipelines, and vector databases. Set crawl depth, page limits, and domain scope.

Code With Aqib

AI-Powered Web Content & Link Extractor

scrapercoder/ai-powered-web-content-link-extractor

Crawls websites to extract clean, structured content for AI/LLM use, ideal for training datasets, knowledge bases, and RAG systems. Json output includes: * text: Normalized page content * links: Extracted sub-URLs

wallnut.ai

177

Website Content Crawler

parseforge/website-content-crawler

Crawl any website and pull clean Markdown content ready for AI! Follow links across a whole domain and extract page text, titles, headings, images, and metadata. Perfect for building RAG pipelines, training datasets, knowledge bases, and vector databases. Start crawling content in minutes!