Pricing

Pay per usage

Go to Store

Website Content Crawler

Try for free

Developed by

Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.0 (40)

Pricing

Pay per usage

1386

Total users

53K

Monthly users

7.8K

Runs succeeded

>99%

Issues response

6.8 days

Last modified

3 days ago

Developer tools

374

346

Website Content Crawler stuck - cost keeps increasing

Opened 2 days ago by digtital_moose, last comment 2 days ago by digtital_moose

How can i get all hidden fields in my actor result

Opened 2 days ago by mohit1.vdoit, last comment 2 days ago by mohit1.vdoit

Http website inaccessible

Opened 3 days ago by souheil, last comment 3 days ago by souheil

Didn't crawl the entire page and seemed to do it in no particular orer

Opened 4 days ago by arsia, last comment 4 days ago by arsia

No text parsed from from webpage.

Opened 4 days ago by formidable_quagmire, last comment 4 days ago by formidable_quagmire

Avoid query parameters when crawling websites

Opened 5 days ago by innovum_admin, last comment 5 days ago by innovum_admin

To much time

Opened 5 days ago by florian-morina, last comment 5 days ago by Jiří Spilka (jiri.spilka)

Getting 403 from public page

Opened 6 days ago by formidable_quagmire, last comment 5 days ago by formidable_quagmire

No text parsed from from webpage.

Opened 6 days ago by formidable_quagmire, last comment 6 days ago by Jindřich Bär (jindrich.bar)

2 failed crawled websites

Opened 7 days ago by bor.cerlini, last comment 7 days ago by Jiří Spilka (jiri.spilka)

crawling cannot be done with arabic website in english

Opened 10 days ago by aswinthazhath, last comment 6 days ago by Jindřich Bär (jindrich.bar)

Crawling with markdown give half of the data where on the other pages gives us complete data

Opened 12 days ago by formidable_quagmire, last comment 12 days ago by formidable_quagmire

Timeout and no data

Opened 14 days ago by Autocom, last comment 7 days ago by Jiří Spilka (jiri.spilka)

There was an uncaught exception during the run of the Actor and it was not handled

Opened 18 days ago by formidable_quagmire, last comment 6 days ago by Jindřich Bär (jindrich.bar)

2025-05-13T10:00:05.221Z WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. page.evaluate: Execution context was destroyed, most likely because of a navigation.

Opened 19 days ago by formidable_quagmire, last comment 7 days ago by Jiří Spilka (jiri.spilka)

Error: Cannot run Actor (Network Error)

Opened 19 days ago by dawieharmse, last comment 19 days ago by dawieharmse

CORS Error

Opened 20 days ago by fmateen, last comment 18 days ago by fmateen

Is there a way to crawl URL from the visible HTML after removing "removeElementsCssSelector"

Opened 20 days ago by formidable_quagmire, last comment 18 days ago by shashank734

Cookie Banner is not removed

Opened a month ago by Joe11, last comment 18 days ago by Jakub Kopecký (jakub.kopecky)

Crawling is stuck (10h+)

Opened a month ago by jauns-ai, last comment a month ago by Jakub Kopecký (jakub.kopecky)

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

553

3.8

AI-Powered Web Content & Link Extractor

scrapercoder/ai-powered-web-content-link-extractor

Crawls websites to extract clean, structured content for AI/LLM use, ideal for training datasets, knowledge bases, and RAG systems. Json output includes: * text: Normalized page content * links: Extracted sub-URLs

wallnut.ai

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David Deng

858

4.4

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

5.0

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

5.0

Dynamic Markdown Scraper

louisdeconinck/dynamic-markdown-scraper

Effortlessly feed LLM AIs with clean Markdown using our advanced web scraper. Seamlessly scrape dynamic, JavaScript-rendered websites while preserving original formatting. Ideal for AI training, documentation, and content migration.

Louis Deconinck

5.0

Extract-any-webpage-content-for-llm

ai-developer/extract-any-webpage-content-for-llm

Fast and easy way to extract data from any webpage and are LLM friendly. The tool lets you easily extract content from any website. Ideal for researchers, marketers, and developers.

aideveloper

450

Web Scraper

apify/web-scraper

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

Apify

85K

4.4

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David Deng

338

4.1

RAG Web Browser

apify/rag-web-browser

Web browser for OpenAI Assistants, RAG pipelines, or AI agents, similar to a web browser in ChatGPT. It queries Google Search, scrapes the top N pages, and returns their content as Markdown for further processing by an LLM. It can also scrape individual URLs. Supports Model Context Protocol (MCP).

Apify

2.9K

4.3