Pricing

Pay per usage

Go to Store

Website Content Crawler

Try for free

Developed by

Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.6 (38)

Pricing

Pay per usage

1307

Total users

49.3k

Monthly users

6.9k

Runs succeeded

>99%

Issue response

3.8 days

Last modified

7 days ago

Developer tools

360

334

2025-05-13T10:00:05.221Z WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. page.evaluate: Execution context was destroyed, most likely because of a navigation.

Opened 21 hours ago by formidable_quagmire, last comment 21 hours ago by formidable_quagmire

Error: Cannot run Actor (Network Error)

Opened a day ago by dawieharmse, last comment a day ago by dawieharmse

CORS Error

Opened 2 days ago by fmateen, last comment 2 days ago by fmateen

Is there a way to crawl URL from the visible HTML after removing "removeElementsCssSelector"

Opened 2 days ago by formidable_quagmire, last comment 2 days ago by formidable_quagmire

Cookie Banner is not removed

Opened 12 days ago by Joe11, last comment 12 days ago by Joe11

Crawling is stuck (10h+)

Opened 14 days ago by jauns-ai, last comment 9 days ago by Jakub Kopecký (jakub.kopecky)

Crawling a small list of pdf urls hangs and crashes the crawler repetitively

Opened 20 days ago by uglyrobot, last comment 16 days ago by Jakub Kopecký (jakub.kopecky)

Feature Request: Automatic Recrawling Within Same Task Run for RAG System Integration

Opened 20 days ago by sprouto_net, last comment 16 days ago by Jakub Kopecký (jakub.kopecky)

Is it possible to speed up the processing time?

Opened 21 days ago by sheldon-supreme, last comment 20 days ago by Jakub Kopecký (jakub.kopecky)

simple page is throwing an error

Opened 22 days ago by burgundy_zebra, last comment 9 days ago by Jakub Kopecký (jakub.kopecky)

Crawling did not support cookie rotating

Opened 23 days ago by meddlesome, last comment 23 days ago by meddlesome

Execution context was destroyed

Opened a month ago by benjaminprevot, last comment a month ago by benjaminprevot

can we get the images on the pages too?

Opened a month ago by disarming_rutabaga, last comment 20 days ago by disarming_rutabaga-owner

Exclude Start URL and Disallowed Paths from Output + Return Clean JSON Structure

Opened a month ago by rudy-seo, last comment 20 days ago by Jakub Kopecký (jakub.kopecky)

Error on Zapier Actor

Opened a month ago by insiderperks-owner, last comment 18 days ago by Jiří Spilka (jiri.spilka)

Issue Crawling Content from Paid Websites Like New York Times

Opened a month ago by onlinereach, last comment a month ago by Jakub Kopecký (jakub.kopecky)

crawler wont click on a specific button

Opened a month ago by shikh.sn2021, last comment a month ago by Jakub Kopecký (jakub.kopecky)

Adsterra .com

Opened a month ago by Tijjeboy, last comment a month ago by Jiří Spilka (jiri.spilka)

number of saved lines

Opened a month ago by kocsi, last comment 16 days ago by Jakub Kopecký (jakub.kopecky)

Large number of requests fail

Opened a month ago by cirez_d, last comment a month ago by cirez_d

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

527

3.8

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

5.0

AI-Powered Web Content & Link Extractor

scrapercoder/ai-powered-web-content-link-extractor

Crawls websites to extract clean, structured content for AI/LLM use, ideal for training datasets, knowledge bases, and RAG systems. Json output includes: * text: Normalized page content * links: Extracted sub-URLs

wallnut.ai

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

5.0

Ai Web Scraper - Extract Data With Ease

eloquent_mountain/ai-web-scraper-extract-data-with-ease

Ai Web Scraper enables scraping for everyone, including non-techies! It uses Google's Gemini LLM to scrape websites with natural language commands. It dynamically extracts data, no selector input needed, handles dynamic content and cookie consent, avoids bot detection, outputs JSON or other formats.

Paco

304

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David Deng

697

4.2

RAG Web Browser

apify/rag-web-browser

Web browser for OpenAI Assistants, RAG pipelines, or AI agents, similar to a web browser in ChatGPT. It queries Google Search, scrapes the top N pages, and returns their content as Markdown for further processing by an LLM. It can also scrape individual URLs. Supports Model Context Protocol (MCP).

Apify

2.3k

4.4

AI Web Scraper - Powered by Crawl4AI

raizen/ai-web-scraper

A blazing-fast AI web scraper powered by Crawl4AI. Perfect for LLMs, AI agents, AI automation, model training, sentiment analysis, and content generation. Supports deep crawling, multiple extraction strategies and flexible output (Markdown/JSON). Seamlessly integrates with Make.com, n8n, and Zapier.

Raizen Technology

1.0

Web Scraper

apify/web-scraper

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

Apify

82.6k

4.5

Extract-any-webpage-content-for-llm

ai-developer/extract-any-webpage-content-for-llm

Fast and easy way to extract data from any webpage and are LLM friendly. The tool lets you easily extract content from any website. Ideal for researchers, marketers, and developers.