Pricing

from $4.00 / 1,000 results

Try for free

Go to Apify Store

Website Content Crawler

Try for free

Pricing

from $4.00 / 1,000 results

Rating

0.0

(0)

Developer

Admo Solutions

Actor stats

Bookmarked

Total users

Monthly active users

5 hours ago

Last modified

Why use this Actor?

AI/RAG Ready: Extract clean Markdown format perfect for LLM training and RAG pipelines
Smart Crawling: Follows links with configurable depth and page limits
JavaScript Support: Uses Playwright to handle JS-heavy websites
Clean Output: Removes ads, navigation, footers, and boilerplate content
Respectful: Configurable delays between requests

How to use

Enter one or more start URLs
Set crawl depth (how many link hops to follow)
Set max pages limit
Click Start and download your results

Input

Field	Description	Default
Start URLs	URLs to start crawling from	Required
Max Crawl Depth	Link hops from start URLs (0=start only)	2
Max Pages	Maximum pages to crawl	100
Delay Between Requests	Politeness delay in seconds	1
Include External Links	Follow links to other domains	false

Output

Each crawled page includes:

url: Page URL
title: Page title
text: Full text content (plain text)
markdown: Clean Markdown format (headers, paragraphs, lists, code blocks)
metaDescription: HTML meta description
headings: List of headings with levels
wordCount: Total word count
depth: Crawl depth from start URL

{
    "url": "https://docs.python.org/3/tutorial/classes.html",
    "title": "9. Classes — Python 3.12.5 documentation",
    "text": "9. Classes Classes provide a means of bundling data and functionality together...",
    "markdown": "# 9. Classes\n\nClasses provide a means of bundling data and functionality together...",
    "metaDescription": "This chapter aims to present the many new features of Python 3...",
    "headings": [{"level": 1, "text": "9. Classes"}, {"level": 2, "text": "A Word About Names and Objects"}],
    "wordCount": 2847,
    "depth": 1
}

Download as JSON, CSV, HTML, or Excel.

💰 Pricing

This Actor uses the Pay Per Event pricing model:

Event	Price
Per page scraped	$4.00 / 1,000

For example, scraping 100 pages costs: $0.40

Tips

Use depth 0-2 for documentation sites
Use depth 1-3 for blogs and content sites
Increase delay for slower servers
Use maxPages to control costs

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

154K

4.6

(233)

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

4.5K

4.9

(7)

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

303

1.9

(2)

Website Content Crawler

parseforge/website-content-crawler

Crawls websites from starting URLs and returns each page's full text content as Markdown and cleaned HTML, ready for AI and LLM pipelines.

ParseForge

126

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

Mikolabs

5.0

(1)

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

1.1K

3.0

(1)

Website Content Crawler

alizarin_refrigerator-owner/website-crawler

Crawl websites for SEO audits. Extracts HTML, title, meta tags, headings, links, & text content from pages. Automatic sitemap detection & parsing Extracts metadata (title, description, OG tags) Heading structure (H1, H2, H3) Internal & external link analysis Image extraction w/alt text Word count

The Howlers

139

Website Content Crawler — Markdown for LLM & RAG

brilliant_gum/web-content-crawler

Deep-crawl websites and extract LLM-ready Markdown with OG tags, JSON-LD, author, dates, token estimates, native RAG chunking, language filtering, content-hash dedup, and per-page error reporting. Enforced timeouts. Zero silent failures.

Yuliia Kulakova

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

941

2.3

(3)

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

567

3.7

(3)

🧪High-Volume Website Content & Media Scraper

caring_dizi/blog-content-scraper-fixed

🧪Crawling Done Right! Let me now what you think, what or where or how i can improve my actor, and i am all for constructive criticism. So please message if you have any questions. Enjoy and have a good day.