Pricing

Pay per usage

Website Content Crawler

🌐 Full website crawler that extracts structured content (text, headings, metadata, links, images) from any domain. Free platform compute pricing.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Luan

Actor stats

Bookmarked

Total users

Monthly active users

31 minutes ago

Last modified

✨ Features

🌐 Full website crawler that extracts structured content (text, headings, metadata, links, images) from any domain. Free platform compute pricing.
Handles pagination for large datasets
Supports proxy configuration for reliable scraping
Exports data in JSON, CSV, and Excel formats
Built on Apify's reliable cloud infrastructure
Easy to integrate with webhooks and API

🔧 How It Works

Configure input — set your search parameters, URLs, or filters
Run the actor — it handles all the scraping automatically
Get your data — download results in JSON, CSV, or via API

📋 Input Parameters

See the .actor/input_schema.json for full configuration options. Key parameters include:

Target URLs, search queries, or identifiers
Pagination limits and filters
Proxy configuration
Output format preferences

📊 Output

The actor returns structured data in JSON format. See the .actor/OUTPUT_SCHEMA.json for detailed field descriptions.

💰 Pricing

Model: Free to use with platform compute pricing
You only pay for Apify platform compute time — no per-result charges
$5 free monthly credit for new Apify users
No subscription required

🚀 Quick Start

# Run via Apify CLI
apify call website-content-crawler

# Or use the API
curl -X POST "https://api.apify.com/v2/acts/flamoqad35tLmtiuD/runs" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input": {}}'

📚 Use Cases

Market research and competitive analysis
Lead generation and sales prospecting
Social media monitoring and brand tracking
Data-driven decision making
Academic research and trend analysis

🔗 Integration

This actor can be integrated with:

Webhooks for real-time data streaming
Apify API for programmatic access
Zapier/Make for no-code automation
Custom pipelines via direct API calls

⚠️ Disclaimer

Use this actor in compliance with the target platform's terms of service and applicable laws. Data scraping should respect robots.txt and rate limits.

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

129K

4.6

(203)

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

3.9K

4.9

(7)

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

302

1.9

(2)

🧪High-Volume Website Content & Media Scraper

caring_dizi/blog-content-scraper-fixed

🧪Crawling Done Right! Let me now what you think, what or where or how i can improve my actor, and i am all for constructive criticism. So please message if you have any questions. Enjoy and have a good day.

Jeff Halverson

147

5.0

(2)

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

934

2.3

(3)

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

1.1K

3.0

(1)

Website Content Extractor

tropical_prune/website-content-extractor

CAPABILITIES: extract_content, convert_to_markdown, batch_urls, extract_metadata. INPUT: URLs (single or array), with optional selectors and output format. OUTPUT: structured JSON with title, text, metadata, word_count. FORMATS: json, markdown, text. PRICING: PPE $0.001/page.

Bado

Website Content Crawler

alizarin_refrigerator-owner/website-crawler

Crawl websites for SEO audits. Extracts HTML, title, meta tags, headings, links, & text content from pages. Automatic sitemap detection & parsing Extracts metadata (title, description, OG tags) Heading structure (H1, H2, H3) Internal & external link analysis Image extraction w/alt text Word count

The Howlers

101

Website Content Text Extractor

smart-digital/website-content-text-extractor

Extract visible text content from websites as structured JSON blocks. Supports multi-URL batch processing, header/footer/cookie exclusion, and optional form extraction. Perfect for content analysis and translation workflows.

My Smart Digital

5.0

(1)

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.