Under maintenance

Pricing

Pay per usage

Try for free

Go to Apify Store

Website Content Crawler

Under maintenance

Try for free

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Syed Rupom

Actor stats

Bookmarked

Total users

Monthly active users

3 hours ago

Last modified

Features

Full site crawling: Follows internal links up to configurable depth
Smart content extraction: Auto-detects main content, strips nav/header/footer/ads
Multiple output formats: Markdown (AI-ready), plain text, or raw HTML
JavaScript rendering: Full Puppeteer-based crawling handles React, Vue, and dynamic sites
JSON-LD extraction: Structured data schemas embedded in pages
Configurable depth & page limits: Control exactly how much to crawl
Custom selectors: Target specific content areas or remove specific elements
Subdomain support: Optionally follow links to subdomains

Output Fields Per Page

Field	Description
`url`	Original URL
`loaded_url`	Final URL after redirects
`title`	Page `<title>`
`description`	Meta description
`author`	Author meta tag
`keywords`	Meta keywords
`og_image`	Open Graph image URL
`canonical`	Canonical URL
`lang`	Page language code
`h1`	Main heading
`h2s`	Top subheadings (up to 10)
`text`	Clean plain text (format=text)
`markdown`	Markdown-formatted content (format=markdown)
`html`	Content HTML (format=html)
`json_ld`	JSON-LD structured data objects
`depth`	Crawl depth from start URL
`referrer`	Page that linked here
`load_time_ms`	Page load time in ms
`status_code`	HTTP status code
`links_found`	Number of links on the page
`crawled_at`	ISO timestamp

Input

{
  "startUrls": [{"url": "https://docs.example.com"}],
  "maxPages": 100,
  "maxDepth": 3,
  "includeSubdomains": false,
  "outputFormat": "markdown",
  "extractSelector": "article",
  "removeSelectors": [".sidebar", ".related-posts"],
  "proxyConfiguration": {"useApifyProxy": false}
}

Use Cases

AI Training Data: Extract clean, structured web content at scale
RAG Pipelines: Feed documentation sites into vector databases (Pinecone, Qdrant, Weaviate)
Custom ChatGPT: Build knowledge bases from product documentation
Content Auditing: Extract and analyze all text across a website
Competitive Research: Extract competitor content for analysis
Documentation Indexing: Index technical docs for search

Tips

Set maxDepth: 0 to only scrape the start URLs without following links
Use extractSelector: "main" to target only the main content area
Set outputFormat: "markdown" for best results with AI/LLM ingestion
Most public sites work without proxies; enable proxies for rate-limited sites

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

131K

4.6

(205)

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

4.9

(7)

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

302

1.9

(2)

🧪High-Volume Website Content & Media Scraper

caring_dizi/blog-content-scraper-fixed

🧪Crawling Done Right! Let me now what you think, what or where or how i can improve my actor, and i am all for constructive criticism. So please message if you have any questions. Enjoy and have a good day.

Jeff Halverson

147

5.0

(2)

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

934

2.3

(3)

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

1.1K

3.0

(1)

Website Content Extractor

tropical_prune/website-content-extractor

CAPABILITIES: extract_content, convert_to_markdown, batch_urls, extract_metadata. INPUT: URLs (single or array), with optional selectors and output format. OUTPUT: structured JSON with title, text, metadata, word_count. FORMATS: json, markdown, text. PRICING: PPE $0.001/page.

Bado

Website Content Crawler

alizarin_refrigerator-owner/website-crawler

Crawl websites for SEO audits. Extracts HTML, title, meta tags, headings, links, & text content from pages. Automatic sitemap detection & parsing Extracts metadata (title, description, OG tags) Heading structure (H1, H2, H3) Internal & external link analysis Image extraction w/alt text Word count

The Howlers

105

Website Content Text Extractor

smart-digital/website-content-text-extractor

Extract visible text content from websites as structured JSON blocks. Supports multi-URL batch processing, header/footer/cookie exclusion, and optional form extraction. Perfect for content analysis and translation workflows.

My Smart Digital

5.0

(1)

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

Website Content Crawler

jasondev/website-content-crawler

A powerful web crawler that extracts text content from websites, optimized for AI models, Large Language Models (LLMs), vector databases, and Retrieval-Augmented Generation (RAG) pipelines.