Pricing

from $3.99 / 1,000 results

Try for free

Go to Apify Store

Website Content Crawler (Advanced Feature)

Try for free

Website Content Crawler extracts clean text, Markdown, HTML, metadata, and links from websites. It checks broken links, redirects, and external URLs while supporting SEO audits, content analysis, web crawling, LLM/RAG pipelines, and search indexing.

Pricing

from $3.99 / 1,000 results

Rating

0.0

(0)

Developer

ScrapeVanta

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

0.1.0 — Broken Link & Redirect Checker variant

New: checkLinkStatus — every link found on a crawled page (up to maxLinksPerPage) is live-checked (HEAD, falling back to GET) and reported with its HTTP status, whether it's broken, and whether/where it redirects (links[] + linkCheckSummary on each page record).
New: checkExternalLinks, linkCheckTimeoutSecs, linkCheckConcurrency to scope and tune the link audit independently of the page-crawl settings.
New: one extra dataset row per broken/redirecting link (type: "linkIssue", isChild: true, parentUrl), plus a dedicated 🔗 Link Audit dataset view — a flat, exportable table of every broken/redirecting link found in the run, separate from the nested per-page detail.
New: redirects are followed manually hop-by-hop (not via the HTTP client's own redirect history), so redirectedTo/redirectChain always reflect the true final destination.
New: per-run URL cache in the link checker — a link that appears on many pages (nav/footer/ social links) is checked once and the result reused everywhere else it's linked from.
Everything from the base crawler (1.0.0, below) is unchanged and still works exactly the same when checkLinkStatus is turned off.

1.0.0 (base)

Initial Apify Actor release, ported from the standalone Website Content Crawler script.
4-tier anti-bot ladder: curl_cffi (Chrome/Firefox TLS+HTTP2 fingerprint) → vendor-aware challenge detection (Cloudflare, DataDome, PerimeterX, Akamai, Imperva, Kasada, AWS WAF, Turnstile, reCAPTCHA, hCaptcha, SiteGround) → Camoufox headless browser → optional CapSolver widget-captcha solving → per-domain session/cookie/UA reuse.
New: Apify Proxy fallback ladder (no proxy → Datacenter → Residential, retried up to 3 times, sticky once Residential is reached), replacing the previous fixed third-party proxy list.
New: results are pushed to the dataset in real time, one page at a time, instead of being buffered into a single JSON file.
New: four dataset views (Overview, Full content, SEO & metadata, Anti-bot debug) so results render as separate, purpose-built tables in Console instead of one wide table.
New: optional HTML snapshots and screenshots now save to the Actor's key-value store (public URLs) instead of local disk.
New: friendly, emoji-tagged real-time logging of collection progress and proxy escalations; internal ladder mechanics moved behind an opt-in verbose debug toggle.

Apify Store Actors Scraper 2025

jupri/apify-store

💫 Apify.com Store Extension

cat

148

Sqoosh Image Compressor

eunit/sqoosh-image-compressor

Optimize images for SEO with the Squoosh Actor on Apify. Batch compress, resize, and convert images to WebP, AVIF, and MozJPEG to boost site speed and Core Web Vitals. Automate high-performance image optimization for web scraping and developer workflows with ease.

Emmanuel Uchenna

5.0

(2)

GetApp Scraper

alaricus/getapp-scraper

Extract detailed data from GetApp. Gather product specs, pricing, features, integrations, and ratings. Easily collect software info by keyword, category, or URL to power your market research and competitive analysis.

Alaricus

Advanced Bing Scraper

dainty_screw/advanced-bing-scraper

Harness the power of our Advanced Bing Scraper to extract accurate and up-to-date search results from Bing quickly. Ideal for market research, SEO analysis, and competitive intelligence. Supports global Bing domains.

codemaster devops

5.0

(2)

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

303

1.9

(2)

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

Mikolabs

5.0

(1)

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

566

3.7

(3)

Website Content Crawler

jasondev/website-content-crawler

A powerful web crawler that extracts text content from websites, optimized for AI models, Large Language Models (LLMs), vector databases, and Retrieval-Augmented Generation (RAG) pipelines.

Jason Giang

Website Content Crawler

novashieldai/website-content-crawler

Universal website crawler that extracts clean text/markdown content, metadata, links, and images from any URL. Features sitemap parsing, robots.txt respect, and multi-page BFS crawling with depth control.

Ali haydar Karadaş

AST Website Content Crawler for RAG

lukas459/ai-web-to-markdown-crawler-llm-rag-optimized

Advanced web crawler extracting token-chunked Markdown for LLMs & RAG. Features dual-engine hybrid parsing (IBM Docling & Trafilatura) to preserve tables and layouts. Delivers clean text split by native tiktoken counts with rich metadata (headings, levels, tokens) for instant vector search.

lukas schmeck

5.0

(2)

Wordpress Email Scraper Fast

solid-scraper/wordpress-email-scraper-fast

📧 WordPress Email Scraper Fast extracts emails from websites quickly and accurately for lead generation. Perfect for marketers, agencies & sales teams—save time, boost outreach, and find targeted contacts with ease. 🚀

SolidScraper

Website Content Crawler (Advanced Feature)

Changelog

0.1.0 — Broken Link & Redirect Checker variant