Pricing

from $0.10 / 1,000 page extracteds

Try for free

Go to Apify Store

Website Content Crawler Lite

Try for free

Fast, reliable HTTP website crawler for clean text, Markdown, HTML, metadata, and links. Built for AI/RAG, SEO audits, monitoring, and automation with robots.txt, retries, proxy support, and bounded exports.

Pricing

from $0.10 / 1,000 page extracteds

Rating

0.0

(0)

Developer

Fetch Cat

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Changelog

0.1.40 · 12 Nov 2026

Fix: Large files, slow storage writes and redirects to excluded or off-site pages no longer stop the whole crawl; those pages are skipped with a clear reason.
Fix: Missing pages and hidden email-protection links are no longer reported as blocked pages.

0.1.39 · 12 Nov 2026

Fix: Public pages behind unavailable AllOrigins links can now be recovered with clear source attribution and no duplicate page charges.

0.1.37 · 9 Nov 2026

Fix: Temporary page-loading failures no longer cancel unrelated queued pages.

0.1.36 · 9 Nov 2026

Fix: Temporary access failures preserve independent queued pages.

0.1.35 · 9 Nov 2026

Fix: Interrupted page loads no longer cancel unrelated queued pages.

0.1.34 · 9 Nov 2026

Fix: Incomplete page loads no longer cancel unrelated queued pages.

0.1.33 · 9 Nov 2026

Fix: Independent queued pages continue within the original run limits after an access failure; useful output is retained and systemic failures remain explicit.

0.1.32 · 9 Nov 2026

Fix: Improved compatibility when loading public pages.

0.1.31 · 9 Nov 2026

Improvement: Improved public-page availability while preserving your access settings and safety limits.

0.1.30 · 9 Nov 2026

Fix: Client Challenge pages and AWS WAF challenge responses are excluded from useful results and page charges, without rejecting ordinary pages that embed CAPTCHA configuration.
Fix: Fatal access failures preserve completed pages and accurate result, failure and pending-work counts.
Improvement: Customer max-cost and platform timeout limits stop work gracefully before startup or further result charges, preserving collected pages.

0.1.29 · 31 Oct 2026

Fix: Pages with embedded CAPTCHA configuration are no longer mistaken for challenge pages when their public content is available.
Fix: Single-pass page parsing and faster Markdown conversion reduce crawl overhead. Bounded parser processes prevent memory accumulating across pages.
Improvement: Markdown exports preserve useful table cells alongside headings, lists, links, images and code.
Fix: Resumed crawls preserve durable result counts and original limits, with less recovery-storage overhead.
Improvement: Unsupported personal identity-lookup URLs are skipped before target requests, with an explicit warning and no result charges; useful public pages in mixed batches are preserved.
Improvement: Temporary access failures preserve your chosen access settings; exhausted failures remain explicit.

0.1.22 · 27 Oct 2026

Improvement: Added conservative memory checks before parsing complex HTML and saving large crawl checkpoints. Resource-budget failures remain explicit, and ordinary useful page extraction is preserved.

0.1.21 · 27 Oct 2026

Fix: Invalid input and conflicting access settings are rejected before crawling. Explicit paid-access opt-outs are preserved.
Improvement: Crawls stop starting requests when the page budget is exhausted. Resuming a run preserves its original processing window and does not repeat completed extraction.
Improvement: Oversized downloads are stopped earlier, and resumed crawls retrieve only the saved page metadata needed to avoid duplicates.
Documentation: Simplified the public examples and clarified access choices and crawl limits.

0.1.19 · 26 Oct 2026

Documentation: Clarified intended audiences, practical data workflows and AI requests, and standardized historical release notes. Added descriptions of related tools.

0.1.18 · 25 Oct 2026

Fix: Empty and unavailable pages now finish successfully with clear per-URL reasons while keeping useful pages from mixed crawls.
Improvement: Exports contain usable pages only. Skip details are available in the run summary instead of diagnostic-only rows.

25 Sept 2026

Improvement: Improved the input form by keeping everyday controls visible and grouping specialized settings under Advanced options.

0.1.13 · 1 Aug 2026

Fix: More reliable optional managed access was available in this release.
Improvement: Direct access remained the default, with optional access settings checked before crawling and result charges, in this release.
Improvement: Page loading and robots checks respected the original processing deadline in this release.
Feature: Matching pending-work checkpoints and stored-URL reconciliation for resuming deadline-stopped runs without duplicate saved URLs were available in this release.
Improvement: A thirty-second shutdown reserve before the five-minute platform timeout was available in this release.

0.1.11 · 14 Jul 2026

Feature: Concurrent crawling, temporary-error recovery, optional managed access and stopping before timeout were available in this release.
Improvement: Response-size, content and link limits protecting large pages and dataset size were available in this release.
Feature: Redirect and private-network protection, block-page detection, multi-article extraction and run summaries were available in this release.
Fix: Truthful failure reporting for zero-result and fatal-error runs was available in this historical release.
Improvement: Successful page storage was coupled to the page charge while diagnostic rows remained free in this release.
Improvement: Support for documentation sites redirecting public starting URLs to a new domain was available in this release.
Documentation: Updated examples and documentation, along with reliability improvements, were available in this release.
Feature: Configurable inputs and structured results were available in this release.

Cloudflare Bypass

pamberton/cloudflare-bypass

Bypass Cloudflare protected routes. Works for API endpoints too unlike the web crawler.

Pamberton

210

5.0

(2)

OLX Brazil Scraper | Listings, Prices & Seller Data

solidcode/olx-brazil-scraper

[💰 $1.0 / 1K] Scrape OLX Brazil listings at scale. Extract titles, prices, descriptions, photos, seller details, location, and category data from Brazil's largest classifieds marketplace by keyword or URL.

SolidCode

337

4.0

(3)

LinkedIn Post Comment & Reaction Extractor [NO COOKIES] ✅

unseenuser/linkedin-post-comment-reaction-extractor-no-cookies

Extract every commenter and reactor on any public LinkedIn post - profiles, comment text, reaction types, timestamps. The highest-signal lead source on LinkedIn: engaged commenters are 5–10x warmer than cold search results. No login, no cookies. Built for B2B sales teams.

Unseen User

655

5.0

(6)

LinkedIn Profile Posts Scraper - Bulk [NO COOKIES] ✅

unseenuser/LinkedIn-Content

Scrape every post from a list of LinkedIn profiles in one bulk run. One flat row per post: text, hashtags, engagement, reaction breakdowns, media, and the original post behind every repost. Filter by exact dates. No login, no cookies. Built for competitor analysis, social listening and ABM.

Unseen User

369

5.0

(4)

Cloudflare Bypasser

macheta/universal-bypasser

🛡️ Universal Bypasser provides professional-grade access to websites secured by advanced security challenges. Optimized for AI agents and developers, it automatically resolves complex behavioral analysis to deliver clean HTML, parsed JSON, and session cookies. Seamless data extraction at scale.

Anass

387

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

4.8K

4.9

(7)

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

Mikolabs

5.0

(1)

Website Content Crawler – Markdown for AI, Emails & Contacts

locomint/website-crawler-content-contacts

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.

Locomint

Similarweb Traffic Scraper

fetch_cat/similarweb-traffic-scraper

Get Similarweb website traffic, rankings, engagement, channel mix, keywords, geography, and AI referral estimates.

Fetch Cat

213

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

Website Content Crawler Lite