Pricing

from $0.10 / 1,000 page extracteds

Website Content Crawler Lite

Fast, reliable HTTP website crawler for clean text, Markdown, HTML, metadata, and links. Built for AI/RAG, SEO audits, monitoring, and automation with robots.txt, retries, proxy support, and bounded exports.

Pricing

from $0.10 / 1,000 page extracteds

Rating

0.0

(0)

Developer

Hanna Nosova

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Changelog

0.4

Fixed Apify proxy execution by replacing invalid base64/hyphenated session IDs with deterministic SHA-256 identifiers using only allowed characters.
Kept direct access as the default and added proxy setup validation before crawl work and result charging.
Bounded robots, retry backoff, and page requests by the shared graceful deadline.
Added injected lifecycle tests for direct and proxy routes, robots.txt, retries, mixed page outcomes, deadlines, and charge-after-save behavior.
Added fingerprinted pending-work checkpoints and dataset reconciliation so deadline-stopped runs can resume without duplicating stored URLs.
Enforced a 30-second shutdown reserve before the 300-second platform timeout.

0.3

Added bounded concurrency, transient retry/backoff with Retry-After, optional proxy support, and graceful pre-timeout stopping.
Added response, content, and link limits to prevent oversized pages from exhausting memory or exceeding dataset limits.
Added redirect/private-network protection, block-page detection, multi-article extraction, and RUN_SUMMARY.
Fixed zero-result and fatal-error runs incorrectly appearing successful.
Coupled successful page storage to the page charge while keeping diagnostic rows free.
Improved support for documentation sites that safely redirect their public starting URL to a new domain.

0.2

Improved examples, documentation, and reliability.

0.1

Initial version with configurable inputs and structured results.

Website Crawler API — Markdown for RAG

tugelbay/website-content-crawler

Website crawler API for public pages and clean Markdown, text, or HTML output for RAG pipelines, AI agents, documentation indexing, and monitoring. Guide: https://konabayev.com/tools/website-content-crawler/?utm_source=apify_info&utm_medium=referral&utm_campaign=website-content-crawler

Tugelbay Konabayev

Website Content Crawler

rupom888/website-content-crawler

Syed Rupom

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

Website Content Crawler

novashieldai/website-content-crawler

Universal website crawler that extracts clean text/markdown content, metadata, links, and images from any URL. Features sitemap parsing, robots.txt respect, and multi-page BFS crawling with depth control.

Ali haydar Karadaş

AI Website Content Crawler

ilborso/ai-website-content-crawler

A super fast website crawler for Agentic AI integration

Fabio Borsotti

5.0

Website to Markdown Crawler for LLM & RAG

logiover/website-text-markdown-crawler

Crawl any website to clean Markdown and plain text for LLM training and RAG. HTML to Markdown, no API or login. Export website text to CSV or JSON.

Logiover

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

Website to Text & Markdown — AI / RAG Content Crawler

inexhaustible_glass/rag-website-crawler

Scrape any website into clean text & Markdown with RAG-ready chunks and token counts for LLMs, vector databases (Pinecone, Qdrant) and AI chatbots. Also extracts linked PDF/Word/Excel. Anti-block, robots.txt-aware. Website-to-text for beginners, full RAG pipeline for pros. CPU only, no API key.

Hitman studio

Website Content Crawler

crawlerbros/website-content-crawler

Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.

Crawler Bros

Website to Markdown Crawler - Full-Site Text for LLMs & RAG

entranced_gelato/website-to-markdown-crawler

Crawl any website from a start URL and get every page as clean text + Markdown for LLMs, RAG, and AI agents. Follows internal links with depth and page limits, strips nav and ads, and returns one structured record per page. A fast, no-config site-to-Markdown crawler.