Website Content Crawler Pro

Pricing

from $2.97 / 1,000 results

Try for free

Go to Apify Store

Website Content Crawler Pro

Try for free

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

Pricing

from $2.97 / 1,000 results

Rating

5.0

(1)

Developer

halam

Maintained by Community

Actor stats

Bookmarked

391

Total users

Monthly active users

6 days ago

Last modified

Actor uses Standby mode

Beta

The Standby mode is a new, lightweight method for using Actors. Instead of starting an Actor for each input and waiting for results, the Actor remains ready in the background to handle arbitrary HTTP requests, just like any web or API server.

Actor URL

Send a HTTP request to the this URL, and wait for the response. To call the Actor, you need to have an Apify account. Replace <YOUR_API_TOKEN> with your API token, which you can find under Settings > Integrations in Apify Console.Learn more.

To view the specific endpoint of this Actor, go to README.

https://datascoutapi--website-content-crawler-pro.apify.actor?token=<YOUR_API_TOKEN>

Max requests per run

maxRequestsPerActorRunnumber

The maximum number of concurrent HTTP requests a single standby Actor run can accept. If this limit is exceeded, the system starts a new Actor run to handle the request, which will take a few seconds.

Default value of this property is 3

Desired requests per run

desiredRequestsPerActorRunnumber

The desired number of concurrent HTTP requests a single standby Actor run can handle. If this limit is exceeded, the system preemptively starts a new Actor run to handle additional requests.

Default value of this property is 3

Memory (MB)

memoryMbytesnumber

The amount of memory (RAM) allocated for the Actor standby run, in megabytes. With more memory, the Actor can typically handle more requests in parallel, but the number of compute units consumed and the cost also grow.

Default value of this property is 1024

Idle timeout (seconds)

idleTimeoutSecsnumber

If a standby Actor run doesn’t receive any HTTP requests in this amount of time, the system will finish the run. When a new request comes, the system might need to start a new standby Actor run to handle it, which will take a few seconds. Higher idle timeout increases the responsiveness but increases the costs as the Actor needs to keep running longer.

Default value of this property is 300

Build

buildenum

Build version of the Actor that will be used for the standby run.

Default value of this property is "latest"

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

100K

4.5

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

870

3.9

Quick Website Content Scraper ( Extract Text for RAG & LLMs )

automateitplease/ai-web-content-scraper-extract-text-for-rag-llms

Extract clean text from any website for AI/LLM applications. Supports both static and JavaScript-rendered sites (React, Vue, Angular). Perfect for RAG systems, chatbot training, and content analysis.

AutomateItPlease Workflow And Automaton Ops

Website Content Text Extractor

smart-digital/website-content-text-extractor

Extract visible text content from websites as structured JSON blocks. Supports multi-URL batch processing, header/footer/cookie exclusion, and optional form extraction. Perfect for content analysis and translation workflows.

My Smart Digital

5.0

Webpage to Markdown

extremescrapes/webpage-to-markdown

This actor cost-effectively converts websites into structured markdown optimized for AI processing. It extracts webpage content, formats it into clean markdown, and ensures compatibility with AI models.

Extreme Scrapes

159

5.0

Website Scraper

snipercoder/website-scraper

Scrape websites effortlessly

Sniper Coder

145

Website Content Crawler

alizarin_refrigerator-owner/website-crawler

Crawl websites for SEO audits. Extracts HTML, title, meta tags, headings, links, & text content from pages. Automatic sitemap detection & parsing Extracts metadata (title, description, OG tags) Heading structure (H1, H2, H3) Internal & external link analysis Image extraction w/alt text Word count

John Rippy

Website Content Extractor

fastidious_drawer/website-content-extractor

This extractor lets you extract content from any website with a single or multiple URLs. Use selectors to choose specific sections like the body and exclude elements like headers or navigation. It also extracts images and links, providing data in JSON and DataTable formats for easy processing.

fastidious_drawer

155

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David Deng

3.1K

4.7

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.