Pricing

Pay per usage

Go to Store

Website Content Crawler

Try for free

Developed by

Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

4.6 (38)

Pricing

Pay per usage

1224

Monthly users

6.4k

Runs succeeded

>99%

Response time

4.6 days

Last modified

2 days ago

Developer tools

Back to issues Create new issue

Seeing many `NS_ERROR_X` errors for a crawl

Closed

embrace_ai opened this issue

I've tried a couple of configurations for crawling a particular website and it is failing to process the first page due to a variety of errors such as NS_ERROR_NET_INTERRUPT NS_ERROR_PROXY_CONNECTION_REFUSED and NS_ERROR_NET_TIMEOUT

I've tried using residential proxies, the playwright:firefox crawler, and increased the memory and timeout parameters for my task but have had no success.

The information provided by enabling debugLog and debugMode haven't helped me figure out the cause.

Thank you!

Jindřich Bär (jindrich.bar)

Hello and thank you for your interest in this Actor!

The page you are trying to scrape seems to be misconfigured - while it doesn't respond on https://bmc.com (the connection never goes through and proxy / browser eventually times out), it does respond on https://www.bmc.com.

A quick look into the DNS records shows that the apex domain (without the www) points to a different address than the www domain. The apex domain server doesn't seem to listen on the 443 port - this is why the https://bmc.com request never resolves. Interesting point - the apex domain server listens on port 80 and correctly redirects you to the www URL - i.e. going to http://bmc.com (http, not https) does work too. This definitely feels broken and probably hurts their SEO performance - if you somehow know the admin of that website, it's something they should look into.

You can easily fix your scraper run by replacing the https://bmc.com start URL with https://www.bmc.com. You can reset the rest of the settings to default (mostly proxies, RESIDENTIALS are more expensive and you can easily scrape this website with DATACENTER). Check out my example run here - you can see that WCC indeed scrapes the contents of the website.

Thank you again for bringing this up!

Add comment

Pricing

Pricing model

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

489

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

Extract-any-webpage-content-for-llm

ai-developer/extract-any-webpage-content-for-llm

Fast and easy way to extract data from any webpage and are LLM friendly. The tool lets you easily extract content from any website. Ideal for researchers, marketers, and developers.

aideveloper

418

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David Deng

559

AI-Powered Web Content & Link Extractor

scrapercoder/ai-powered-web-content-link-extractor

Crawls websites to extract clean, structured content for AI/LLM use, ideal for training datasets, knowledge bases, and RAG systems. Json output includes: * text: Normalized page content * links: Extracted sub-URLs

wallnut.ai

Dynamic Markdown Scraper

louisdeconinck/dynamic-markdown-scraper

Effortlessly feed LLM AIs with clean Markdown using our advanced web scraper. Seamlessly scrape dynamic, JavaScript-rendered websites while preserving original formatting. Ideal for AI training, documentation, and content migration.

Louis Deconinck

Web Scraper

apify/web-scraper

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

Apify

80.7k

RAG Web Browser

apify/rag-web-browser

Web browser for OpenAI Assistants, RAG pipelines, or AI agents, similar to a web browser in ChatGPT. It queries Google Search, scrapes the top N pages from the results, and returns their cleaned content as Markdown for further processing by an LLM. It can also scrape individual URLs. Supports MCP.

Apify

1.8k