Pricing

Pay per usage

Try for free

Go to Store

Web Scraper

Try for free

Developed by

Apify

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

4.5 (23)

Pricing

Pay per usage

920

Total users

90K

Monthly users

4.9K

Runs succeeded

>99%

Issues response

7.8 days

Last modified

2 months ago

Developer tools

Open source

Back to issues Create new issue

Crawler took a long time to complete, way more than average

Closed

remitaffin opened this issue

It was just hanging, so I aborted it

Mustafa99

same with me.

Jindřich Bär (jindrich.bar)

Hello @remitaffin, and thank you for your interest in this Actor.

From the log, the Actor was waiting to load the page but hit a navigation timeout after 60 seconds. This can happen for various reasons - for example, slow response from the target server or proxy-related delays.

To shorten the navigation stage, you can change the "waitUntil" option from "networkidle2" to "load" or "domcontentloaded".

I also see that you've since successfully scraped the target site, so I'll go ahead and close this issue. Feel free to reach out again if you run into anything else.

Add comment

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

4.7

Puppeteer Scraper

apify/puppeteer-scraper

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.

Apify

8.4K

5.0

Web Scraping API

zeeb0t/web-scraping-api---scrape-any-website

Web Scraping API that quickly and reliably scrapes any website—no selectors required. Premium proxies, CAPTCHA solving, JavaScript rendering, and automated structured data extraction are all included. It’s just $2 per 1,000 web pages scraped, with no minimum spend.

Anthony Ziebell

1.2K

5.0

Playwright Scraper

apify/playwright-scraper

Crawls websites with the headless Chromium, Chrome, or Firefox browser and Playwright library using a provided server-side Node.js code. Supports both recursive crawling and a list of URLs. Supports login to a website.

Apify

3.6

BeautifulSoup Scraper

apify/beautifulsoup-scraper

Crawls websites using raw HTTP requests. It parses the HTML with the BeautifulSoup library and extracts data from the pages using Python code. Supports both recursive crawling and lists of URLs. This Actor is a Python alternative to Cheerio Scraper.

Apify

870

4.2

HTML Scraper pro

scrapingxpert/html-scraper-pro

The HTML Scraper Pro is a powerful tool designed to extract the HTML source code and metadata from websites. It uses advanced web scraping techniques to retrieve the full HTML content of web pages,page title and HTTP status code.This tool is ideal for data extraction, website analysis, and archiving

scrapingxpert

100

RAG Web Browser

apify/rag-web-browser

Web browser for OpenAI Assistants, RAG pipelines, or AI agents, similar to a web browser in ChatGPT. It queries Google Search, scrapes the top N pages, and returns their content as Markdown for further processing by an LLM. It can also scrape individual URLs. Supports Model Context Protocol (MCP).

Apify

4.6K

4.4

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

64K

4.3

Cloudflare Web Scraper

ecomscrape/cloudflare-web-scraper

Advanced web scraper designed to extract data from Cloudflare-protected websites with CAPTCHA bypass, proxy rotation, and JavaScript execution capabilities.

ecomscrape

144

5.0

Instant web data scraper - Scrape any website

curious_coder/instant-web-scraper

Scrape any public and private website data by providing just URL and optionally cookies and proxy information. This scraper is similar to instant data scraper but runs on cloud and can be used as API too!