No credit card required

Website Content Crawler

apify/website-content-crawler

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Do you want to learn more about this Actor?

Get a demo

Back to issues Create new issue

Failed Crawling for G2 web pages

Closed

motivated_leaflet opened this issue

Problem: Crawling requests for G2 never succeeds in runs after 10 session rotations. This is consistent with run over the last few days.

Not expecting this as there are G2 scraping actors and the site overall does not seem anti-scraping. Thank you!

Lukáš Křivka (lukaskrivka)

Hello,

Thanks for the report. Actually, g2.com has one of the strongest Cloudflare protection we have seen (at least some types of pages). There is a customized approach to scrape it, the team will look into how to incorporate it to this actor.

motivated_leaflet

Thank you! Looking forward to the update. G2 is a pretty key part of our project use case.

Jiří Spilka (jiri.spilka)

Hi, I apologize for the very late response. :(
I’m currently reviewing the issues, attempting to reproduce them, and providing solutions.

There have been many improvements to the Website Content Crawler and the underlying Crawlee library over time.

I tested it again, and while the protection remains strong, we are now able to crawl g2.com pages. However, the crawling process may take some time as session rotation is required when the crawler gets blocked.

12024-12-18T07:59:07.726Z WARN  AdaptiveCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session...
22024-12-18T07:59:07.727Z Received blocked status code: 403

Please see my example run with residential proxy settings. I aborted it to avoid wasting resources.

I’ll close this issue for now, but feel free to ask any questions or raise a new issue.

Add comment

Developer

Apify

Actor Metrics

4.2k monthly users
873 stars
>99% runs succeeded
23 hours response time
Created in Mar 2023
Modified 5 days ago

Categories

Developer tools

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David Deng

190

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David Deng

120

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

278

Example Website Screenshot Crawler

dz_omar/example-website-screenshot-crawler

Automated website screenshot crawler using Pyppeteer and Apify. This open-source actor captures screenshots from specified URLs, uploads them to the Apify Key-Value Store, and provides easy access to the results, making it ideal for monitoring website changes and archiving web content.

Omar Abdlhakim

Web Scraper

apify/web-scraper

Crawls arbitrary websites using the Chrome browser and extracts data from pages using JavaScript code. The Actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.

Apify

74.1k

350

Video Link Crawler

infoweaver/video-link-crawler

Effortlessly discover and extract video links from any website with our powerful Video Link Crawler within few seconds. Starting from a specified URL, it navigates through web pages, identifies video content, and compiles structured datasets.! Try it Now!

InfoWeaver

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

82.8k

749

📩📍 Google Maps Email Extractor

lukaskrivka/google-maps-with-contact-details

Extract Google Maps contact details. Scrape websites of Google Maps places for contact details and get email addresses, website, location, address, zipcode, phone number, social media links. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Lukáš Křivka

11.8k

349

Glassdoor Scraper (Pay Per Result)

memo23/glassdoor-scraper-ppr

Extract comprehensive employee reviews from Glassdoor effortlessly. Gather ratings, comments, and company responses to power your HR insights and market research.

Muhamed Didovic

Amazon Product Scraper

junglee/Amazon-crawler

Use this Amazon scraper to collect data based on URL and country from the Amazon website. Extract product information without using the Amazon API, including reviews, prices, descriptions, and Amazon Standard Identification Numbers (ASINs). Download data in various structured formats.