No credit card required

Extended GPT Scraper

drobnikj/extended-gpt-scraper

No credit card required

Extract data from any website and feed it into GPT via the OpenAI API. Use ChatGPT to proofread content, analyze sentiment, summarize reviews, extract contact details, and much more.

Do you want to learn more about this Actor?

Get a demo

Back to issues Create new issue

lot of problems with large crawls

Closed

knowing_omen opened this issue

Hello,

If you could have a look at our actors log, because we have around 3000 websites to crawl, it goes well on the begining and after a lot of errors appears when crawling very basic websites. I'm ok with the ssl errors, but not the websites that have no problems.

Could you please try to find tune your script to avoid so many retries of websites that load correctly ?

knowing_omen

pwoChdGX9BndgCIrJ

knowing_omen

Anyone here ?

Lukáš Průša (lukas.prusa)

Hi Hosting, thanks for opening this issue!

Yes, I can confirm that this is a bug in the scraper. Exactly as you say, this is a problem for large crawls.

Basically, the crawler tries to automatically scale up and down the requests concurrency, but it looks like for large enough crawls with a lot of memory, it scales too rapidly up and down periodically. We will investigate this and find a proper value for the scaling function so that it stays in an optimal speed.

Btw we will look into the SSL error, I think we could adjust the browser to ignore it, because that's most likely just a website security concern. Though I can't promise that, because it might not be possible to remove that from the browser.

I will keep you updated here, thanks!

knowing_omen

ok thank you, keep us in touch please, more large crawls to come

Lukáš Průša (lukas.prusa)

Hi again, thanks for your patience!

We've just updated the scraper with both of the fixes :) It will now scale accordingly and ignore the HTTPS certificate errors for the broken websites. Note that it will scale up a little slower than previously, but at least it should not overshoot it anymore.

Try it out and let me know how it works, thanks!

Add comment

Developer

Jakub Drobník

Actor Metrics

61 monthly users
55 stars
99% runs succeeded
7.6 days response time
Created in Jun 2023
Modified 21 days ago

Categories

Lead generation

Open source

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

29.6k

767

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract data from hundreds of Facebook posts from one or multiple Facebook pages and profiles. Get post URL, post text, page or profile URL, timestamp, number of likes, shares, comments, and more. Download the data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

16k

215

Contact Details Scraper

vdrmota/contact-info-scraper

Free email extractor and lead scraper to extract and download emails, phone numbers, Facebook, Twitter, LinkedIn, and Instagram profiles from any website. Extract contact information at scale from lists of URLs and download the data as Excel, CSV, JSON, HTML, and XML.

Vojta Drmota

25.8k

247

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

5.8k

📩📍 Google Maps Email Extractor

lukaskrivka/google-maps-with-contact-details

Extract Google Maps contact details. Scrape websites of Google Maps places for contact details and get email addresses, website, location, address, zipcode, phone number, social media links. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Lukáš Křivka

9.2k

252

Instagram Scraper

apify/instagram-scraper

Scrape and download Instagram posts, profiles, places, hashtags, photos, and comments. Get data from Instagram using one or more Instagram URLs or search queries. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

63.6k

647

Facebook Groups Scraper

apify/facebook-groups-scraper

Extract data from one or multiple public Facebook groups. Get group and post URLs, post text, comments, timestamp, likes and comments count, and basic commentator info. Download the data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

7.6k

Facebook Comments Scraper

apify/facebook-comments-scraper

Extract data from hundreds of Facebook comments from one or multiple Facebook posts. Get comment text, timestamp, likes count and basic commenter info. Download the data in JSON, CSV, Excel and use it in apps, spreadsheets, and reports.

Apify

7.3k

Google Maps Extractor

compass/google-maps-extractor

Extract data from hundreds of places fast. Scrape Google Maps by keyword, category, location, URLs & other filters. Get addresses, contact info, opening hours, popular times, prices, menus & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.