Web Scraper avatar
Web Scraper

Pricing

Pay per usage

Go to Store
Web Scraper

Web Scraper

Developed by

Apify

Apify

Maintained by Apify

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

4.5 (22)

Pricing

Pay per usage

676

Total users

81.6k

Monthly users

3.9k

Runs succeeded

>99%

Response time

31 days

Last modified

10 days ago

IP

How to increase memory use to speed up scraping?

Closed

incomparable_pad opened this issue
2 years ago

I noticed that in each web scraper run, only 4GB is consumed and it takes about 30s to 1m++ to scrap once which seems slow. How can I speed this up? How can I increase the mem usage? Attached is my json config.

IP

incomparable_pad

2 years ago

Btw, on your console. I've set 32GB as my mem setting.

IP

incomparable_pad

2 years ago

Any updates?

misa avatar

Hello, assuming that you're starting all your runs via API, you'll need to add a ?memory=[desired megabytes] e.g. ?memory=8192 param to the URL. More info at: https://docs.apify.com/api/v2#/reference/actors/run-collection/run-actor

IP

incomparable_pad

2 years ago

We're using a plugin and I don't think that plugin uses API. It sends a JSON file to you. Below is the JSON file and I added "memory" element into the JSON data but the memory consumed by each run is still 4GB and not 8GB.

{ "runMode": "DEVELOPMENT", "startUrls": [ { "url": "https://www.pricefh.net/listings?pg=13", "method": "GET" } ], "keepUrlFragments": false, "linkSelector": "a[href]", "pseudoUrls": [ { "purl": "https://apify.com[(/[\w-]+)?]" } ], "pageFunction": "// The function accepts a single argument: the "context" object.\n// For a complete list of its properties and functions,\n// see https://apify.com/apify/web-scraper#page-function \nasync function pageFunction(context) {\n // This statement works as a breakpoint when you're trying to debug your code. Works only with Run mode: DEVELOPMENT!\n // debugger; \n\n // jQuery is handy for finding DOM elements and extracting data from them.\n // To use it, make sure to enable the "Inject jQuery" option.\n const $ = context.jQuery;\n const pageTitle = $('title').first().text();\n const pageContent = $('html').html();\n\n // Print some information to actor log\n context.log.info(URL: ${context.request.url}, TITLE: ${pageTitle});\n\n // Manually add a new page to the queue for scraping.\n // To make this work, make sure the "Use request queue" option is enabled.\n //context.enqueueRequest({ url: 'http://www.example.co... [trimmed]

misa avatar

Hello, thank you for providing more details. Since this is not really actor related issue, but it's more concerning your private settings, I'd please ask you to contact our support team here: https://apify.com/contact and choose the open live chat option in the How to get in touch with us section. My colleagues on support will be more than happy to assist you.

IP

incomparable_pad

2 years ago

Open live chat is an AI bot. It doesn't understand my prob. Can I speak with someone more experienced to handle my matter? Please provide me an email address or ask them to read this support. I want to upgrade to your yearly plan but can't do it if support is under par.

misa avatar

Hello, once in the live chat you can click on one of the options e.g. "I need help with something else" and you'll be able to describe your problem, which will be then attentively answered by one of my colleagues on the technical support. Alternatively, you can also email support@apify.com, which will be handled by the same support team as in the live chat.

IP

incomparable_pad

2 years ago

HI Misa. With all due respect, but it's overwhelmingly perplexing to me why you're making me doing more work just to request for tech support from Apify? Are you not part of Apify? It seems i have to communicate around you just to get my message through. And this is not the first time this has happened. Can't you just share my issue with your tech team and get them to reply this issue? Why must I create TWO tickets for the same issue? It seems other APIs from lesser countries handle their support system better than here.

IP

incomparable_pad

2 years ago

I'm completely not satisfied with the way how you are handling this matter. And I'm not asking you to move mountains.

misa avatar

Hello Melvin, I'm very sorry for the poor experience you've had so far and any subsequent inconvenience and frustration this has created. We’ll consider your feedback to handle similar issues more smoothly in the future.

zuzka avatar

Hey, I heard you managed to solve the issue on support, so closing this one.