
Web Scraper
Pricing
Pay per usage

Web Scraper
Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.
4.5 (22)
Pricing
Pay per usage
676
Total users
81.6k
Monthly users
3.9k
Runs succeeded
>99%
Response time
31 days
Last modified
10 days ago
How to increase memory use to speed up scraping?
Closed
I noticed that in each web scraper run, only 4GB is consumed and it takes about 30s to 1m++ to scrap once which seems slow. How can I speed this up? How can I increase the mem usage? Attached is my json config.
incomparable_pad
Btw, on your console. I've set 32GB as my mem setting.
incomparable_pad
Any updates?
Hello, assuming that you're starting all your runs via API, you'll need to add a ?memory=[desired megabytes] e.g. ?memory=8192 param to the URL. More info at: https://docs.apify.com/api/v2#/reference/actors/run-collection/run-actor
incomparable_pad
We're using a plugin and I don't think that plugin uses API. It sends a JSON file to you. Below is the JSON file and I added "memory" element into the JSON data but the memory consumed by each run is still 4GB and not 8GB.
{
"runMode": "DEVELOPMENT",
"startUrls": [
{
"url": "https://www.pricefh.net/listings?pg=13",
"method": "GET"
}
],
"keepUrlFragments": false,
"linkSelector": "a[href]",
"pseudoUrls": [
{
"purl": "https://apify.com[(/[\w-]+)?]"
}
],
"pageFunction": "// The function accepts a single argument: the "context" object.\n// For a complete list of its properties and functions,\n// see https://apify.com/apify/web-scraper#page-function \nasync function pageFunction(context) {\n // This statement works as a breakpoint when you're trying to debug your code. Works only with Run mode: DEVELOPMENT!\n // debugger; \n\n // jQuery is handy for finding DOM elements and extracting data from them.\n // To use it, make sure to enable the "Inject jQuery" option.\n const $ = context.jQuery;\n const pageTitle = $('title').first().text();\n const pageContent = $('html').html();\n\n // Print some information to actor log\n context.log.info(URL: ${context.request.url}, TITLE: ${pageTitle}
);\n\n // Manually add a new page to the queue for scraping.\n // To make this work, make sure the "Use request queue" option is enabled.\n //context.enqueueRequest({ url: 'http://www.example.co... [trimmed]
Hello, thank you for providing more details. Since this is not really actor related issue, but it's more concerning your private settings, I'd please ask you to contact our support team here: https://apify.com/contact and choose the open live chat option in the How to get in touch with us section. My colleagues on support will be more than happy to assist you.
incomparable_pad
Open live chat is an AI bot. It doesn't understand my prob. Can I speak with someone more experienced to handle my matter? Please provide me an email address or ask them to read this support. I want to upgrade to your yearly plan but can't do it if support is under par.
Hello, once in the live chat you can click on one of the options e.g. "I need help with something else" and you'll be able to describe your problem, which will be then attentively answered by one of my colleagues on the technical support. Alternatively, you can also email support@apify.com, which will be handled by the same support team as in the live chat.
incomparable_pad
HI Misa. With all due respect, but it's overwhelmingly perplexing to me why you're making me doing more work just to request for tech support from Apify? Are you not part of Apify? It seems i have to communicate around you just to get my message through. And this is not the first time this has happened. Can't you just share my issue with your tech team and get them to reply this issue? Why must I create TWO tickets for the same issue? It seems other APIs from lesser countries handle their support system better than here.
incomparable_pad
I'm completely not satisfied with the way how you are handling this matter. And I'm not asking you to move mountains.
Hello Melvin, I'm very sorry for the poor experience you've had so far and any subsequent inconvenience and frustration this has created. We’ll consider your feedback to handle similar issues more smoothly in the future.

Hey, I heard you managed to solve the issue on support, so closing this one.