Reed.co.uk Scraper avatar

Reed.co.uk Scraper

Try for free

1 day trial then $30.00/month - No credit card required now

Go to Store
Reed.co.uk Scraper

Reed.co.uk Scraper

lexis-solutions/reed-co-uk-scraper
Try for free

1 day trial then $30.00/month - No credit card required now

The Reed.co.uk scraper is a web scraping tool that retrieves job postings from Reed.co.uk, a job search website in the UK 🇬🇧

TE

Anyway in the JSON (Using API) to Increase the RAM

Closed

testtheworkout opened this issue
7 months ago

{ "proxyConfiguration": { "useApifyProxy": false }, "startUrls": [ { "url": "https://www.reed.co.uk/jobs/data-jobs?salaryFrom=90000&contract=true&dateCreatedOffSet=lastthreedays&excludeSalaryDescriptions=16%2C32%2C64" } ], "workFromHome": false }

At the moment it's only using 2gb - I'd like to increase it 8gb as it's taken 5 to 10 times longer than my other scrapers to complete (for less records) and I think it's the RAM :)

lexis-solutions avatar

Hi, you can use the "memory" parameter of the Apify API to achieve that. More info: https://docs.apify.com/api/v2?utm_source=app#/reference/actors/run-actor-synchronously/run-task

TE

testtheworkout

7 months ago

Thank you for your quick response. If I'm honest, I'm not very technical, and I'm struggling to find how to do this in the documentation. Are you able to please share a new JSON input with the memory parameter set to 8GB included I can use? Thank you for your help on this.

lexis-solutions avatar

What tools do you use to trigger a run a scrape using the API? (Postman, Insomnia, curl, else)? Or you're using the Apify dashboard?

TE

testtheworkout

7 months ago

It uses a Python Script - I copy the JSON configuration from the dashboard into a config file (JSON) and then run it using Python in VS Code, which then triggers the script to run by an API. I think I just need the updated JSON values in the first message that I can then put into the Config file when I run the Python Script.

lexis-solutions avatar

As a clarification, the JSON in the first comment controls the actor's behaviour. The actor cannot set its own RAM limits, so providing them there makes no difference. What you want to control is Apify's behaviour, which can be done by modifying the API call to Apify. You could try getting help from

Closing the issue now, as it relates to Apify rather than the actor, feel free to re-open if anything else comes up. Have a great day!

Developer
Maintained by Community

Actor Metrics

  • 7 monthly users

  • 9 stars

  • >99% runs succeeded

  • Created in Mar 2023

  • Modified 3 months ago

Categories