Reed.co.uk Scraper
1 day trial then $30.00/month - No credit card required now
Reed.co.uk Scraper
1 day trial then $30.00/month - No credit card required now
The Reed.co.uk scraper is a web scraping tool that retrieves job postings from Reed.co.uk, a job search website in the UK 🇬🇧
Anyway in the JSON (Using API) to Increase the RAM
Closed
{ "proxyConfiguration": { "useApifyProxy": false }, "startUrls": [ { "url": "https://www.reed.co.uk/jobs/data-jobs?salaryFrom=90000&contract=true&dateCreatedOffSet=lastthreedays&excludeSalaryDescriptions=16%2C32%2C64" } ], "workFromHome": false }
At the moment it's only using 2gb - I'd like to increase it 8gb as it's taken 5 to 10 times longer than my other scrapers to complete (for less records) and I think it's the RAM :)
Lexis Solutions (lexis-solutions)
Hi, you can use the "memory" parameter of the Apify API to achieve that. More info: https://docs.apify.com/api/v2?utm_source=app#/reference/actors/run-actor-synchronously/run-task
testtheworkout
Thank you for your quick response. If I'm honest, I'm not very technical, and I'm struggling to find how to do this in the documentation. Are you able to please share a new JSON input with the memory parameter set to 8GB included I can use? Thank you for your help on this.
Lexis Solutions (lexis-solutions)
What tools do you use to trigger a run a scrape using the API? (Postman, Insomnia, curl, else)? Or you're using the Apify dashboard?
testtheworkout
It uses a Python Script - I copy the JSON configuration from the dashboard into a config file (JSON) and then run it using Python in VS Code, which then triggers the script to run by an API. I think I just need the updated JSON values in the first message that I can then put into the Config file when I run the Python Script.
Lexis Solutions (lexis-solutions)
As a clarification, the JSON in the first comment controls the actor's behaviour. The actor cannot set its own RAM limits, so providing them there makes no difference. What you want to control is Apify's behaviour, which can be done by modifying the API call to Apify. You could try getting help from
- Google/ChatGPT on how to do this by providing the API documentation we provided above
- The Apify Discord https://discord.gg/crawlee-apify-801163717915574323
- If nothing helps, contact us on our website https://www.lexis.solutions/ to share the python script privately, and we'll help you tweak it.
Closing the issue now, as it relates to Apify rather than the actor, feel free to re-open if anything else comes up. Have a great day!
Lexis Solutions- 1 monthly user
- 9 stars
- 100.0% runs succeeded
- 1.9 days response time
- Created in Mar 2023
- Modified 29 days ago
