Craigslist Scraper (Pay Per Result)
Pricing
$10.00 / 1,000 results
Craigslist Scraper (Pay Per Result)
Extract data from classified advertisements on Craigslist. Scrape contact details from jobs, housing, items wanted, items for sale, services, community service, gigs, events and resumes listed on Craigslist. Download listings data in JSON, XML, Excel, and other versatile
0.0 (0)
Pricing
$10.00 / 1,000 results
4
Monthly users
9
Runs succeeded
>99%
Response time
12 hours
Last modified
6 days ago
How to avoid duplicate listings between actor runs?
I'm using this actor to collect rental listings and running it periodically.
Currently I'm receiving the same listings in subsequent runs, which is costing extra credits and creating duplicates in my database.
Is there a configuration option or parameter that would allow me to only receive new listings that haven't been collected in previous runs? Ideally I'd like a solution that doesn't require code modifications.
Thanks so much!
Hi evee,
Regarding duplicates, each result has id that should be unique for result. Even if you collect twice same listing, it should have the same id. So if you put in database that this column is unique, that should solve your duplication issue on database level.
The scraper is stateless, it doesn't know what was scraped in some of previous runs.
That having in mind, maybe it would be interesting for you maxAge input property. If you running the scraper daily, you could set it to 1, that way you would get only listing that were added in last 24 hours. If you are running weekly you could set it to 7, that way you would get only listing that were added in last 7 days.
In craigslist itself you can also add filter to show you only listing that were added in last 24 hours.
How often are you running the scraper?
evee
Hey, Thanks for the quick reply. So maxAge is a good idea but I want to make sure I'm not missing any listings which could happen if I'm scraping as the user is uploading to Craigslist.
I'm new Apify and would have though deduplication would be a core Apify feature that developers wouldn't need to turn on or off. Like a toggle on all actors to only scrape net new items.
I'm running the scraper daily.
Thanks!
evee
Also I tested the maxAge and that doesn't seem to be working. Looking at the search URL there's around 10 in the last 24 hours but the scraper was going up to 58 listings I had to abort, I set the maxAge to 1. Lmk if I'm doing anything incorrectly
evee
I just discovered another issue, it's not respecting the maximum charged amount I'll make another issue ticket thanks!
Let me test what is problem with maxAge. Previously it was working as expected.
I will get back to you with update.
evee
thank you!
Could you share with me your run configuration or run itself. I've tested maxAge configuration locally and on platform and it seems to work. Maybe there is some additional issue that is connected to specific search that you are using.
Pricing
Pricing model
Pay per resultThis Actor is paid per result. You are not charged for the Apify platform usage, but only a fixed price for each dataset of 1,000 items in the Actor outputs.
Price per 1,000 items
$10.00
