Hey mate, I'll definitely check the run and see if the limit is from Glassdoor or something.

Regarding your second question, if the date is already present in the output, just export it to CSV or Excel and sort by that column. Let me know if that works for you!

Hey mate, thanks for the swift answer!

I know I can do that, thanks! I was wondering if it was possible to filter from the source, before scraping, so I can get say only the latests reviews

I see, you want to get the new ones, and the ones you scraped you want to filter out. I can do something for it...

I just checked your URL that you scraped for Glassdoor, seems I can get only 169 items when I go directly to their site, here is my last page: https://www.glassdoor.es/Opiniones/Iberdrola-Opiniones-E10548_P17.htm?filter.iso3Language=spa can you check as well and let me know :)

I think maybe you get only the reviews that are in Spanish language or something, not the ones that are in English can you confirm this?

Hey you are right, we got confused because your other scraper for indeed retrieves all the reviews, but that has to do with how the website works, not with the scraper.

in your experience, do you know if glassdoor has an URL that provides all reviews without filtering for language?

can you send me the link or something of that so I can compare the runs and data scraped?

try to scrape over their English version and let me know how it goes?

Indeed movistar run: https://console.apify.com/actors/dF5kdT7Tb2oPi4DIY/runs/iQ1VhD5KXgrVoVkck#log Glassdoor movistar english run: https://console.apify.com/actors/cAbCkTzBPP0HFO50f/runs/gi58NVIemN6SjSjcW#log Glassdoor movistar spanish run: https://console.apify.com/actors/cAbCkTzBPP0HFO50f/runs/cYE5GrKLeEZjkd19x#log

I noticed something, on the glassdoor scraper, for the movistar english scrape, it successfully detects 76 items (see picture), but then it continues to add more items to the items lenght variable, also some reviews seem to be duplicated (see other picture)

On the website you can see that in english Movistar has 76 reviews, in Spanish it has 861 reviews but neither number seems to be reflected on the results screen.

are these quirks product of mistakes on my end or is there something I'm missing?

I see now, I misread it, we can't compare two sites as each has its own data...

I think regarding Glassdoor we can't trust the numbers, and probably you can't scrape all reviews directly maybe I can try a few tricks to see if I can scrape all the reviews. Also I scrape all reviews as long as they return items/reviews, so I don't look page numbers or something....