Hey mate, did you use my 'deduplication'(in Apify they call it also monitoring mode) option in UI, just give it a name and it will store all jobs (in the first run) and next time when you run it, it will skip the ones that have already scraped. I hope this answers your question

But how exactly does that work? The information tab doesn't explain it well enough for me. Where is the info being stores? For how long is the data stored? What are the limitatons?

And secondly, I'd still really like to only scrape jobs that were posted within the last 24 hours. The deduplication is a different issue.

The jobs are stored in the Key Value Store of Apify; everything is on this platform, and data is stored as long as you want without limitations.

You're making it more complicated than it is. On Indeed, use the filter 'Last 24 hours' and enter the keyword you want, then copy the URL into the scraper and write something in the Deduplication input. That's it. Indeed will always provide you with jobs from the last 24 hours related to your search criteria, and my scraper will handle the deduplication or monitoring.

I have a question: I see that the 'deduplication' input confuses you. I was thinking if I could simplify it. What if, instead of the deduplication input, I added just a radio button that says: 'Monitoring mode, get only new items...' Would this be better and help simplify things?

Haha i tend to overcomplicate things, sorry. It's just a bit unclear in the information tab.

Now on, indeed, I am doing what you suggested. I have it set to only show jobs that are from the last 24 hours, however I for some reason and extracting jobs that are beyond 24 hours. That's why I wanted to see if we can add an additional parameter within the actor itself.

Now, thank you for the explanation and clarification on the deduplication stuff, I am surprised that the data is stored without limitations. My main confusion was that this wasn't really specified, so I didn't really know how it works. So I will go ahead and run some tests with a data store and hopefully it solves the duplication issue for the most part. In terms of the issue, unless I think you can add a specific filter, for some reason the filter URL isn't quite cutting it.

hehe, no problem mate.

I would say, let's see how it goes. Give it a try and let me know. If you still have problems with the 'date' thing, I'll add something to the UI like you suggested, just to include items that are in that date range or something.