Web Scraper

Here is my code. I want to call some logic whenever a url is scraped: input.startUrls = urls; const run = await client .actor(properties.apifyCredentials.actorId) .call(input); // Run x logic when a url is scraped, not when all urls are scraped at end of job const { items } = await client.dataset(run.defaultDatasetId).listItems();

Hello and thank you for your interest in this Actor!

Just to make sure I understand your question - this is more about Apify and less about this specific Actor, right?

If you want to run some code whenever an Actor produces a result, you basically have two options:

• The "easy" option: polling the Dataset API:
  • here, we retrieve the run id (and the default dataset id) and repeatedly ask the server, how many results are in the dataset.
  • this is very simple, but may be not "granular" enough for some use cases - if the Actor stores tens of results per second, the code will only see the results appear in multiples of ten (this might actually be a good thing, saving you some computation power).

1// Run the Actor and get the run ID immediately - `waitSecs: 0` makes the call resolve immediately (and returns a reference to a "running" run).
2    const { id: runId, defaultDatasetId } = await client.actor("yourActorId").call(input, { waitSecs: 0 });
3
4    const interval = setInterval(async () => {
5        const runInfo = await client.run(runId).get();
6        if (runInfo?.status !== 'RUNNING') {
7            clearInterval(interval);
8        }
9
10        const dataset = await client.dataset(defaultDatasetId).get();
11        console.log(`The dataset currently contains ${dataset?.itemCount} items.`);
12    }, 1000);

• The "precise" option: make your Actor call your own API
  • With this option, you implement some "notification" system in your Actor so it calls your server whenever it stores a new result. Please note that this might not be possible with Web Scraper due to browser CORS policies.
  • This requires much more setup - a private server reachable from the Internet, the actual server application listening on the API and the Actor-side logic.

So, the choice is yours - and largely depends on your use case for this. If the precise timing is not the top priority for you, the first option is IMO perfect.

Does this solve your problem? Let us know if you have any more questions. Thanks!