Can't get pagination working.

Hi,

Thanks for your question! The issue is that links are being enqueued before the new page content has fully loaded. To fix this, you must wait for the new content to appear before proceeding. You can do this by adding this line:

await waitFor(`.search-result-article:nth-child(${i*10 + 1})`, { timeoutMillis });

at the end of your loop. Altogether, your Page Function (or at least the inner loop) should look something like this:

1for (let i = 1; i < 10; i++) {
2            log.info('Waiting for the "Mehr laden" button.');
3            try {
4                // Default timeout first time.
5                await waitFor('input[value="Mehr laden"]', { timeoutMillis });
6            } catch (err) {
7                // Ignore the timeout error.
8                log.info('Could not find the "Mehr laden", '
9                    + 'we\'ve reached the end.');
10                break;
11            }
12            log.info('Clicking the "Mehr laden" button.');
13            $(buttonSelector).click();
14
15            await waitFor(`.search-result-article:nth-child(${i*10 + 1})`, { timeoutMillis });
16        }

The :nth-child CSS pseudoclass ensures that the (nth*10+1st) (so 11th, 21st, 31st...) search result is loaded before you click the button for second, third... time.

I'll close this issue now, but feel free to ask additional questions if you have any. Cheers!

Thanks for your quick reply. I tested it already but it won't work.

https://console.apify.com/actors/runs/2LpcOC0aIFHE3skqG#log

Sorry, I forgot to mention this. This is happening because the timeout you're setting in the Page Function is too short.

In my Run, I updated the code to timeoutMillis = 3000 (3 seconds), which makes the Actor work as expected. See my run here - I aborted it prematurely, but you can see that the Actor enqueues 101 requests.

Feel free to shoot us a message if anything else is unclear. Cheers!

Thank you, now all crystal clear. You are the MAN.

Can I ask you a question?

https://firmen.wko.at/-/?produkt=%22bastlerbedarf%22+%22bauglas%22+%22bauholz%2c+konstruktionsholz%2c+zimmermannsware%22+%22boden-%2c+wand-+und+deckenbel%c3%a4ge%22+%22dachmaterial+%28vom+ziegel+bis+zur+teerpappe%29%22+%22dicht-+und+d%c3%a4mmaterial+und+isolierungen%22+%22drainagen%2c+rohre%2c+schachte%22+%22elektrobedarf+%28kabel%2c+schalter%2c+steckdosen%2c...%29%22+%22estriche%2c+fassaden%2c+verputze%22+%22farben%2c+schutzanstriche%22+%22fenster%22+%22fertigteilh%c3%a4user%22+%22fliesen+und+sanit%c3%a4rbedarf%22+%22gartenbedarf+und+gartengestaltung%22+%22grundbaustoffe+%28ziegel%2c+zement%2c+%26quot%3baufgehendes+mauerwerk%26quot%3b%2c+decken%2c...%29%22+%22holzh%c3%a4user%22+%22maschinen%2c+werkzeuge%2c+befestigungsmaterial%22+%22sauna%22+%22schwimmbad%22+%22sonnenschutz%22+%22treppen+und+gel%c3%a4nder%22+%22trockener+innenausbau%22+%22t%c3%bcren%2c+tore%22+%22verschalungstechnik+und+ger%c3%bcste%22+%22wintergarten%22&filter=7&firma=

From the url above why the scraper only scrapes 111 results it should be 250? And even from 111 results it leaves 3-4 always out, don't understand why.

Thank you if you have time for me.

I think this is caused by the condition in your for loop - see:

1... 
2for (i=1;i<11;i++) {
3...

If each button press (in each loop iteration) loads only 10 results, this code can load only 11 * 10 = 110 results. Try bumping this limit to a higher number or switching this for loop for a while loop (checking the existence of the button) or similar. You might also want to bump the Page Function timeout, so that the Actor manages to enqueue all the pages before it hits the timeout.

See my example run with a higher search results limit (30 iterations - this is just an arbitrary number higher than 25) and a longer Page Function timeout:

https://console.apify.com/view/runs/jPsIk8B85zdBwBzdB

And even from 111 results it leaves 3-4 always out, don't understand why.

I'm not too sure about this part, I don't see this in my run, though.

Hope this helped. Feel free to shoot us another message if anything else is unclear. Cheers!

You are so right, I should have known that sorry.

Thank you for your patience.