
Github Search Scraper
Pricing
$25.00/month + usage

Github Search Scraper
Github search scraper. Get all data from search results list
0.0 (0)
Pricing
$25.00/month + usage
4
Total users
25
Monthly users
5
Runs succeeded
>99%
Last modified
2 months ago

Fails on page 10 every time
Closed
Please see the actor run. I made a resurrect attempt, but it fails at the same page.

We are looking into it, probably anti bot protections

I successfully added my cookies, which improved the process, but there were still some errors. In the end, I managed to run 100 pages. I think it would be helpful to include a line in the README stating that the antibot sometimes fails and that it's better to add a cookie.

We found the issue, nothing related to anti bot protections:
When not providing cookies, github sometimes returns 429 responses to slow the requests made to the server
Initially, we ask cookies only if you need to search a url with type=code (mandatory for the extraction to work)
Now you should be able to handle the 100 pages without cookies