GitHub Scraper — Repos, Users, Stars & Contributors

Reads GitHub's public REST API: repository search, specific repositories, and user/organisation profiles. No token required — everything here is public data.

What you get

recordType	One per	Carries
REPOSITORY	repo	stars, real watcher count, forks, open issues, topics, licence, language, size, default branch, created/updated/pushed dates, clone URLs — plus language breakdown, contributors and releases if you switch them on
USER	login	name, company, blog, location, public email (when published), bio, followers, following, public repo/gist counts, join date
SEARCH_SUMMARY	query	GitHub's own total_count, how much of it is reachable, incomplete_results, and the remaining API budget
ERROR	failed input	a named reason — every input maps to at least one row

Three ways in, and they combine: searchQueries, explicit repositories (owner/repo), and users.

The field named watchers_count is not watchers

This is the one that quietly ruins analyses. GitHub's watchers_count is a legacy alias for the star count, and so is watchers. Measured on apify/crawlee in a single response:

stars              25,521
watchers_count     25,521   <- identical: it is stars
subscribers_count     132   <- the actual number of watchers

The field whose name says "watchers" was wrong by a factor of 193, and it looks entirely reasonable while being wrong. Worse, subscribers_count is absent from search results entirely — only the full repository endpoint carries it.

So this actor emits:

• stars — the star count, named for what it is
• watchers — the real count, and null unless the full repository was fetched, because inventing it from the alias would just be reporting stars twice
• watchersCountAliasOfStars and starsEqualWatchersCountAlias — the alias kept under a name that cannot be misread

Explicit repositories always use the full endpoint, so they always have the real number. Search results need includeFullRepo (one extra request each).

Rate limits are the real constraint

An anonymous run gets 60 core requests per hour and 10 searches per minute. That is the binding limit on this actor — not bandwidth, not concurrency.

Exhaustion arrives as HTTP 403, not 429, with a message rather than a Retry-After. The usual reaction to a 403 — rotate the fingerprint and retry — spends more of a budget that is keyed to the address, so this actor detects it from the body and the x-ratelimit-remaining header, never retries it, and stops the enrichment pass cleanly instead of emitting a wall of identical failures. Every summary carries rateLimit and skippedForRateBudget.

Budget arithmetic worth doing before a big run: each enrichment toggle costs one extra request per repository. Fifty repos with all four toggles on is 200 core requests — more than three hours of anonymous budget.

Two ways to buy more:

• githubToken — optional, off by default, and the actor is fully functional without it. Your own personal access token with no scopes selected raises core to 5,000/hour. It buys throughput, not access.
• A residential proxy — the anonymous budget is per exit address.

Search reaches 1,000 rows, whatever the count says

A query claiming 32,847,285 repositories will hand you 1,000. Page 11 is an honest HTTP 422 ("Only the first 1000 search results are available"), but the total_count sitting beside the items reads like a promise. Summaries report totalCount, estimatedReachable, estimatedUnreachable and hitSearchResultCap.

The way past it is to split the query — by language, star band, or date range — which is why searchQueries is a list.

Smaller things this handles

• incomplete_results — GitHub sets this when its own search times out and returns a partial answer with HTTP 200. It was false on every query measured here, which is exactly why it is easy to forget. Always on the summary.
• per_page above 100 is silently clamped, not refused. Both the requested and effective values are reported.
• Languages come back as raw byte counts. The percentage share is computed here.
• An unknown search qualifier is honest: scraper zzzbogus:xyz returns zero, not the unfiltered baseline (261,663 for scraper alone).

Notes

• api.github.com/robots.txt is HTTP 404, which RFC 9309 §2.3.1.3 treats as no restrictions. No WAF: 3 of 3 TLS profiles returned identical JSON.
• Search calls are paced separately (~6.5s apart) from core calls, because the two budgets are different and search is the tighter one.