GitHub Scraper — Repos, Users, Issues & Releases

GitHub as clean rows: repositories with stars, forks, topics and real activity signals; user profiles with the fields search results leave out; contributors, issues, pull requests and releases.

No login needed. Add a free token if you want it fast.

---

Two things this does that other GitHub scrapers don't

1. It gets past the 1,000-result search wall

GitHub's search API tells you how many repositories match, then refuses to give you more than 1,000 of them. Page 11 comes back 422 — "Only the first 1000 search results are available".

Measured: language:rust reports 1,277,619 repositories and hands over 1,000.

This Actor splits an oversized query into star-count ranges that each fall below the ceiling, then merges the results. The slices are disjoint by construction, so nothing is duplicated and nothing is missed — verified in the test suite on every release.

	Repositories reachable
A plain search query	1,000, then a hard 422
Split into star ranges	as many as match, in disjoint slices

2. It waits out rate limits instead of failing on them

The real constraint on GitHub isn't the data, it's the budget: 60 requests per hour and 10 searches per minute without a token. Most scrapers sprint into that wall and return an error, or worse, a partial result.

This one reads x-ratelimit-remaining and x-ratelimit-reset, paces itself to the budget it actually has, and waits for the window when it runs out. It also distinguishes a 403 that means "rate limited" from a 403 that means "you can't see this" — GitHub uses the same status for both, and retrying the second one forever is a good way to waste a run.

Supply a free personal access token (no scopes needed for public data) and the ceiling becomes 5,000/hour and 30 searches/minute.

---

What you get

Repositories — fullName, owner, description, url, homepage, language, stars, forks, watchers, openIssues, topics, license, isFork, isArchived, isTemplate, defaultBranch, sizeKb, createdAt, updatedAt, pushedAt, plus daysSincePush and isActive — derived fields that answer "is this project alive?" without you doing date maths on every row.

Users — login, name, company, location, blog, email, twitter, bio, hireable, followers, following, publicRepos, createdAt.

GitHub's user search returns almost none of that — no name, no company, no location, no follower count. This Actor fetches the full profile for every result, because those are the fields anyone doing lead-gen or recruiting is actually paying for.

Issues & pull requests — number, title, state, author, body, labels, comments, timestamps. Pull requests are typed separately from issues, because GitHub returns both from the issues endpoint and counting them together silently inflates issue totals.

Releases — tag, name, author, isPrerelease, body, assetCount, and downloads summed across all assets.

Contributors — login, contributions, profile URL, ranked.

---

Modes

Mode	What it returns
Search repositories	Any GitHub search, past the 1,000 wall
Search users	People and organisations, with full profiles hydrated
Repositories of a user/org	Everything one account owns
Repository contributors	Who built it, ranked by commits
Repository issues & PRs	Open, closed or all — typed apart
Repository releases	Versions, notes and download counts

---

Typical uses

• Developer lead generation. Users in a location, above a follower threshold, with company and website filled in.
• Recruiting. Contributors to the exact libraries your stack uses, ranked by contribution.
• Competitive and ecosystem research. Every repository in a topic or language, with activity signals to separate live projects from abandoned ones.
• Dependency and supply-chain review. Licence, archived status, last push, release cadence.
• Market sizing. How many projects exist in a niche, and how that changes over time.
• Feeding an LLM or agent. One row per item, consistent field names, no HTML.

---

Example input

Popular Python scraping repos, no forks, no dead projects:

{
  "mode": "searchRepos",
  "searchQuery": "web scraping",
  "language": "python",
  "minStars": 500,
  "excludeForks": true,
  "excludeArchived": true,
  "sortBy": "stars",
  "maxItems": 200
}

Open issues across specific repositories:

{
  "mode": "issues",
  "repositories": ["apify/crawlee", "scrapy/scrapy"],
  "issueState": "open",
  "maxItems": 300
}

Who actually builds a project:

{ "mode": "contributors", "repositories": ["apify/crawlee"], "maxItems": 100 }

---

Settings that matter

• githubToken — the biggest lever here. Unauthenticated, GitHub allows 60 requests an hour; with a free personal token it's 5,000. Any run of real size wants one, and it's only ever used to sign requests.
• partition — on by default. This is what gets past GitHub's hard 1,000-result search ceiling, by slicing one query into narrower ones. Turn it off only for small searches where you'd rather save the extra calls.
• minStars / language / topic / license — these narrow the search before rows are produced, so they cut your bill as well as the noise.
• excludeForks / excludeArchived — forks and abandoned repos are most of what makes a raw GitHub search useless for research.
• maxItems — your cost ceiling. Set it deliberately.

---

Reliability

Official REST API only. There is no HTML parsing and no bot-detection workaround anywhere in this Actor.

A 57-assertion live test suite runs before every release, including assertions that the 1,000 wall still exists, that sliced results stay unique, that per_page still silently clamps at 100, and that the issues endpoint still mixes pull requests in. When GitHub changes something, the test says which thing.

The suite is deliberately frugal with live calls and skips gracefully when the unauthenticated budget is low — a test that fails on rate limits rather than on real problems is worse than no test at all.

---

Pricing

Pay per event. You are charged for the run start and for each row returned — nothing else, and platform compute is included free. Filters that GitHub understands are pushed into the search query itself, so they don't burn your result budget.

---

Questions or a bug?

Open an issue on the Issues tab of this Actor. Include your input JSON and the run ID — that's usually enough to reproduce it immediately.