# GitHub Scraper — Repos, Users, Issues & Releases (`pixflor/github-scraper`) Actor

Scrape GitHub repositories, users, contributors, issues and releases from the official REST API. Gets past the 1,000-result search wall by slicing queries, and waits out rate limits instead of failing on them.

- **URL**: https://apify.com/pixflor/github-scraper.md
- **Developed by:** [Pixflor](https://apify.com/pixflor) (community)
- **Categories:** Developer tools, Lead generation, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 row scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## GitHub Scraper — Repos, Users, Issues & Releases

GitHub as clean rows: repositories with stars, forks, topics and real activity signals; user
profiles with the fields search results leave out; contributors, issues, pull requests and
releases.

No login needed. Add a free token if you want it fast.

***

### Two things this does that other GitHub scrapers don't

#### 1. It gets past the 1,000-result search wall

GitHub's search API tells you how many repositories match, then refuses to give you more than
1,000 of them. Page 11 comes back **422 — "Only the first 1000 search results are available"**.

Measured: `language:rust` reports **1,277,619 repositories** and hands over **1,000**.

This Actor splits an oversized query into star-count ranges that each fall below the ceiling,
then merges the results. The slices are disjoint by construction, so nothing is duplicated and
nothing is missed — verified in the test suite on every release.

| | Repositories reachable |
|---|---|
| A plain search query | **1,000**, then a hard 422 |
| Split into star ranges | **as many as match**, in disjoint slices |

#### 2. It waits out rate limits instead of failing on them

The real constraint on GitHub isn't the data, it's the budget: **60 requests per hour and 10
searches per minute** without a token. Most scrapers sprint into that wall and return an
error, or worse, a partial result.

This one reads `x-ratelimit-remaining` and `x-ratelimit-reset`, paces itself to the budget it
actually has, and waits for the window when it runs out. It also distinguishes a **403 that
means "rate limited"** from a 403 that means "you can't see this" — GitHub uses the same status
for both, and retrying the second one forever is a good way to waste a run.

Supply a free personal access token (no scopes needed for public data) and the ceiling becomes
5,000/hour and 30 searches/minute.

***

### What you get

**Repositories** — `fullName`, `owner`, `description`, `url`, `homepage`, `language`, `stars`,
`forks`, `watchers`, `openIssues`, `topics`, `license`, `isFork`, `isArchived`, `isTemplate`,
`defaultBranch`, `sizeKb`, `createdAt`, `updatedAt`, `pushedAt`, plus **`daysSincePush`** and
**`isActive`** — derived fields that answer "is this project alive?" without you doing date
maths on every row.

**Users** — `login`, `name`, `company`, `location`, `blog`, `email`, `twitter`, `bio`,
`hireable`, `followers`, `following`, `publicRepos`, `createdAt`.

GitHub's user *search* returns almost none of that — no name, no company, no location, no
follower count. This Actor fetches the full profile for every result, because those are the
fields anyone doing lead-gen or recruiting is actually paying for.

**Issues & pull requests** — `number`, `title`, `state`, `author`, `body`, `labels`,
`comments`, timestamps. **Pull requests are typed separately from issues**, because GitHub
returns both from the issues endpoint and counting them together silently inflates issue totals.

**Releases** — `tag`, `name`, `author`, `isPrerelease`, `body`, `assetCount`, and `downloads`
summed across all assets.

**Contributors** — `login`, `contributions`, profile URL, ranked.

***

### Modes

| Mode | What it returns |
|---|---|
| **Search repositories** | Any GitHub search, past the 1,000 wall |
| **Search users** | People and organisations, with full profiles hydrated |
| **Repositories of a user/org** | Everything one account owns |
| **Repository contributors** | Who built it, ranked by commits |
| **Repository issues & PRs** | Open, closed or all — typed apart |
| **Repository releases** | Versions, notes and download counts |

***

### Typical uses

- **Developer lead generation.** Users in a location, above a follower threshold, with company
  and website filled in.
- **Recruiting.** Contributors to the exact libraries your stack uses, ranked by contribution.
- **Competitive and ecosystem research.** Every repository in a topic or language, with
  activity signals to separate live projects from abandoned ones.
- **Dependency and supply-chain review.** Licence, archived status, last push, release cadence.
- **Market sizing.** How many projects exist in a niche, and how that changes over time.
- **Feeding an LLM or agent.** One row per item, consistent field names, no HTML.

***

### Example input

Popular Python scraping repos, no forks, no dead projects:

```json
{
  "mode": "searchRepos",
  "searchQuery": "web scraping",
  "language": "python",
  "minStars": 500,
  "excludeForks": true,
  "excludeArchived": true,
  "sortBy": "stars",
  "maxItems": 200
}
```

Open issues across specific repositories:

```json
{
  "mode": "issues",
  "repositories": ["apify/crawlee", "scrapy/scrapy"],
  "issueState": "open",
  "maxItems": 300
}
```

Who actually builds a project:

```json
{ "mode": "contributors", "repositories": ["apify/crawlee"], "maxItems": 100 }
```

***

### Settings that matter

- **`githubToken`** — the biggest lever here. Unauthenticated, GitHub allows 60 requests an
  hour; with a free personal token it's 5,000. Any run of real size wants one, and it's only
  ever used to sign requests.
- **`partition`** — on by default. This is what gets past GitHub's hard 1,000-result search
  ceiling, by slicing one query into narrower ones. Turn it off only for small searches where
  you'd rather save the extra calls.
- **`minStars` / `language` / `topic` / `license`** — these narrow the search *before* rows are
  produced, so they cut your bill as well as the noise.
- **`excludeForks` / `excludeArchived`** — forks and abandoned repos are most of what makes a
  raw GitHub search useless for research.
- **`maxItems`** — your cost ceiling. Set it deliberately.

***

### Reliability

Official REST API only. There is no HTML parsing and no bot-detection workaround anywhere in
this Actor.

A 57-assertion live test suite runs before every release, including assertions that the 1,000
wall still exists, that sliced results stay unique, that `per_page` still silently clamps at
100, and that the issues endpoint still mixes pull requests in. When GitHub changes something,
the test says which thing.

The suite is deliberately frugal with live calls and skips gracefully when the unauthenticated
budget is low — a test that fails on rate limits rather than on real problems is worse than no
test at all.

***

### Pricing

Pay per event. You are charged for the run start and for each row returned — nothing else, and
platform compute is included free. Filters that GitHub understands are pushed into the search
query itself, so they don't burn your result budget.

***

### Questions or a bug?

Open an issue on the **Issues** tab of this Actor. Include your input JSON and the run ID —
that's usually enough to reproduce it immediately.

# Actor input Schema

## `mode` (type: `string`):

Pick one. 'searchRepos' searches all of GitHub. 'searchUsers' finds people or organisations and fetches their full profiles. 'userRepos' lists everything one account owns. The last three read one or more specific repositories.

## `searchQuery` (type: `string`):

GitHub search syntax works here, e.g. 'machine learning', 'location:israel followers:>500', 'org:apify'. Required for the two search modes.

## `username` (type: `string`):

Only used by 'Repositories of a user/org'. The account whose repositories you want, e.g. apify.

## `repositories` (type: `array`):

Only used by the contributors, issues and releases modes. Accepts 'owner/repo', a GitHub URL, or a git@ remote.

## `githubToken` (type: `string`):

A free personal access token. Without one you get 60 requests/hour and 10 searches/minute, which is fine for small runs; with one you get 5,000/hour and 30 searches/minute. Needs no scopes at all for public data. Create one at github.com/settings/tokens.

## `language` (type: `string`):

Only repositories in this language, e.g. Python, Rust, TypeScript. Sent to GitHub as a search qualifier so it does not waste your result budget.

## `topic` (type: `string`):

Only repositories tagged with this GitHub topic, e.g. machine-learning, cli, kubernetes.

## `license` (type: `string`):

SPDX licence key, e.g. mit, apache-2.0, gpl-3.0.

## `minStars` (type: `integer`):

Only repositories with at least this many stars.

## `minForks` (type: `integer`):

Only repositories with at least this many forks.

## `excludeForks` (type: `boolean`):

Drop repositories that are forks of something else.

## `excludeArchived` (type: `boolean`):

Drop repositories the owner has archived.

## `activeOnly` (type: `boolean`):

Keep only repositories pushed to in the last 90 days. Useful for separating live projects from abandoned ones.

## `issueState` (type: `string`):

Only used by the issues mode. Note GitHub returns pull requests from this endpoint too - they are labelled as such in the output.

## `partition` (type: `boolean`):

GitHub only serves the first 1,000 results of any search, however many million match. With this on, an oversized query is automatically split into star-count ranges that each fall under that ceiling, so you can retrieve far more. Leave it on unless you specifically want only the top 1,000.

## `sortBy` (type: `string`):

How to order results. Stars, forks and recently-updated are passed to GitHub's own sort where possible.

## `maxItems` (type: `integer`):

How many rows to return, and what you pay for. Large values are reachable in search modes because oversized queries are split automatically.

## Actor input object example

```json
{
  "mode": "searchRepos",
  "searchQuery": "machine learning",
  "username": "apify",
  "repositories": [
    "apify/crawlee"
  ],
  "language": "rust",
  "topic": "machine-learning",
  "license": "mit",
  "minStars": 0,
  "minForks": 0,
  "excludeForks": false,
  "excludeArchived": false,
  "activeOnly": false,
  "issueState": "open",
  "partition": true,
  "sortBy": "default",
  "maxItems": 100
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("pixflor/github-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("pixflor/github-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call pixflor/github-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pixflor/github-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/egx2SXyW2msTaNjiV/builds/udJ8p3I4Cv05fN6uh/openapi.json