# GitHub Scraper - Repos, Users, Stars & Contributors (`scrapyx/github-repos-scraper`) Actor

Scrapes GitHub repositories, users and organisations from the public REST API: stars, forks, topics, licence, languages, contributors and releases. Reports the REAL watcher count, tracks the 60-per-hour anonymous budget, and flags the 1,000-result search cap.

- **URL**: https://apify.com/scrapyx/github-repos-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## GitHub Scraper — Repos, Users, Stars & Contributors

Reads GitHub's public REST API: repository search, specific repositories, and
user/organisation profiles. **No token required** — everything here is public
data.

### What you get

| `recordType` | One per | Carries |
| --- | --- | --- |
| `REPOSITORY` | repo | stars, **real watcher count**, forks, open issues, topics, licence, language, size, default branch, created/updated/pushed dates, clone URLs — plus language breakdown, contributors and releases if you switch them on |
| `USER` | login | name, company, blog, location, public email (when published), bio, followers, following, public repo/gist counts, join date |
| `SEARCH_SUMMARY` | query | GitHub's own `total_count`, how much of it is reachable, `incomplete_results`, and the **remaining API budget** |
| `ERROR` | failed input | a named reason — every input maps to at least one row |

Three ways in, and they combine: `searchQueries`, explicit `repositories`
(`owner/repo`), and `users`.

### The field named `watchers_count` is not watchers

This is the one that quietly ruins analyses. GitHub's `watchers_count` is a
**legacy alias for the star count**, and so is `watchers`. Measured on
`apify/crawlee` in a single response:

```
stars              25,521
watchers_count     25,521   <- identical: it is stars
subscribers_count     132   <- the actual number of watchers
```

The field whose name says "watchers" was wrong by a factor of 193, and it looks
entirely reasonable while being wrong. Worse, **`subscribers_count` is absent
from search results entirely** — only the full repository endpoint carries it.

So this actor emits:

- `stars` — the star count, named for what it is
- `watchers` — the real count, and **`null` unless the full repository was
  fetched**, because inventing it from the alias would just be reporting stars
  twice
- `watchersCountAliasOfStars` and `starsEqualWatchersCountAlias` — the alias
  kept under a name that cannot be misread

Explicit `repositories` always use the full endpoint, so they always have the
real number. Search results need `includeFullRepo` (one extra request each).

### Rate limits are the real constraint

**An anonymous run gets 60 core requests per hour and 10 searches per minute.**
That is the binding limit on this actor — not bandwidth, not concurrency.

**Exhaustion arrives as HTTP 403, not 429**, with a message rather than a
`Retry-After`. The usual reaction to a 403 — rotate the fingerprint and retry —
spends more of a budget that is keyed to the *address*, so this actor detects it
from the body and the `x-ratelimit-remaining` header, never retries it, and
stops the enrichment pass cleanly instead of emitting a wall of identical
failures. Every summary carries `rateLimit` and `skippedForRateBudget`.

Budget arithmetic worth doing before a big run: each enrichment toggle costs
**one extra request per repository**. Fifty repos with all four toggles on is
200 core requests — more than three hours of anonymous budget.

Two ways to buy more:

- **`githubToken`** — optional, off by default, and the actor is fully
  functional without it. Your own personal access token with **no scopes
  selected** raises core to 5,000/hour. It buys throughput, not access.
- **A residential proxy** — the anonymous budget is per exit address.

### Search reaches 1,000 rows, whatever the count says

A query claiming 32,847,285 repositories will hand you 1,000. Page 11 is an
honest HTTP 422 (*"Only the first 1000 search results are available"*), but the
`total_count` sitting beside the items reads like a promise. Summaries report
`totalCount`, `estimatedReachable`, `estimatedUnreachable` and
`hitSearchResultCap`.

The way past it is to **split the query** — by language, star band, or date
range — which is why `searchQueries` is a list.

### Smaller things this handles

- **`incomplete_results`** — GitHub sets this when its own search times out and
  returns a *partial* answer with HTTP 200. It was `false` on every query
  measured here, which is exactly why it is easy to forget. Always on the
  summary.
- **`per_page` above 100 is silently clamped**, not refused. Both the requested
  and effective values are reported.
- **Languages come back as raw byte counts.** The percentage share is computed
  here.
- **An unknown search qualifier is honest**: `scraper zzzbogus:xyz` returns
  zero, not the unfiltered baseline (261,663 for `scraper` alone).

### Notes

- `api.github.com/robots.txt` is HTTP 404, which RFC 9309 §2.3.1.3 treats as no
  restrictions. No WAF: 3 of 3 TLS profiles returned identical JSON.
- Search calls are paced separately (~6.5s apart) from core calls, because the
  two budgets are different and search is the tighter one.

# Actor input Schema

## `searchQueries` (type: `array`):

GitHub search queries, e.g. 'scraper language:python stars:>1000'. Each query can reach at most 1,000 results however many GitHub reports — split by language, star band or date range to get past that. An unknown qualifier is honest: it returns zero, not the unfiltered baseline.

## `repositories` (type: `array`):

Specific repositories as 'owner/repo' or a github.com URL. These always come from the full repository endpoint, so the real watcher count is available without any extra toggle.

## `users` (type: `array`):

GitHub logins, e.g. 'apify'. Returns the profile; switch on 'Include their repositories' to walk what they own.

## `sort` (type: `string`):

Leave as best match unless you need a specific ordering.

## `order` (type: `string`):

Only applies when a sort is chosen.

## `perPage` (type: `integer`):

100 is GitHub's maximum. A larger value is not refused — it is silently clamped to 100, and the summary reports both what was asked and what applied.

## `maxReposPerQuery` (type: `integer`):

Set 0 to walk to the search cap, which is 1,000 rows per query.

## `includeFullRepo` (type: `boolean`):

One extra request per repository. This is the ONLY way to get the true watcher count — GitHub's watchers\_count field in search results is a legacy alias for stars (measured: 29,979 stars, 29,979 'watchers\_count', 174 real watchers).

## `includeLanguages` (type: `boolean`):

One extra request per repository. Returns bytes per language, plus a percentage share computed here.

## `includeContributors` (type: `boolean`):

One extra request per repository, with each contributor's commit count.

## `includeReleases` (type: `boolean`):

One extra request per repository, with tag, date, prerelease flag and total asset downloads.

## `maxContributors` (type: `integer`):

Top contributors to keep per repository, ranked by commit count. Caps the row size; the contributors endpoint itself pages at 100.

## `maxReleases` (type: `integer`):

Most recent releases to keep per repository, newest first, each with its tag, date, prerelease flag and total asset download count.

## `includeUserRepos` (type: `boolean`):

For each login in 'Users or organisations', also emit the repositories they own.

## `githubToken` (type: `string`):

OPTIONAL and off by default — the actor works fully without it, because all of this data is public. A token only buys throughput: anonymous runs get 60 core requests an hour and 10 searches a minute, a token raises core to 5,000 an hour. Use your own personal access token with no scopes selected.

## `maxConcurrency` (type: `integer`):

Kept low: the constraint here is GitHub's hourly budget, not bandwidth.

## `minRequestInterval` (type: `integer`):

Politeness pacing for core requests. Search calls are paced separately at ~6.5s, because anonymous search allows only 10 per minute.

## `proxyConfiguration` (type: `object`):

Optional and OFF by default. No WAF was observed. Note the anonymous rate limit is keyed to the exit address, so a proxy changes the budget picture as well as the routing.

## Actor input object example

```json
{
  "searchQueries": [
    "language:python topic:web-scraping"
  ],
  "repositories": [
    "apify/crawlee",
    "scrapy/scrapy"
  ],
  "users": [
    "apify"
  ],
  "sort": "",
  "order": "desc",
  "perPage": 100,
  "maxReposPerQuery": 50,
  "includeFullRepo": false,
  "includeLanguages": false,
  "includeContributors": false,
  "includeReleases": false,
  "maxContributors": 25,
  "maxReleases": 10,
  "includeUserRepos": false,
  "maxConcurrency": 2,
  "minRequestInterval": 1,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "scraper language:python stars:>1000"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/github-repos-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "searchQueries": ["scraper language:python stars:>1000"] }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/github-repos-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "scraper language:python stars:>1000"
  ]
}' |
apify call scrapyx/github-repos-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/github-repos-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/EA70wgrQgaJtrOxql/builds/MZ4LPYTprvTWaj0fT/openapi.json