# GitHub Repository Metadata & README Extractor (`nimbryxo-studio/github-repository-metadata-and-readme-extractor`) Actor

Fetch stars, forks, topics, license, language breakdown and README text for GitHub repositories, by search query or a direct list of repository URLs. An optional personal token raises the request limit.

- **URL**: https://apify.com/nimbryxo-studio/github-repository-metadata-and-readme-extractor.md
- **Developed by:** [Nimbryxo-studio](https://apify.com/nimbryxo-studio) (community)
- **Categories:** Developer tools, Automation, Open source
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.49 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## GitHub Repository Metadata & README Extractor

Pull one clean, flat row per public GitHub repository: identity and owner, stars, forks, the real watcher count, topics, license, language byte counts, README text, and optional releases, contributors, and 52 weeks of commit activity. Find repositories with a GitHub search expression, your own URL list, a user/organisation listing, or a topic. No token, cookie, or proxy setup is required.

### What this actor does

- **Four discovery modes** — search by query, fetch an explicit list of URLs or `owner/repo` slugs, list every repository of a user or organisation, or browse a topic.
- **Search qualifiers become form fields** — language, license, minimum stars/forks, created/pushed dates, `in:` scope, fork/archive toggles, sort key, and sort direction. You can also paste any native GitHub qualifier into the query itself.
- **Complete repository metadata** — every reliable field from GitHub's repository payload, including visibility, clone/git/ssh URLs, feature flags, topics count, license identifiers, and the real subscriber (watcher) count.
- **README text** — fetched from GitHub's raw content host first, then the API when the file uses another name, decoded as UTF-8, and capped at 500 KB.
- **Optional enrichment** — releases, ranked contributors, 52-week commit activity, and the latest commit, each independently switchable.
- **Rate-limit aware** — reads GitHub's headers, reports the remaining budget in the run status, and keeps the base records if the anonymous core budget runs out mid-run.
- **HTML fallback** — in direct mode, if the core API budget is exhausted, repository counters are recovered from the public repository page so the run still returns real rows.

### Output per repository

- `sourceUrl`, `htmlUrl`, `recordType`, `scrapedAt` — canonical page URL and provenance
- `id`, `nodeId`, `name`, `fullName` — identity
- `owner` — `login`, `id`, `avatarUrl`, `htmlUrl`, `type`
- `description`, `homepage`, `primaryLanguage`, `visibility`, `defaultBranch`
- `stars`, `forks`, `watchersCount`, `watchers` (real watcher count, when known), `subscribersCount`, `networkCount`, `openIssues`, `size`
- `topics[]`, `topicsCount`
- `license` — `spdxId`, `name`, `url`, `key`
- `createdAt`, `updatedAt`, `pushedAt`
- `isFork`, `isArchived`, `isDisabled`, `isTemplate`, `isPrivate`
- `hasIssues`, `hasProjects`, `hasDownloads`, `hasWiki`, `hasPages`, `hasDiscussions`, `allowForking`, `webCommitSignoffRequired`
- `cloneUrl`, `sshUrl`, `gitUrl`, `svnUrl`, `mirrorUrl`
- `readme` — `name`, `path`, `content`, `size`, `htmlUrl`, `downloadUrl`
- `languages{}` — byte count per language, plus `languageCount`
- `latestRelease`, `releases[]` — tag, name, draft/prerelease flags, dates, asset count
- `contributors[]` — login, id, type, contributions, profile URL; plus `contributorCount` and `hasMoreContributors`
- `commitActivity[]`, `commitActivityTotal`, `activeWeeks`, `latestCommit` — 52-week history and the newest default-branch commit
- `searchScore` — GitHub's relevance score, search mode only

Empty fields are omitted at every depth; a field a repository genuinely lacks is never returned as `null`. A repository that cannot be fetched produces an error row with `type: "github_repo_intelligence_error"`, a `reason` (`not_found`, `invalid_url`, `fetch_failed`, `rate_limit`), a `message`, and the identifier that failed, so one bad URL does not stop the batch.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `mode` | string | `"search"` | `search`, `direct`, `owner`, or `topic` |
| `searchQuery` | string | `""` | GitHub search expression (mode=search); optional keyword filter in owner/topic mode |
| `repositoryUrls` | array | `[]` | Repository URLs or `owner/repo` slugs (mode=direct) |
| `owner` | string | `""` | User or organisation to list (mode=owner) |
| `topic` | string | `""` | Topic tag to browse (mode=topic) |
| `sortBy` | string | `"stars"` | `stars`, `forks`, `updated`, or `help-wanted-issues` |
| `sortOrder` | string | `"desc"` | `desc` or `asc` |
| `language` | string | `""` | Common language dropdown; empty means any |
| `languageCustom` | string | `""` | Any other GitHub language name; overrides the dropdown |
| `license` | string | `""` | License dropdown; empty means any |
| `searchIn` | string | `""` | Where the free text matches: name/description/readme/topics |
| `minStars` | integer | `0` | Minimum star count; 0 means no filter |
| `minForks` | integer | `0` | Minimum fork count; 0 means no filter |
| `createdAfter` | string | `""` | Only repositories created on/after `YYYY-MM-DD` |
| `pushedAfter` | string | `""` | Only repositories pushed on/after `YYYY-MM-DD` |
| `includeForks` | boolean | `false` | Include forks in search/owner/topic results |
| `includeArchived` | boolean | `true` | Keep archived repositories in the results |
| `maxResults` | integer | `30` | Hard cap on repositories returned (1–1,000) |
| `includeReadme` | boolean | `true` | Include the decoded README |
| `includeTopics` | boolean | `true` | Include topic tags |
| `includeLanguages` | boolean | `true` | Include language byte counts |
| `includeRepoDetails` | boolean | `false` | Add real watcher/network counts in search, owner, and topic modes |
| `includeReleases` | boolean | `false` | Include the newest releases |
| `releasesLimit` | integer | `10` | Releases kept per repository (1–100) |
| `includeContributors` | boolean | `false` | Include the ranked contributor list |
| `contributorsLimit` | integer | `30` | Contributors kept per repository (1–500) |
| `includeActivity` | boolean | `false` | Include 52-week commit activity and the latest commit |
| `githubToken` | string | `""` | Optional personal access token; only raises limits |

The daily test prefill is `mode=search`, `searchQuery="stars:>10000 language:python"`, `repositoryUrls=["https://github.com/apify/crawlee"]`, `maxResults=30`; it uses no token and returns 30 rows.

#### Example: search for maintained Rust projects

```json
{
  "mode": "search",
  "searchQuery": "language:rust stars:>5000",
  "pushedAfter": "2025-01-01",
  "sortBy": "stars",
  "maxResults": 100
}
```

#### Example: fetch a known list

```json
{
  "mode": "direct",
  "repositoryUrls": [
    "https://github.com/apify/crawlee",
    "pallets/flask",
    "git@github.com:psf/requests.git"
  ],
  "includeReleases": true,
  "includeContributors": true,
  "includeActivity": true
}
```

#### Example: every repository of an organisation

```json
{
  "mode": "owner",
  "owner": "apify",
  "language": "python",
  "minStars": 100,
  "sortBy": "updated"
}
```

#### Example: browse a topic

```json
{
  "mode": "topic",
  "topic": "llm",
  "includeForks": false,
  "maxResults": 200
}
```

### Use cases

- **Dependency due diligence** — check license, archived status, last push, latest release, contributor depth, and commit activity before adopting a library.
- **Ecosystem mapping** — enumerate `topic:llm`, `language:rust`, or an organisation's repository portfolio.
- **Leaderboards and dashboards** — rank a repository set by stars, forks, or recent activity with stable numeric fields.
- **LLM and RAG pipelines** — pair structured metadata with the full README text for retrieval or code analysis.
- **Release and community tracking** — follow release cadence and contributor counts over time.

### Rate limits and behaviour

GitHub allows about **60 core requests per hour** and **10 search requests per minute** without a token. This actor:

- tracks `X-RateLimit-Remaining`, `X-RateLimit-Limit`, and the reset time on every response and reports them in the run status;
- fetches READMEs from GitHub's raw content host, which does not consume the core budget;
- keeps emitting base records and skips only optional enrichment when the core budget is exhausted;
- retries `429` and `5xx` responses with backoff, handles secondary rate limits, and retries a `202` statistics response once;
- treats a rejected optional token as a soft failure and continues anonymously.

A personal access token raises the limits to 5,000 core requests per hour and 30 search requests per minute. It is optional and never used by the daily test.

### Limitations

- **Search is capped at 1,000 results per query** by GitHub. Split a large universe into `stars:` ranges (for example `stars:10000..50000`, then `stars:5000..9999`) and run once per range.
- **The contributor endpoint is capped at 500 entries** by GitHub. `hasMoreContributors` is set when GitHub's `Link` header shows more pages.
- **Commit activity can take a moment to compute.** If GitHub still returns `202` after one retry, the field is omitted for that repository instead of invented.
- **Private repositories** require a token with access to them. The actor never asks for cookies or a login.
- **Sorting by "Recently updated" follows GitHub's own ordering, which is approximate.** Re-sort on the `updatedAt` field if you need a strict order. Rows are pushed as enrichment finishes, so use `index` (the search rank) to restore the source order.
- **`license.url` points at GitHub's license API** (`api.github.com/licenses/<id>`). Like any API URL it returns 403 while the anonymous hourly quota is exhausted; the dataset itself is unaffected.
- **No issues, pull requests, discussions, or trending data.** This actor focuses on repository metadata and README content.

### FAQ

**Do I need a GitHub token?** No. Small runs work anonymously. Supply a token only for larger batches or private repositories.

**Why is `watchers` missing on some rows?** GitHub aliases `watchers_count` to stars in search payloads, so that value is published as `watchersCount` (not as a misleading watcher count). `watchers` and `subscribersCount` are the real watcher count, filled in direct mode and when `includeRepoDetails` is on.

**Do I need a proxy?** No. GitHub limits requests by IP and token, not by blocking this actor.

**Can I use my own search qualifiers?** Yes. The query field accepts the full GitHub repository-search grammar; the form filters are appended to it.

**What happens if a repository is missing?** The actor emits an error row for that repository and continues with the rest.

**Is this affiliated with GitHub?** No. It reads GitHub's public REST API and public repository pages.

### Data source

GitHub's public REST API (`api.github.com`), its raw content host (`raw.githubusercontent.com`), and public repository pages on `github.com`. No authentication is required for public data.

# Actor input Schema

## `mode` (type: `string`):

How to find repositories. 'search' runs a GitHub search query, 'direct' fetches an explicit list, 'owner' lists a user or organisation's repositories, and 'topic' lists repositories tagged with a topic.

## `searchQuery` (type: `string`):

GitHub search expression (mode=search). Supports qualifiers such as 'stars:>10000', 'language:python', 'topic:llm', 'user:apify', 'pushed:>2025-01-01', and 'in:name,description'. In owner/topic mode it is an optional keyword filter added to the owner or topic.

## `repositoryUrls` (type: `array`):

Repositories to fetch (mode=direct). Accepts https://github.com/owner/repo, a trailing slash, .git, the git@github.com:owner/repo.git SSH form, or a plain owner/repo slug. Duplicates are removed.

## `owner` (type: `string`):

GitHub user or organisation whose repositories should be listed (mode=owner). Accepts 'apify' or 'https://github.com/apify'.

## `topic` (type: `string`):

Topic tag to browse (mode=topic), e.g. 'llm', 'machine-learning', or 'web-scraping'. A leading 'topic:' is accepted and stripped.

## `sortBy` (type: `string`):

Sort key for search, owner, and topic results.

## `sortOrder` (type: `string`):

Ascending or descending order for the selected sort key.

## `language` (type: `string`):

Only repositories whose primary language matches. Pick a common language here, or type any other language in 'Custom language'.

## `languageCustom` (type: `string`):

Any GitHub language name not listed above (e.g. 'Hare', 'Gleam'). Overrides the Language dropdown when non-empty.

## `license` (type: `string`):

Only repositories released under this license.

## `searchIn` (type: `string`):

Where the free-text part of searchQuery must match (mode=search). Applies only when searchQuery contains free text.

## `minStars` (type: `integer`):

Only repositories with at least this many stars. 0 means no star filter.

## `minForks` (type: `integer`):

Only repositories with at least this many forks. 0 means no fork filter.

## `createdAfter` (type: `string`):

Only repositories created on or after this date (YYYY-MM-DD).

## `pushedAfter` (type: `string`):

Only repositories with a push on or after this date (YYYY-MM-DD). Useful for finding maintained projects.

## `includeForks` (type: `boolean`):

Include fork repositories in search, owner, and topic results. Direct mode always returns the repositories you list.

## `includeArchived` (type: `boolean`):

Keep archived repositories in the results. Turn this off to find actively maintained projects only.

## `maxResults` (type: `integer`):

Hard cap on repositories returned. GitHub's search API itself is capped at 1,000 results.

## `includeReadme` (type: `boolean`):

Fetch and include the decoded README text (capped at 500 KB).

## `includeTopics` (type: `boolean`):

Include the repository's topic tags.

## `includeLanguages` (type: `boolean`):

Include the byte count per programming language (one API request per repository).

## `includeRepoDetails` (type: `boolean`):

Fetch the repository detail endpoint in search, owner, and topic modes to add the real watcher count, network count, and organisation/parent fields (one extra API request per repository). Direct mode always fetches these.

## `includeReleases` (type: `boolean`):

Fetch the newest releases, including tag, dates, prerelease flag, and asset count (one request per repository).

## `releasesLimit` (type: `integer`):

How many releases to keep when 'Include releases' is on.

## `includeContributors` (type: `boolean`):

Fetch the ranked contributor list with contribution counts (one request per repository; GitHub caps this endpoint at 500 contributors).

## `contributorsLimit` (type: `integer`):

How many contributors to keep when 'Include contributors' is on.

## `includeActivity` (type: `boolean`):

Fetch 52 weeks of commit activity and the latest commit (up to two requests per repository; GitHub may need a moment to compute the statistics).

## `githubToken` (type: `string`):

Optional personal access token. It only raises the request limit from 60 to 5,000 requests per hour and is never required. A classic token with no scopes is enough for public data.

## Actor input object example

```json
{
  "mode": "search",
  "searchQuery": "language:rust stars:>5000 pushed:>2025-01-01",
  "repositoryUrls": [
    "https://github.com/apify/crawlee"
  ],
  "owner": "",
  "topic": "",
  "sortBy": "stars",
  "sortOrder": "desc",
  "language": "",
  "languageCustom": "",
  "license": "",
  "searchIn": "",
  "minStars": 0,
  "minForks": 0,
  "createdAfter": "2024-01-01",
  "pushedAfter": "2025-01-01",
  "includeForks": false,
  "includeArchived": true,
  "maxResults": 30,
  "includeReadme": true,
  "includeTopics": true,
  "includeLanguages": true,
  "includeRepoDetails": false,
  "includeReleases": false,
  "releasesLimit": 10,
  "includeContributors": false,
  "contributorsLimit": 30,
  "includeActivity": false
}
```

# Actor output Schema

## `repositories` (type: `string`):

Dataset containing all scraped GitHub repository records (plus per-repository error rows for invalid or missing repositories).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "search",
    "searchQuery": "stars:>10000 language:python",
    "repositoryUrls": [
        "https://github.com/apify/crawlee"
    ],
    "includeForks": false,
    "includeArchived": true,
    "maxResults": 30,
    "includeReadme": true,
    "includeTopics": true,
    "includeLanguages": true,
    "includeRepoDetails": false,
    "includeReleases": false,
    "includeContributors": false,
    "includeActivity": false
};

// Run the Actor and wait for it to finish
const run = await client.actor("nimbryxo-studio/github-repository-metadata-and-readme-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "search",
    "searchQuery": "stars:>10000 language:python",
    "repositoryUrls": ["https://github.com/apify/crawlee"],
    "includeForks": False,
    "includeArchived": True,
    "maxResults": 30,
    "includeReadme": True,
    "includeTopics": True,
    "includeLanguages": True,
    "includeRepoDetails": False,
    "includeReleases": False,
    "includeContributors": False,
    "includeActivity": False,
}

# Run the Actor and wait for it to finish
run = client.actor("nimbryxo-studio/github-repository-metadata-and-readme-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "search",
  "searchQuery": "stars:>10000 language:python",
  "repositoryUrls": [
    "https://github.com/apify/crawlee"
  ],
  "includeForks": false,
  "includeArchived": true,
  "maxResults": 30,
  "includeReadme": true,
  "includeTopics": true,
  "includeLanguages": true,
  "includeRepoDetails": false,
  "includeReleases": false,
  "includeContributors": false,
  "includeActivity": false
}' |
apify call nimbryxo-studio/github-repository-metadata-and-readme-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nimbryxo-studio/github-repository-metadata-and-readme-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/dd0ij1xXQE2gGm55r/builds/amhOs5VaUzW1sK9t5/openapi.json