# GitHub Repository Scraper (`vulnv/github-repository-scraper`) Actor

Scrape and extract GitHub repository data, metadata, statistics, stars, forks, issues, and project information from multiple repositories at once.

- **URL**: https://apify.com/vulnv/github-repository-scraper.md
- **Developed by:** [VulnV](https://apify.com/vulnv) (community)
- **Categories:** Developer tools, Automation, Other
- **Stats:** 18 total users, 0 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## GitHub Repository Scraper - Extract Repository Data at Scale

### Overview

The **GitHub Repository Scraper** is a powerful Apify Actor designed to extract comprehensive data from GitHub repositories efficiently. Perfect for competitive analysis, market research, developer insights, or building repository databases — this scraper provides detailed information about repositories, statistics, and project metadata.

✅ Bulk URL processing | ✅ Comprehensive repository data | ✅ Statistics extraction | ✅ Metadata analysis | ✅ Concurrent processing

***

#### **Complete Repository Data Extraction**

- **Basic Information** — Repository name, description, owner, creation date
- **Statistics** — Stars, forks, watchers, usage metrics
- **Technical Details** — Programming languages, file counts, commit information
- **Project Metadata** — Topics, license information, default branch
- **Enhanced Repository Data** — GitHub IDs, clone URLs, file listings, branch info
- **Owner Information** — Detailed owner profiles with avatars and organization status
- **Repository Structure** — File counts, directory listings, README information
- **Access URLs** — Multiple clone formats (HTTPS, SSH, GitHub CLI), download links

#### **Key Features**

- **Bulk Processing** — Process multiple GitHub repository URLs in one run
- **Smart URL Parsing** — Automatically extracts repository paths from full GitHub URLs
- **Proxy Support** — Built-in Apify proxy integration for reliable scraping
- **Error Handling** — Robust error handling with detailed status reporting
- **Clean JSON Output** — Structured, ready-to-use data format
- **Concurrent Processing** — Configurable concurrency for optimal performance
- **Format Flexibility** — Accepts various URL formats and automatically normalizes them

***

### 🧾 Input Configuration

Submit an array of GitHub repository URLs via the input schema:

```json
{
  "urls": [
    "https://github.com/microsoft/vscode",
    "https://github.com/facebook/react",
    "https://github.com/nodejs/node",
    "https://github.com/torvalds/linux"
  ],
  "maxConcurrency": 5,
  "includeNotFound": false,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}
```

#### **Input Parameters**

1. **URLs** (required):
   - Array of GitHub repository URLs to scrape
   - **Supported formats**: `https://github.com/owner/repo`, `github.com/owner/repo`
   - Invalid URLs will be automatically filtered out with warnings

2. **Max Concurrency** (optional):
   - Number of concurrent requests for scraping (1-20)
   - Default: 5
   - Higher values = faster processing but may increase chance of rate limiting

3. **Include Not Found** (optional):
   - Whether to include repositories that return 404 (not found) in the results
   - Default: false
   - When enabled, includes error information for non-existent repositories

4. **Proxy Configuration** (recommended):
   - Configure Apify proxy settings to avoid rate limiting
   - **Recommended for bulk scraping operations**
   - **Format**:
     ```json
     "proxyConfiguration": {
       "useApifyProxy": true,
       "apifyProxyGroups": ["RESIDENTIAL"]
     }
     ```
   - **Available proxy groups**: `RESIDENTIAL`, `DATACENTER`, `GOOGLE_SERP`
   - Use `RESIDENTIAL` for best reliability when scraping GitHub

#### **Proxy Configuration Examples**

**For small-scale scraping (< 100 repositories):**

```json
"proxyConfiguration": {
  "useApifyProxy": true,
  "apifyProxyGroups": ["DATACENTER"]
}
```

**For large-scale or production scraping (recommended):**

```json
"proxyConfiguration": {
  "useApifyProxy": true,
  "apifyProxyGroups": ["RESIDENTIAL"]
}
```

**No proxy (not recommended for bulk operations):**

```json
// Omit proxyConfiguration entirely - may result in rate limiting
```

***

### 📤 Output Format

Each GitHub repository returns comprehensive structured data including enhanced metadata extracted from GitHub's embedded data:

```json
{
  "url": "https://github.com/microsoft/vscode",
  "repoPath": "microsoft/vscode",
  "success": true,
  "data": {
    "url": "https://github.com/microsoft/vscode",
    "type": "repo",
    "description": "Visual Studio Code",
    "website": "https://code.visualstudio.com",
    "forkedfrom": null,
    "tags": ["editor", "typescript", "electron", "ide"],
    "usedby": 250000,
    "watchers": 3200,
    "stars": 162000,
    "forks": 28500,
    "langs": [
      {"name": "TypeScript", "perc": "93.2%"},
      {"name": "JavaScript", "perc": "4.1%"},
      {"name": "CSS", "perc": "1.5%"}
    ],

    // Enhanced data from GitHub's embedded JSON
    "id": 41881900,
    "name": "vscode",
    "full_name": "microsoft/vscode",
    "owner": "microsoft",
    "default_branch": "main",
    "is_fork": false,
    "is_empty": false,
    "is_private": false,
    "is_org_owned": true,
    "created_at": "2015-09-03T20:23:30.000Z",
    "clone_url": "https://github.com/microsoft/vscode.git",
    "ssh_url": "git@github.com:microsoft/vscode.git",
    "api_url": "https://api.github.com/repos/microsoft/vscode",

    // Owner information
    "owner_info": {
      "login": "microsoft",
      "type": "Organization",
      "url": "https://github.com/microsoft",
      "avatar_url": "https://avatars.githubusercontent.com/u/6154722?v=4"
    },

    // File and repository structure
    "file_count": 15420,
    "files": [
      {"name": "README.md", "path": "README.md", "type": "file"},
      {"name": "package.json", "path": "package.json", "type": "file"},
      {"name": "src", "path": "src", "type": "directory"}
    ],

    // Clone and download URLs
    "clone_urls": {
      "https": "https://github.com/microsoft/vscode.git",
      "ssh": "git@github.com:microsoft/vscode.git",
      "github_cli": "gh repo clone microsoft/vscode"
    },
    "download_url": "/microsoft/vscode/archive/refs/heads/main.zip",

    // Branch and commit information
    "ref_info": {
      "name": "main",
      "type": "branch",
      "current_oid": "585acf48f88e399989d54f001029424b2b7c358a",
      "can_edit": false
    },
    "commit_count": "185,234",

    // README information
    "readme_info": {
      "displayName": "README.md",
      "repoName": "vscode",
      "refName": "main",
      "path": "README.md",
      "loaded": true
    },

    // Metadata
    "enriched_at": "2024-12-29T15:30:45.123Z",
    "data_source": "github_scraper_enhanced"
  }
}
```

#### **Error Handling**

Failed repositories return structured error information:

```json
{
  "url": "https://github.com/invalid/repo",
  "repoPath": "invalid/repo",
  "success": false,
  "error": "Repository not found or private"
}
```

When `includeNotFound` is enabled, 404 repositories return structured data:

```json
{
  "url": "https://github.com/nonexistent/repo",
  "repoPath": "nonexistent/repo",
  "success": true,
  "data": {
    "exists": false,
    "error": "Repository not found",
    "statusCode": 404
  }
}
```

**Common Error Cases:**

- `Repository not found or private` — Repository doesn't exist or is private
- `Network error` — Connection issues or scraping errors
- Invalid URLs are filtered out before processing with warning logs

***

### 💼 Common Use Cases

#### **Competitive Analysis & Market Research**

- Analyze competitor repositories and project activity
- Track technology trends through repository statistics
- Research popular libraries and frameworks in specific domains
- Monitor open source project adoption rates

#### **Developer & Technology Research**

- Study programming language usage patterns
- Analyze repository structures and best practices
- Research active open source projects in specific technologies
- Track development activity and contribution patterns

#### **Portfolio & Investment Analysis**

- Research technology companies and their open source contributions
- Analyze developer productivity and project health metrics
- Track repository growth and community engagement
- Identify trending projects and technologies

#### **Academic & Educational Research**

- Study software development patterns and practices
- Analyze open source community dynamics
- Research programming language evolution
- Track educational resource repositories

***

### 📊 Output & Export Options

#### **Dataset Storage**

- All extracted data stored in Apify dataset
- Each repository becomes one dataset item
- Status tracking for successful and failed extractions

#### **Export Formats**

- **JSON** — Raw structured data for API integration
- **CSV** — Spreadsheet-compatible format for analysis
- **Excel** — Formatted spreadsheet with repository data

#### **Data Processing**

- Clean, validated URLs
- Structured error reporting
- Comprehensive logging for troubleshooting

***

### ⚡ Quick Start Guide

1. **Configure Input**:
   - Add GitHub repository URLs to the `urls` array
   - Set desired `maxConcurrency` (recommended: 5-10)
   - Configure `proxyConfiguration` with `useApifyProxy: true` and appropriate proxy groups for reliable scraping

2. **Run the Actor**:
   - Execute through Apify Console or API
   - Monitor progress through real-time logs
   - Review extracted data in the dataset

3. **Export Results**:
   - Download data in your preferred format
   - Integrate with your existing tools and workflows

***

### 🆘 Support & Feedback

For questions, feature requests, or technical support:

- Visit the [Apify Community Forum](https://forum.apify.com)
- Contact us through the Apify platform
- Submit issues for improvements and bug reports

***

### 🌟 Explore More Actors

✨ **Need more scraping solutions?** Discover additional actors on Apify for comprehensive web automation and data extraction. Explore our full range of tools at 🌐 [Explore More Actors on Apify](https://apify.com/vulnv).

📧 For inquiries or custom development, reach out at apify@vulnv.com.

# Actor input Schema

## `urls` (type: `array`):

List of GitHub repository URLs to scrape. Each URL should be a full GitHub repository URL (e.g., https://github.com/user/repo).

## `proxyConfiguration` (type: `object`):

Select proxies to be used by your scraper.

## `maxConcurrency` (type: `integer`):

Maximum number of concurrent requests to process at once

## `includeNotFound` (type: `boolean`):

Include repositories that return 404 (not found) in the results with error information

## Actor input object example

```json
{
  "urls": [
    "https://github.com/nelsonic/adoro",
    "https://github.com/microsoft/vscode",
    "https://github.com/facebook/react"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxConcurrency": 5,
  "includeNotFound": false
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://github.com/nelsonic/adoro",
        "https://github.com/microsoft/vscode",
        "https://github.com/facebook/react"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("vulnv/github-repository-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "https://github.com/nelsonic/adoro",
        "https://github.com/microsoft/vscode",
        "https://github.com/facebook/react",
    ],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("vulnv/github-repository-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://github.com/nelsonic/adoro",
    "https://github.com/microsoft/vscode",
    "https://github.com/facebook/react"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call vulnv/github-repository-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,vulnv/github-repository-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/GARvnXc0S7gL2OGik/builds/1GmQzZN4YM47Dx4M2/openapi.json