GitHub Repository Scraper - Extract Repository Data at Scale

Overview

The GitHub Repository Scraper is a powerful Apify Actor designed to extract comprehensive data from GitHub repositories efficiently. Perfect for competitive analysis, market research, developer insights, or building repository databases — this scraper provides detailed information about repositories, statistics, and project metadata.

✅ Bulk URL processing | ✅ Comprehensive repository data | ✅ Statistics extraction | ✅ Metadata analysis | ✅ Concurrent processing

---

Complete Repository Data Extraction

• Basic Information — Repository name, description, owner, creation date
• Statistics — Stars, forks, watchers, usage metrics
• Technical Details — Programming languages, file counts, commit information
• Project Metadata — Topics, license information, default branch
• Enhanced Repository Data — GitHub IDs, clone URLs, file listings, branch info
• Owner Information — Detailed owner profiles with avatars and organization status
• Repository Structure — File counts, directory listings, README information
• Access URLs — Multiple clone formats (HTTPS, SSH, GitHub CLI), download links

Key Features

• Bulk Processing — Process multiple GitHub repository URLs in one run
• Smart URL Parsing — Automatically extracts repository paths from full GitHub URLs
• Proxy Support — Built-in Apify proxy integration for reliable scraping
• Error Handling — Robust error handling with detailed status reporting
• Clean JSON Output — Structured, ready-to-use data format
• Concurrent Processing — Configurable concurrency for optimal performance
• Format Flexibility — Accepts various URL formats and automatically normalizes them

---

🧾 Input Configuration

Submit an array of GitHub repository URLs via the input schema:

{
  "urls": [
    "https://github.com/microsoft/vscode",
    "https://github.com/facebook/react",
    "https://github.com/nodejs/node",
    "https://github.com/torvalds/linux"
  ],
  "maxConcurrency": 5,
  "includeNotFound": false,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Input Parameters

1. URLs (required):

   • Array of GitHub repository URLs to scrape
   • Supported formats: https://github.com/owner/repo, github.com/owner/repo
   • Invalid URLs will be automatically filtered out with warnings

2. Max Concurrency (optional):

   • Number of concurrent requests for scraping (1-20)
   • Default: 5
   • Higher values = faster processing but may increase chance of rate limiting

3. Include Not Found (optional):

   • Whether to include repositories that return 404 (not found) in the results
   • Default: false
   • When enabled, includes error information for non-existent repositories

4. Proxy Configuration (recommended):

   • Configure Apify proxy settings to avoid rate limiting
   • Recommended for bulk scraping operations
   • Format:

     "proxyConfiguration": {
       "useApifyProxy": true,
       "apifyProxyGroups": ["RESIDENTIAL"]
     }
     

   • Available proxy groups: RESIDENTIAL, DATACENTER, GOOGLE_SERP
   • Use RESIDENTIAL for best reliability when scraping GitHub

Proxy Configuration Examples

For small-scale scraping (< 100 repositories):

"proxyConfiguration": {
  "useApifyProxy": true,
  "apifyProxyGroups": ["DATACENTER"]
}

For large-scale or production scraping (recommended):

"proxyConfiguration": {
  "useApifyProxy": true,
  "apifyProxyGroups": ["RESIDENTIAL"]
}

No proxy (not recommended for bulk operations):

// Omit proxyConfiguration entirely - may result in rate limiting

---

📤 Output Format

Each GitHub repository returns comprehensive structured data including enhanced metadata extracted from GitHub's embedded data:

{
  "url": "https://github.com/microsoft/vscode",
  "repoPath": "microsoft/vscode",
  "success": true,
  "data": {
    "url": "https://github.com/microsoft/vscode",
    "type": "repo",
    "description": "Visual Studio Code",
    "website": "https://code.visualstudio.com",
    "forkedfrom": null,
    "tags": ["editor", "typescript", "electron", "ide"],
    "usedby": 250000,
    "watchers": 3200,
    "stars": 162000,
    "forks": 28500,
    "langs": [
      {"name": "TypeScript", "perc": "93.2%"},
      {"name": "JavaScript", "perc": "4.1%"},
      {"name": "CSS", "perc": "1.5%"}
    ],

    // Enhanced data from GitHub's embedded JSON
    "id": 41881900,
    "name": "vscode",
    "full_name": "microsoft/vscode",
    "owner": "microsoft",
    "default_branch": "main",
    "is_fork": false,
    "is_empty": false,
    "is_private": false,
    "is_org_owned": true,
    "created_at": "2015-09-03T20:23:30.000Z",
    "clone_url": "https://github.com/microsoft/vscode.git",
    "ssh_url": "git@github.com:microsoft/vscode.git",
    "api_url": "https://api.github.com/repos/microsoft/vscode",

    // Owner information
    "owner_info": {
      "login": "microsoft",
      "type": "Organization",
      "url": "https://github.com/microsoft",
      "avatar_url": "https://avatars.githubusercontent.com/u/6154722?v=4"
    },

    // File and repository structure
    "file_count": 15420,
    "files": [
      {"name": "README.md", "path": "README.md", "type": "file"},
      {"name": "package.json", "path": "package.json", "type": "file"},
      {"name": "src", "path": "src", "type": "directory"}
    ],

    // Clone and download URLs
    "clone_urls": {
      "https": "https://github.com/microsoft/vscode.git",
      "ssh": "git@github.com:microsoft/vscode.git",
      "github_cli": "gh repo clone microsoft/vscode"
    },
    "download_url": "/microsoft/vscode/archive/refs/heads/main.zip",

    // Branch and commit information
    "ref_info": {
      "name": "main",
      "type": "branch",
      "current_oid": "585acf48f88e399989d54f001029424b2b7c358a",
      "can_edit": false
    },
    "commit_count": "185,234",

    // README information
    "readme_info": {
      "displayName": "README.md",
      "repoName": "vscode",
      "refName": "main",
      "path": "README.md",
      "loaded": true
    },

    // Metadata
    "enriched_at": "2024-12-29T15:30:45.123Z",
    "data_source": "github_scraper_enhanced"
  }
}

Error Handling

Failed repositories return structured error information:

{
  "url": "https://github.com/invalid/repo",
  "repoPath": "invalid/repo",
  "success": false,
  "error": "Repository not found or private"
}

When includeNotFound is enabled, 404 repositories return structured data:

{
  "url": "https://github.com/nonexistent/repo",
  "repoPath": "nonexistent/repo",
  "success": true,
  "data": {
    "exists": false,
    "error": "Repository not found",
    "statusCode": 404
  }
}

Common Error Cases:

• Repository not found or private — Repository doesn't exist or is private
• Network error — Connection issues or scraping errors
• Invalid URLs are filtered out before processing with warning logs

---

💼 Common Use Cases

Competitive Analysis & Market Research

• Analyze competitor repositories and project activity
• Track technology trends through repository statistics
• Research popular libraries and frameworks in specific domains
• Monitor open source project adoption rates

Developer & Technology Research

• Study programming language usage patterns
• Analyze repository structures and best practices
• Research active open source projects in specific technologies
• Track development activity and contribution patterns

Portfolio & Investment Analysis

• Research technology companies and their open source contributions
• Analyze developer productivity and project health metrics
• Track repository growth and community engagement
• Identify trending projects and technologies

Academic & Educational Research

• Study software development patterns and practices
• Analyze open source community dynamics
• Research programming language evolution
• Track educational resource repositories

---

📊 Output & Export Options

Dataset Storage

• All extracted data stored in Apify dataset
• Each repository becomes one dataset item
• Status tracking for successful and failed extractions

Export Formats

• JSON — Raw structured data for API integration
• CSV — Spreadsheet-compatible format for analysis
• Excel — Formatted spreadsheet with repository data

Data Processing

• Clean, validated URLs
• Structured error reporting
• Comprehensive logging for troubleshooting

---

⚡ Quick Start Guide

1. Configure Input:

   • Add GitHub repository URLs to the urls array
   • Set desired maxConcurrency (recommended: 5-10)
   • Configure proxyConfiguration with useApifyProxy: true and appropriate proxy groups for reliable scraping

2. Run the Actor:

   • Execute through Apify Console or API
   • Monitor progress through real-time logs
   • Review extracted data in the dataset

3. Export Results:

   • Download data in your preferred format
   • Integrate with your existing tools and workflows

---

🆘 Support & Feedback

For questions, feature requests, or technical support:

• Visit the Apify Community Forum
• Contact us through the Apify platform
• Submit issues for improvements and bug reports

---

🌟 Explore More Actors

✨ Need more scraping solutions? Discover additional actors on Apify for comprehensive web automation and data extraction. Explore our full range of tools at 🌐 Explore More Actors on Apify.

📧 For inquiries or custom development, reach out at apify@vulnv.com.