GitHub Repository Scraper

An Apify Actor that extracts comprehensive information from GitHub repository pages. Built with Python, httpx, and BeautifulSoup.

Features

Scrapes the following data points for one or multiple GitHub repositories:

• Basic Info: Repository URL, owner, repository name, description, and website.
• Stats: Star count, fork count.
• Activity: Latest release tag and date, commit count, and default branch.
• Metadata: Topics, primary programming languages (with percentages), and license.
• Content: The HTML content of the README (truncated to first 5000 characters).

Input Configuration

The Actor accepts input via a JSON schema.

Field	Type	Description
repo_urls	Array	A list of GitHub repository URLs to scrape (e.g., ["https://github.com/apify/apify-sdk-python"] or [{"url": "..."}]).

Example Input:

{
  "repo_urls": [
    "https://github.com/apify/apify-sdk-python",
    "https://github.com/microsoft/vscode"
  ]
}

Output Data Structure

The Actor stores results in the default dataset.

Example Output:

{
  "url": "https://github.com/apify/apify-sdk-python",
  "owner": "apify",
  "repo_name": "apify-sdk-python",
  "description": "Apify SDK for Python—The official library for building Apify Actors...",
  "website": "https://docs.apify.com/sdk/python",
  "stars": 173,
  "forks": 25,
  "default_branch": "master",
  "commit_count": 1140,
  "topics": ["python", "automation", "sdk", "proxy", "scraping"],
  "languages": [
    { "name": "Python", "percentage": "72.7%" },
    { "name": "MDX", "percentage": "24.3%" },
    { "name": "JavaScript", "percentage": "2.7%" }
  ],
  "license": "Apache-2.0 license",
  "latest_release": { "tag": "3.4.1", "date": "2026-05-29T10:05:29Z" },
  "readme_html": "<article class=\"markdown-body entry-content container-lg\" itemprop=\"text\">...",
  "scraped_at": "2026-07-12T05:00:00+00:00"
}

Local Development

1. Ensure you have Python 3.11+ installed.
2. Install dependencies: pip install -r requirements.txt
3. Prepare input in storage/key_value_stores/default/INPUT.json.
4. Run the Actor: python -m src or apify run.