---

GitHub Repository & Trending Scraper

---

A powerful GitHub scraper that extracts repository data from GitHub – either by search (keywords, language, stars range) or by fetching trending repositories (daily/weekly/monthly). Get clean, structured JSON output perfect for developer tools, analytics dashboards, research, or building your own GitHub insights platform.

👉 Focus keyword: github scraper – built for flexibility, speed, and reliability.

---

✨ Features

• 🔍 Search mode – Search repositories by keyword, language, stars, forks, etc. (supports GitHub search syntax).
• 📈 Trending mode – Fetch trending repositories for any language (daily, weekly, monthly).
• 📊 Rich repository data – Extracts name, description, URL, stars, forks, watchers, language, topics, license, dates, owner info, and even the README URL.
• 🌐 Proxy support – Built‑in Apify proxy with residential groups to avoid rate limiting.
• 📦 Scalable – Set max_results from 1 to 1000+ (respects GitHub pagination).
• 🧹 Clean output – Consistent schema with null values where data is missing.
• ⚡ Fast & efficient – Uses concurrency and request queuing to maximise throughput.
• 🔌 Easy integration – Run on Apify platform or use as a Node.js module.

---

🚀 How It Works

1. Choose mode – search (custom query) or trending (pre‑defined trending lists).
2. Provide parameters – For search: query, language, sort, order. For trending: language, period.
3. Scraping process – The actor sends requests to GitHub, parses repository data, and follows pagination.
4. Proxy rotation – For large jobs, residential proxies help avoid IP‑based blocks.
5. Output – Returns a clean JSON array of repository objects.

---

📥 Input Schema

The actor accepts the following input fields. Depending on mode, different fields are required.

Field	Type	Default	Description
mode	String	"search"	Either "search" or "trending".
query	String	""	(Search mode) GitHub search query (e.g., "machine learning"). Supports advanced syntax.
language	String	""	(Search or trending) Filter by programming language (e.g., "python", "javascript").
sort	String	"stars"	(Search mode) Sort by: "stars", "forks", "updated".
order	String	"desc"	(Search mode) "desc" or "asc".
period	String	"daily"	(Trending mode) "daily", "weekly", or "monthly".
max_results	Integer	25	Maximum number of repositories to return (1–1000).
proxyConfiguration	Object	{ "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }	Proxy settings. See Proxy Configuration.
include_readme	Boolean	false	If true, includes the raw README URL (always included, but this flag is for internal use).

Example Input (Search Mode)

{
  "mode": "search",
  "query": "machine learning",
  "language": "python",
  "sort": "stars",
  "order": "desc",
  "max_results": 20,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Example Input (Trending Mode)

{
  "mode": "trending",
  "language": "javascript",
  "period": "weekly",
  "max_results": 30
}

---

📤 Output Format

The actor returns an array of repository objects. Each object contains the following fields:

Field	Type	Description
type	String	Always "repository".
name	String	Full repository name (e.g., "owner/repo").
description	String or null	Repository description.
url	String	GitHub URL of the repository.
stars	Integer	Number of stars.
forks	Integer	Number of forks.
watchers	Integer	Number of watchers (usually same as stars).
language	String or null	Primary programming language.
topics	Array of strings	Repository topics/tags.
license	String or null	License name.
created_at	String	Creation date (YYYY-MM-DD).
updated_at	String	Last update date.
pushed_at	String	Last push date.
open_issues	Integer	Number of open issues.
size_kb	Integer	Repository size in KB.
default_branch	String	Default branch name.
is_fork	Boolean	Whether the repo is a fork.
homepage	String or null	Project homepage URL.
owner	String	Owner login.
owner_type	String	"User" or "Organization".
readme_url	String or null	Raw URL of the README file (if exists).

Example Output (Truncated)

[
  {
    "type": "repository",
    "name": "codecrafters-io/build-your-own-x",
    "description": "Master programming by recreating your favorite technologies from scratch.",
    "url": "https://github.com/codecrafters-io/build-your-own-x",
    "stars": 475006,
    "forks": 44543,
    "watchers": 475006,
    "language": "Markdown",
    "topics": ["awesome-list", "free", "programming", "tutorial-code"],
    "license": null,
    "created_at": "2018-05-09",
    "updated_at": "2026-03-14",
    "pushed_at": "2026-02-21",
    "open_issues": 442,
    "size_kb": 1201,
    "default_branch": "master",
    "is_fork": false,
    "homepage": "https://codecrafters.io",
    "owner": "codecrafters-io",
    "owner_type": "Organization",
    "readme_url": "https://raw.githubusercontent.com/codecrafters-io/build-your-own-x/master/README.md"
  },
  {
    "type": "repository",
    "name": "sindresorhus/awesome",
    "description": "😎 Awesome lists about all kinds of interesting topics",
    "url": "https://github.com/sindresorhus/awesome",
    "stars": 445369,
    "forks": 33527,
    "watchers": 445369,
    "language": null,
    "topics": ["awesome", "awesome-list", "lists", "resources"],
    "license": "Creative Commons Zero v1.0 Universal",
    "created_at": "2014-07-11",
    "updated_at": "2026-03-14",
    "pushed_at": "2026-03-09",
    "open_issues": 80,
    "size_kb": 1534,
    "default_branch": "main",
    "is_fork": false,
    "homepage": "",
    "owner": "sindresorhus",
    "owner_type": "User",
    "readme_url": "https://raw.githubusercontent.com/sindresorhus/awesome/main/README.md"
  }
  // ... up to max_results
]

---

🛠️ Usage

▶️ Run on Apify Console

1. Go to Apify Console and open the Actor page for GitHub Repository & Trending Scraper.
2. Click "Run".
3. Select mode and fill in the required fields.
4. Click "Start" and wait for results.

🔌 Run via Apify API (cURL)

curl -X POST "https://api.apify.com/v2/acts/your-username~github-trending-scraper/runs?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "search",
    "query": "machine learning",
    "language": "python",
    "max_results": 10
  }'

📦 Use as a Node.js Module

Install the package:

$npm install github-trending-scraper

Then use it in your code:

const { scrapeGitHub } = require('github-trending-scraper');

(async () => {
  const repos = await scrapeGitHub({
    mode: 'trending',
    language: 'javascript',
    period: 'weekly',
    max_results: 20
  });
  console.log(repos);
})();

---

🌐 Proxy Configuration

To avoid IP‑based blocking, especially for large scraping jobs, you can configure proxies. The actor integrates seamlessly with Apify Proxy.

Property	Type	Description
useApifyProxy	Boolean	If true, enables Apify Proxy. Default: true.
apifyProxyGroups	Array	Proxy groups: ["RESIDENTIAL"], ["DATACENTER"], or ["SHADER"]. Residential is recommended for GitHub.
proxyUrls	Array	Custom proxy URLs (e.g., ["http://user:pass@proxy.example.com:8080"]). Ignored if useApifyProxy is true.

Example with custom proxies:

{
  "proxyConfiguration": {
    "useApifyProxy": false,
    "proxyUrls": ["http://user:pass@123.45.67.89:8080"]
  }
}

---

❓ FAQ / Troubleshooting

Q: What's the difference between search and trending mode?

A: Search mode uses GitHub's search API to find repositories matching your query. Trending mode scrapes the GitHub trending page (like https://github.com/trending), which shows popular repositories for a given language and time period.

Q: Can I get more than 1000 results?

A: GitHub's search API limits results to 1000. If you need more, you can run multiple queries with different filters. Trending mode is limited by what GitHub shows on the trending pages (usually around 25 per page, up to a few hundred).

Q: Why are some fields null?

A: Some repositories may not have a description, license, topics, etc. The actor sets those fields to null.

Q: I'm getting blocked / rate limited.

A: Enable residential proxies (apifyProxyGroups: ["RESIDENTIAL"]). You can also reduce max_results or add delays (not yet configurable, but planned).

Q: Can I scrape issues or pull requests?

A: Not in this version. This actor focuses on repository metadata only.

---