GitHub Repositories Search Scraper

Search public GitHub repositories by keyword, language and topic, then export names, URLs, stars, forks, licenses and activity dates. Use GitHub Repositories Search Scraper for ecosystem research, developer relations and open-source discovery, with optional public owner profiles.

Quick start

1. Open the input form and enter at least one search query.
2. Set a small result limit, then start the run.
3. Open the dataset and export CSV, Excel or JSON.

{
  "queries": ["web scraping"],
  "language": "python",
  "sort": "stars",
  "order": "desc",
  "maxResults": 20,
  "includeOwnerDetails": false
}

Only public repository search is supported. GitHub rate limits and its 1,000-result search ceiling can prevent complete coverage. The result limit applies across all queries, not to each query separately.

Prefer a video guide? This short Apify walkthrough shows you how to set up and run an Actor. It uses a different Actor, but the setup steps are the same.

Ready-to-run examples

Choose a public example to see its settings before starting a run:

• Monitor Recent GitHub Repositories by Topic
• Find Popular Python GitHub Repositories
• Search GitHub Repositories by Keyword

What you get

One row per unique public repository found by your searches, with identity, source links and available counts, license metadata and activity dates. Optional owner enrichment adds exposed public profile fields when available. Overlapping queries retain the first query that found a repository, without global reranking.

Who is it for?

• Developer-relations teams researching public projects in a language or topic.
• Open-source researchers preparing a source-linked ecosystem shortlist.
• Developer-tool analysts inspecting available repository metadata.
• Data teams building bounded research datasets for their own reporting.

How you can use the data

• Prepare a Python web-scraping shortlist. Use the existing query and language sample with a small total cap. Keep repositoryId, fullName, htmlUrl, available license fields and pushedAt together, then inspect projects manually rather than choosing solely by star counts.
• Refine repository search with supported qualifiers. Keep GitHub qualifiers such as stars or license criteria inside the query. Search-term matches do not establish that a product appears in source code, and this is not a code-content export.
• Review optional public owner context. Enable owner details for a small sample when needed and inspect available profile fields. Public email can be missing. Missing owner profiles leave owner fields absent; unresolved request failures stop the run while preserving earlier saved rows.
• Compare your own dated observations. Join separate datasets by repository ID or full name and preserve query context with activity dates. Counts can change, watchers are not independently verified subscribers and repository popularity does not establish project suitability.

Input settings

Setting	JSON key	Purpose
Search queries	queries	Required nonempty list of GitHub repository search phrases or qualifiers.
Programming language	language	Optional language filter.
Topics	topics	Optional topic filters.
Sort by	sort	best-match, stars, forks or updated.
Sort order	order	desc or asc. Relevance ranking remains controlled by GitHub.
Maximum repositories	maxResults	Total saved-row cap across queries, up to 1,000.
Owner details	includeOwnerDetails	Add optional public owner profile fields.
Connection	connectionMode	automatic (default) starts directly and recovers anonymous rate limits through bounded datacenter connections; direct disables proxy fallback. Token-authenticated requests stay direct.
GitHub token	githubToken	Optional secret for higher GitHub API limits. Never put it in a shared example.

Example inputs

Use the Quick start input as a small first run. Adjust the fields described above for your own data.

Output data

Each unique repository produces one dataset row. Overlapping queries retain the first query that found a repository. Results are collected query by query, not globally re-ranked.

Illustrative output shape only. These values are not a captured repository:

{
  "query": "web scraping",
  "name": "example-repository",
  "fullName": "example-owner/example-repository",
  "repositoryId": 123,
  "htmlUrl": "https://github.com/example-owner/example-repository",
  "description": null,
  "ownerLogin": "example-owner",
  "ownerType": "Organization",
  "stars": 100,
  "forks": 10,
  "language": "Python",
  "topics": ["web-scraping"],
  "licenseName": null,
  "homepage": null,
  "archived": false,
  "fork": false
}

Fields	Meaning
query, name, fullName, repositoryId, htmlUrl	Original query and repository identity.
description, language, topics	Repository description, primary language and topic tags.
ownerLogin, ownerId, ownerType, ownerUrl, ownerAvatarUrl	Basic owner identity and public profile links.
stars, forks, watchers, openIssues	Counts reported by GitHub. watchers is GitHub's repository watcher-count field, not a verified subscriber count.
licenseKey, licenseName, licenseSpdxId	Available license metadata, otherwise null.
createdAt, updatedAt, pushedAt	Creation, metadata update and last-push timestamps.
defaultBranch, size, archived, fork, private	Repository attributes. Size is reported by GitHub in KB. Public search normally returns private: false.
homepage, cloneUrl	Available homepage and HTTPS clone URL.

With includeOwnerDetails: true, successful enrichment can add ownerName, ownerCompany, ownerBlog, ownerLocation, ownerEmail, ownerBio, ownerTwitterUsername, ownerPublicRepos, ownerFollowers, ownerFollowing, ownerCreatedAt and ownerUpdatedAt. These fields may be null or absent. Public email availability is not guaranteed. A missing public owner profile does not discard the repository. Rate limits and other unresolved owner-request failures stop the run rather than claiming complete enrichment.

Download your results

Open your run's dataset, choose Export and select JSON, CSV or Excel. JSON keeps nested fields together. CSV and Excel are useful when you want to review the records in a spreadsheet.

See Apify's download and export guide for the available formats and field options.

Pricing

Pay-per-event pricing uses a start event once after input validation and a result event for each repository result before it is saved. A no-match search can still incur the start charge. No separate paid error or status rows are produced.

See the live Pricing tab for current rates and discounts.

Tips for best results

• Start with a narrow query and small total cap. The cap applies across queries rather than separately to each one.
• Keep qualifiers and separate filters clear. Inspect the effective language, topic and query scope before comparing exports.
• Enable owner enrichment only when useful. It adds requests and has a lower permitted result limit without a token.
• Review the run outcome before retrying. Invalid queries produce visible warnings and skipped-query reasons. Unresolved rate limits fail the run, preserving earlier saved rows and pending searches.
• Read progress records without assuming recovery. Pending searches are checkpoints rather than automatic continuation or a completeness certificate.

Limits and caveats

Only public repository search is supported. GitHub's 1,000-result search ceiling and API rate limits can prevent complete coverage. Filtering or duplicate matches can leave fewer saved rows than the requested cap. Queries run sequentially, with no global reranking.

Small searches can run without a token. Automatic connections start directly. Anonymous rate limits trigger existing datacenter proxy recovery with a visible notice and summary counters, at most 200 proxy requests per run, and at most three attempts per request. This infrastructure cost is borne by the creator, with no extra Actor fee. Choose connectionMode: "direct" to disable proxy fallback. Requests with a GitHub token stay direct because their quota is account-bound. Direct-only rate limits wait for GitHub's reset or retry guidance when it fits the remaining run time. Unresolved rate limits, transport errors and incomplete responses fail the run instead of being reported as successful empty searches. Earlier saved rows remain accessible. Invalid or overly broad queries are skipped with a visible warning and RUN_SUMMARY reason; other queries continue.

Owner enrichment requires additional requests. Without a token, enabling it rejects result limits above 50. Profile fields and public email can be null or absent. Missing owner profiles are allowed; unresolved owner request failures stop the run and preserve earlier rows.

An independent processing cap of 270 seconds (or less when the run timeout requires it) bounds work and retry waits. A deadline reached before required searches finish fails the run. PENDING_SEARCHES, when present, is a checkpoint rather than automatic continuation or a coverage guarantee; resume may repeat earlier results. RUN_SUMMARY reports saved rows, invalid queries and fatal errors. A requested result cap can end collection before every query is visited. Source counts and activity dates change, and unavailable metadata remains null or absent rather than inferred.

Repository watcher counts are source fields rather than independently verified subscriber counts. Metadata does not export source archives or README content, and search matches do not guarantee references inside code.

Automation and integrations

Save your settings for next time

Once you have an input that works for you, save it as an Apify task. This keeps your settings together so you can use them again without filling in the form from scratch.

Run on a schedule

Need fresh results regularly? You can schedule the Actor or a saved task to run at a time that suits you.

Each scheduled run collects the data again. If you want to compare results or highlight changes, add that step in your own workflow.

Send results to your other tools

Use n8n, Make or Zapier to connect this scraper to the rest of your workflow. For example, you could pass its results to an AI assistant or add records to a spreadsheet. Follow the Apify integration guides for your chosen tool.

For a custom connection, an Apify webhook can notify your app when a run finishes or fails. Your app can then fetch the results or handle the failure.

API usage

Want to collect this data from your own app or automation? The examples below send input to the scraper and show how to collect its results.

The Node.js and Python examples wait for the run to finish before reading its results. The cURL example starts the run and returns its details. Wait for that run to finish, then use its defaultDatasetId to fetch the results.

For setup instructions, see Apify's JavaScript client guide, Python client guide and Actor runs API reference.

cURL

curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~github-repositories-search-scraper/runs' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  --data @- <<'JSON'
{
  "queries": [
    "web scraping"
  ],
  "language": "python",
  "sort": "stars",
  "order": "desc",
  "maxResults": 20,
  "includeOwnerDetails": false
}
JSON

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/github-repositories-search-scraper').call({
  "queries": [
    "web scraping"
  ],
  "language": "python",
  "sort": "stars",
  "order": "desc",
  "maxResults": 20,
  "includeOwnerDetails": false
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('fetch_cat/github-repositories-search-scraper').call(run_input={'queries': ['web scraping'], 'language': 'python', 'sort': 'stars', 'order': 'desc', 'maxResults': 20, 'includeOwnerDetails': False})
print(client.dataset(run['defaultDatasetId']).list_items().items)

API usage notes

The REST call starts a run asynchronously. Wait for completion, then use its defaultDatasetId to retrieve dataset items.

MCP and AI agents

You can ask your AI assistant to collect data with GitHub Repositories Search Scraper and help you work with the results. The scraper returns the data, and your assistant can then summarize it, compare records or prepare notes.

Connect your assistant through the Apify MCP server. MCP gives your assistant a way to find this scraper, run it and read the results. Follow the official setup and sign-in guide for your app.

Easier setup with an Apify plugin

If you use a supported coding assistant, the official Apify plugin can set up the connection for you. It connects your assistant to Apify and includes instructions for common tasks. Follow the guide for your particular app.

Use Apify AI in Console

Prefer to keep everything in Apify? Open Apify AI in Console and ask it to find Fetch Cat's GitHub Repositories Search Scraper. You can describe the data you want and ask questions about the results.

Apify AI is currently in beta. It asks for confirmation before starting an Actor, and normal run charges apply.

Choose your assistant

• Claude and Claude Code (Anthropic): Apify setup guide.
• ChatGPT and Codex (OpenAI): Apify setup guides.
• Grok Bot (xAI): Apify plugin guide.
• Gemini CLI (Google): MCP server setup guide. Use the server URL below with the HTTP configuration supported by Gemini CLI.

Connect just this scraper

To keep your setup focused on this scraper, add this server configuration to a compatible MCP client:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=fetch_cat/github-repositories-search-scraper"
    }
  }
}

Sign in to Apify when your client asks. If your client uses a Bearer token instead, follow its setup guide and keep the token in its secure configuration. Configuration formats vary by app, so use the linked guide if your client needs a different format.

Use Apify's general server

Already connected to the general Apify MCP server at https://mcp.apify.com? You can use it to find this scraper without adding another server. Ask your assistant to use GitHub Repositories Search Scraper by Fetch Cat, or give it the identifier fetch_cat/github-repositories-search-scraper.

Your assistant can check the available settings and retrieve the results from the returned run. If the run is still working, it should follow that run rather than start it again. Both connection options use Apify's hosted service.

Example requests

• “Use Fetch Cat’s GitHub Repositories Search Scraper to collect up to 5 Python repositories matching web scraping, ordered by stars descending. Show names, source links, available licenses and last-push dates for manual project review.”
• “Collect at most 5 public repositories for the query and topic I provide. Keep repository IDs and the first matching query without treating the combined results as globally ranked or guaranteed source-code matches.”
• “Retrieve up to 3 public repository matches for my supplied query with owner details enabled. Show available public owner names and profile links, marking missing fields rather than assuming contact information exists.”

Normal Apify charges apply. Check the live Pricing tab before asking your assistant to run a larger batch.

Pay with x402

If your AI agent has a funded wallet, it can pay for Apify usage through x402. This experimental option lets an agent use Apify without a conventional account.

Apify AGI provides a prepaid API token with a spending cap after payment. Your agent can use that token to run this scraper through the Apify API or an authenticated MCP connection.

Some MCP setups also support paying for each request directly. That is a different payment option, so check the official guide for its availability.

Follow the official guide for wallet setup, supported payment options and current terms. Apify handles the payments. Keep wallet keys and payment tokens in your agent's secure configuration.

Other ways for agents to pay

Apify also supports MPP for prepaid access through AGI and Skyfire for eligible Actors. If your agent already uses one of these, see the agentic payment guide to choose the right setup.

FAQ

How do I search GitHub repositories with qualifiers?

Put supported GitHub repository search syntax inside queries, for example language:python stars:>100. The separate language and topics fields further narrow that query.

Can I export GitHub repository results as CSV or JSON?

Yes. Export the dataset in Apify or retrieve its items through the Dataset API. This Actor returns repository metadata, not source-code archives or full README contents.

Do I need a GitHub token?

Not for a small public search. A token can help with higher API limits and optional owner enrichment. It does not expand this product to private-repository extraction.

Why are results missing or empty?

Filters, overlapping queries, GitHub search limits, rejected queries, rate limits or a time limit can reduce results. Inspect the run and summary before concluding there are no matches. Try a narrower valid query or a smaller first run.

Related Actors

• Stack Exchange Questions Scraper — Explore technical questions around a project ecosystem.
• Reddit Scraper — Research public community discussions.
• Web Page to Markdown Extractor — Convert linked project websites to readable text.
• Website Contact Finder — Inspect contacts published on project websites.
• Google News Scraper — Follow public coverage of developer tools.

Support and feedback

For custom additions or integrations, email hello@fetchcat.dev.

Found an error or have a feature request? Open an issue with the run URL, a non-sensitive input example and what you expected.

Need help sharing it? See Apify’s guide to sharing a run when reporting an issue. Sharing a run can give the developer access to its input, logs and results, so use a run that contains no sensitive data. Please leave out API tokens, passwords, cookies, private documents and personal data. Use a public sample URL or a redacted example instead.

Tried this scraper? We'd love to hear how it worked for you. Leave a review to help others choose the right tool and help us improve it. Your feedback means a lot to us.

Made with care for curious people, AI agents and the possibilities in data. ♥