GitHub, npm, PyPI, Crates & Docker Package Intelligence

Scrape and enrich open-source packages and repos across GitHub, npm, PyPI, Rust Crates.io, and Docker Hub with stars, downloads, licenses, and maintainers.

What it does

Searches and enriches open-source repositories and software packages across GitHub, npm, PyPI, Rust Crates.io, and Docker Hub with GitHub stars/forks, 30-day/90-day download velocity, Docker pull counts, SPDX license compliance, maintainer contacts, repository URLs, and release timestamps in one unified schema. Export clean, deduplicated data as JSON, CSV, Excel, XML, or RSS, or connect directly via the Apify REST API, Python/JavaScript SDKs, CLI, MCP Server, Clay, n8n, Make, or Zapier — with no login, cookies, or browser overhead.

Use cases

• DevTool GTM & Developer Relations Lead Generation: Identify fast-growing open-source packages, maintainers (authorOrPublisher, maintainerEmail), and GitHub organizations across AI (mcp, langchain), observability (opentelemetry), and data tooling.
• VC & Private Equity Open-Source Traction Due Diligence: Benchmark developer adoption by combining GitHub starsCount/forksCount with real npm 30-day downloads, Crates.io 90-day downloads, and Docker Hub pull_count (downloadsRecent, downloadsTotal).
• Enterprise AppSec & SBOM License Compliance Auditing: Verify SPDX license declarations, latestVersion, repositoryUrl, and release recency (publishedOrPushedAt) across Python, TypeScript/JavaScript, Rust, and container dependencies.
• AI Coding Agent & RAG Tool Discovery: Track newly published MCP servers, LLM SDKs, and developer packages with onlyNew: true.

Why use this scraper?

Feature	This Actor	Generic Browser / HTML Scrapers
Architecture	Direct structured API / JSON / SSR requests (curl_cffi)	Heavy headless browser automation
Speed & reliability	2–25 seconds for 50–200 items	Minutes of high compute burn
Pricing model	Pay per result ($0.00 actor-start fee)	Monthly rental or high CU overhead
Error handling	Failed inputs logged in ERRORS key-value record and never charged	Bills compute time even on failures

Input parameters

Parameter	Type	Default	Description
queries	array	["mcp", "langchain", "opentelemetry", "pydant...	Keywords or package/repository names to search across GitHub, npm, Rust Crates.io, and Docker Hub.
ecosystems	array	["github", "npm", "pypi", "crates", "docker"]	Developer registries to query and normalize ('github', 'npm', 'pypi', 'crates', 'docker').
pypiPackages	array	["fastapi", "pydantic", "openai", "anthropic"...	Exact Python package names on PyPI to enrich via pypi.org/pypi/{pkg}/json.
enrichNpmDownloads	boolean	true	Attach 30-day monthly download velocity for npm packages.
onlyNew	boolean	false	Deduplicate across runs via Apify Key-Value Store and emit only newly seen packages or releases.
maxItemsPerEcosystem	integer	16	Maximum number of normalized package/repository records to emit per enabled ecosystem (default 5 ecosystems x 16 = 80 records).
delaySeconds	number	0.15	Polite pause between upstream HTTP requests (seconds).

Parameter details

• Search keywords or package names (queries, array): Keywords or package/repository names to search across GitHub, npm, Rust Crates.io, and Docker Hub.
• Package registries & developer ecosystems (ecosystems, array): Developer registries to query and normalize ('github', 'npm', 'pypi', 'crates', 'docker').
• PyPI package names to enrich (pypiPackages, array): Exact Python package names on PyPI to enrich via pypi.org/pypi/{pkg}/json.
• Include 30-day npm download counts (enrichNpmDownloads, boolean): Attach 30-day monthly download velocity for npm packages.
• Only new packages / versions (onlyNew, boolean): Deduplicate across runs via Apify Key-Value Store and emit only newly seen packages or releases.
• Max items per ecosystem (maxItemsPerEcosystem, integer): Maximum number of normalized package/repository records to emit per enabled ecosystem (default 5 ecosystems x 16 = 80 records).
• Delay between requests (delaySeconds, number): Polite pause between requests (default 0.15).

Example input

{
  "queries": [
    "mcp",
    "langchain",
    "opentelemetry",
    "pydantic"
  ],
  "ecosystems": [
    "github",
    "npm",
    "pypi",
    "crates",
    "docker"
  ],
  "pypiPackages": [
    "fastapi",
    "pydantic",
    "openai",
    "anthropic",
    "langchain",
    "httpx",
    "polars",
    "duckdb",
    "uvicorn",
    "sqlalchemy",
    "pytest",
    "ruff",
    "litellm",
    "tiktoken",
    "celery",
    "ray"
  ],
  "enrichNpmDownloads": true,
  "onlyNew": false,
  "maxItemsPerEcosystem": 16
}

Quickstart: API, Python & Automation (Clay / n8n / Make)

Run directly via the Apify REST API (POST /v2/acts/informed_almandine_vzb~github-npm-pypi-docker-package-scraper/runs) or Python SDK:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("informed_almandine_vzb/github-npm-pypi-docker-package-scraper").call(run_input={"queries": ["mcp", "langchain", "opentelemetry", "pydantic"], "ecosystems": ["github", "npm", "pypi", "crates", "docker"], "pypiPackages": ["fastapi", "pydantic", "openai", "anthropic", "langchain", "httpx", "polars", "duckdb", "uvicorn", "sqlalchemy", "pytest", "ruff", "litellm", "tiktoken", "celery", "ray"], "enrichNpmDownloads": true, "onlyNew": false, "maxItemsPerEcosystem": 16})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

Output & example output

One dataset row per result. Default table view columns: ecosystem, packageName, latestVersion, starsCount, downloadsRecent, license, authorOrPublisher, publishedOrPushedAt, repositoryUrl, url. Full structured JSON is stored in the run's default dataset.

Example output item

{
  "ecosystem": "github",
  "packageName": "morluto/rea",
  "latestVersion": "main",
  "description": "Reverse engineer anything with agents, from app behavior down to native binaries.",
  "starsCount": 382870,
  "forksCount": 95792,
  "downloadsRecent": 382870,
  "downloadsTotal": 382870,
  "openIssuesCount": 1062,
  "language": "TypeScript",
  "license": "MIT",
  "authorOrPublisher": "morluto",
  "maintainerEmail": "https://github.com/morluto",
  "keywordsOrTopics": [
    "agent-skills",
    "ai-agents",
    "binary-analysis"
  ],
  "repositoryUrl": "https://github.com/morluto/rea",
  "homepageUrl": "https://rea.tools",
  "publishedOrPushedAt": "2026-10-25T16:19:34Z",
  "url": "https://github.com/morluto/rea",
  "scrapedAt": "2026-10-25T16:20:34+00:00"
}

Pricing (Pay-Per-Result)

This Actor uses transparent Pay-Per-Event pricing with a $0.00 actor-start fee:

Event	What is charged	Price
package-result	Package or repository intelligence record scraped (One normalized open-source package or repository record across GitHub, npm, PyPI, Crates.io, or Docker Hub)	$0.0015 / result ($1.50 / 1,000 results)
apify-actor-start	Actor start fee	$0.00

Concrete pricing math

• package-result (Package or repository intelligence record scraped): 1,000 results = $1.50 (50 results = $0.07, 10,000 results = $15.00). Apify's $5.00/month Free Tier covers up to 3,333 results/month at $0.00 out-of-pocket.
• Empty runs (0 results) & skipped duplicates (onlyNew / onlyChanged): $0.00.

Data and compliance

100% public developer registry metadata accessed via official GitHub REST API, npm Registry API, PyPI JSON API, Crates.io API v1, and Docker Hub API v2. Requests are rate-limited and routed through Apify Proxy. Per-input errors are recorded in the run's ERRORS key-value record and are never charged.

FAQ

Do I need an account, login, or API key?
No. This Actor queries public endpoints directly and requires zero credentials.

What happens if I start a run with empty {} input via API or CLI?
Every input defines identical prefill and default values in .actor/input_schema.json, so an empty {} run automatically returns 50+ live records out of the box.

How does incremental deduplication (onlyNew) work?
When onlyNew: true is enabled, the Actor stores seen record IDs in a named Apify Key-Value Store (open-source-package-intel-state) and emits only newly discovered records on subsequent scheduled runs.

Can I export to CSV, Excel, Google Sheets, Clay, or n8n?
Yes. Every dataset item uses flat, normalized top-level keys (ecosystem, packageName, latestVersion, starsCount, downloadsRecent, license, etc.) for one-click CSV/Excel download or native webhook delivery.