# Package Intel: npm + PyPI Metadata & Vulnerability Scraper (`studious_allergy_mig/package-intel-scraper`) Actor

Normalized npm and PyPI package metadata plus OSV.dev vulnerability signals for dependency auditing and supply-chain monitoring.

- **URL**: https://apify.com/studious\_allergy\_mig/package-intel-scraper.md
- **Developed by:** [Conor G](https://apify.com/studious_allergy_mig) (community)
- **Categories:** Developer tools, Automation, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Package Intel Scraper — npm package metadata API + PyPI package data + dependency vulnerability data

Pull registry metadata and known-vulnerability signals for **npm** and
**PyPI** packages in one normalized dataset. Give it a list of package
names, get back one row per package: latest version, license, maintainer
count, download volume (npm), deprecation status, and a list of known
vulnerabilities from **OSV.dev** — all in a single run, no scraping of HTML
pages, no API keys.

### Why this actor

npm's registry (`registry.npmjs.org`) and PyPI's JSON API
(`pypi.org/pypi/<name>/json`) are both fully public and unauthenticated —
this actor talks to them directly. Vulnerability data comes from
[OSV.dev](https://osv.dev), Google's open, free vulnerability database that
aggregates GitHub Security Advisories, PyPI's own advisory feed, and more —
also public, also no key required. All three endpoints were called live
against real packages before this actor shipped; see Data sources below
for exactly what each returns.

**Honesty note on the market for this:** the closest tools in this space —
Snyk, Socket — build their own dependency-graph and vulnerability
infrastructure in-house rather than buying a scraped feed. This actor is
useful for one-off audits, spreadsheet enrichment, and lightweight
monitoring where standing up a full SCA (software composition analysis)
platform is overkill, but it is not a drop-in replacement for those tools.
Willingness to pay for this specific packaging of public data, via a
scraper marketplace, is inferred from the total lack of competition on the
Apify Store — it has not been proven by paying customers.

### Data sources

| Source | Auth | What it returns |
|---|---|---|
| **npm registry** `GET registry.npmjs.org/<name>` | None — fully public | Full package metadata: every version, license, maintainers, repository, dist-tags |
| **npm downloads API** `GET api.npmjs.org/downloads/point/last-week/<name>` | None — fully public | Weekly download count |
| **PyPI JSON API** `GET pypi.org/pypi/<name>/json` | None — fully public | Latest release metadata, full release history, classifiers, project URLs |
| **OSV.dev** `POST api.osv.dev/v1/query` | None — fully public | Known vulnerabilities affecting the package, with severity and CVE/GHSA aliases |

#### npm registry

The full packument (`registry.npmjs.org/<name>`) includes every published
version, the `time` map (first publish + per-version publish timestamps),
maintainers, license, repository URL, and a `deprecated` flag on the
version object when the maintainer has deprecated it. This actor requests
the full packument rather than the abbreviated (`Accept:
application/vnd.npm.install-v1+json`) form, because the abbreviated form
strips the `time` map this actor needs for `publishedAt`/`lastPublishAt`.

#### npm downloads

`api.npmjs.org/downloads/point/last-week/<name>` is a separate, also-public
endpoint. If it fails or the package is too new to have download data, this
actor emits `null` for `weeklyDownloads` — it does not treat a lookup
failure as zero downloads.

#### PyPI JSON API

Returns the current release's metadata plus a `releases` map keyed by every
version ever published, each with an upload timestamp — used here to
compute `publishedAt` (earliest) and `lastPublishAt` (most recent).

**PyPI does not expose two signals npm does, and this actor is explicit
about that rather than guessing:**

- **No download counts.** PyPI's JSON API has no download-count field
  (BigQuery and pypistats.org are separate rate-limited services this actor
  doesn't call). `weeklyDownloads` is always `null` for PyPI packages.
- **No maintainer account list.** PyPI exposes a single free-text
  `maintainer` string, not a list of registered accounts like npm's
  `maintainers` array. `maintainerCount` is always `null` for PyPI
  packages — this actor will not fabricate a count of 1 or 0 from a
  free-text field.

#### OSV.dev

`POST api.osv.dev/v1/query` with `{"package": {"name": ..., "ecosystem":
...}}` returns every vulnerability currently known to affect the package
(any version), each with a GitHub-style severity label
(LOW/MODERATE/HIGH/CRITICAL) when available, or a raw CVSS vector string
otherwise, plus CVE/GHSA aliases. OSV's ecosystem identifier is `npm`
(lowercase) for npm and `PyPI` (capitalized) for PyPI — both were verified
live before shipping.

### Input

The explicit `{name, ecosystem}` object form — recommended whenever you
know which registry you mean:

```json
{
  "packages": [
    { "name": "express", "ecosystem": "npm" },
    { "name": "requests", "ecosystem": "pypi" },
    { "name": "django", "ecosystem": "pypi" }
  ],
  "includeVulnerabilities": true
}
```

Bare strings also work, and use the `ecosystem` default below — but read
**Bare names and cross-ecosystem ambiguity** first:

```json
{
  "packages": ["express", "lodash"],
  "ecosystem": "npm",
  "includeVulnerabilities": true
}
```

- **`packages`** (array, required for a custom run) — bare names (uses the
  default ecosystem below) or `{"name": ..., "ecosystem": "npm" | "pypi"}`
  objects. Duplicates (same ecosystem + name) are deduped automatically.
  Leave this out entirely and the actor runs a small built-in example set
  (`express`, `lodash`, `requests`, `django`), all given with an explicit
  ecosystem — useful for a first test run.
- **`ecosystem`** (string, default `npm`) — default ecosystem applied to
  any bare-string entry in `packages`. This is still a guess on this
  actor's part, not a statement from you — see below.

#### Bare names and cross-ecosystem ambiguity

**This is the one gotcha that matters most in this actor.** npm and PyPI
are separate namespaces with no coordination between them — the same
string can be registered as two completely unrelated packages, one per
registry. The most damaging version of this is a small, low-download
package on one registry squatting the name of a much bigger, unrelated
package on the other.

`fastapi` is a real example, verified live: on PyPI it's the real,
enormously popular FastAPI web framework. On npm it's an unrelated,
near-abandoned package (a handful of weekly downloads, ISC license,
nothing to do with the Python framework). Ask this actor for a bare
`"fastapi"` with no ecosystem stated and, before this behavior existed, it
silently returned the npm row — wrong answer, presented with total
confidence, for a paid data product.

To fix that, **every bare-name entry is checked against the other
ecosystem too**, one extra request per bare name. If the name resolves on
both:

- the row for the ecosystem actually queried is still the only row
  returned (no duplicate rows per ecosystem — that would inflate a
  pay-per-result bill for padding, not signal),
- `nameAmbiguous` is `true` on that row,
- `alsoFoundIn` lists the other ecosystem(s) the name also exists on,
- a `WARNING`-level log line names both ecosystems and tells you how to
  disambiguate.

Worked example — `{"packages": ["fastapi"], "ecosystem": "npm"}`:

```json
{
  "ecosystem": "npm",
  "name": "fastapi",
  "found": true,
  "latestVersion": "0.0.8",
  "license": "ISC",
  "weeklyDownloads": 676,
  "nameAmbiguous": true,
  "alsoFoundIn": ["pypi"],
  "...": "other fields as usual"
}
```

Run log:

```
WARN Ambiguous name "fastapi": it also exists on pypi as an unrelated package. This row was resolved as npm/fastapi. If that's not the one you meant, disambiguate by passing this entry as { "name": "fastapi", "ecosystem": "..." } instead of a bare string.
```

**To skip the probe entirely and get exactly the package you mean**, use
the explicit object form — `{"name": "fastapi", "ecosystem": "pypi"}`.
Any entry that names its own ecosystem short-circuits the cross-ecosystem
check: no extra request, and `nameAmbiguous` is always `false` for that
row, because you already told this actor which registry you meant.

`nameAmbiguous` and `alsoFoundIn` are present on every row (`false` /
`[]` when there's nothing to flag) — never omitted, never `null`.

- **`includeVulnerabilities`** (boolean, default `true`) — when on, queries
  OSV.dev for each package and populates `vulnerabilityCount` and
  `vulnerabilities`. Adds one extra request per package.
- **`maxPackages`** (integer, default `500`) — caps how many packages are
  processed in one run after deduping.

### Output

One dataset item per requested package. Real sample rows from an actual
run (`includeVulnerabilities: true`):

```json
{
  "ecosystem": "npm",
  "name": "express",
  "found": true,
  "latestVersion": "5.2.1",
  "description": "Fast, unopinionated, minimalist web framework",
  "license": "MIT",
  "homepage": "https://expressjs.com/",
  "repositoryUrl": "https://github.com/expressjs/express",
  "publishedAt": "2010-12-29T19:38:25.450Z",
  "lastPublishAt": "2025-12-01T20:49:43.268Z",
  "maintainerCount": 5,
  "versionCount": 288,
  "weeklyDownloads": 125244975,
  "deprecated": false,
  "nameAmbiguous": false,
  "alsoFoundIn": [],
  "vulnerabilityCount": 5,
  "vulnerabilities": [
    { "id": "GHSA-cm5g-3pgc-8rg4", "aliases": ["CVE-2024-10491"], "severity": "MODERATE", "summary": "Express ressource injection" },
    { "id": "GHSA-jj78-5fmv-mv28", "aliases": ["CVE-2024-9266"], "severity": "LOW", "summary": "Express Open Redirect vulnerability" }
  ],
  "errors": [],
  "scrapedAt": "2026-07-31T04:52:33.252Z"
}
```

```json
{
  "ecosystem": "pypi",
  "name": "requests",
  "found": true,
  "latestVersion": "2.34.2",
  "description": "Python HTTP for Humans.",
  "license": "Apache-2.0",
  "homepage": "https://requests.readthedocs.io",
  "repositoryUrl": "https://github.com/psf/requests",
  "publishedAt": "2011-02-14T08:49:42.641660Z",
  "lastPublishAt": "2026-05-14T19:25:26.443000Z",
  "maintainerCount": null,
  "versionCount": 160,
  "weeklyDownloads": null,
  "deprecated": null,
  "nameAmbiguous": false,
  "alsoFoundIn": [],
  "vulnerabilityCount": 16,
  "vulnerabilities": [
    { "id": "GHSA-9hjg-9r4m-mvj7", "aliases": ["CVE-2024-47081"], "severity": "MODERATE", "summary": "Requests vulnerable to .netrc credentials leak via malicious URLs" },
    { "id": "GHSA-9wx4-h78v-vm56", "aliases": ["CVE-2024-35195"], "severity": "MODERATE", "summary": "Requests Session object does not verify requests after making first request with verify=False" }
  ],
  "errors": [],
  "scrapedAt": "2026-07-31T04:52:33.253Z"
}
```

Note `maintainerCount` and `weeklyDownloads` are `null` for the PyPI row —
that's the ecosystem gap described above, not a failed lookup. A package
that doesn't exist on the registry gets `"found": false` with every
metadata field `null` and a message in `errors`, rather than being silently
dropped from the dataset.

#### Run summary

At the end of a run, a per-package status summary is written to the
default key-value store under `PACKAGE_SUMMARY` — which packages resolved,
which were not found, and any fetch errors. The run only fails (non-zero
exit) when **every** requested package fails to resolve — one bad package
name in a long list won't sink the run.

### Use cases

- **Dependency auditing** — feed your `package.json`/`requirements.txt`
  dependency list in and get back license, maintenance status, and known
  vulnerabilities for every package in one pass.
- **Supply-chain monitoring** — re-run weekly against a fixed list of
  packages your organization depends on to catch newly disclosed
  vulnerabilities or a package going deprecated.
- **Package research** — compare download volume, maintainer count, and
  release cadence across a set of candidate libraries before adopting one.
- **Registry analytics** — build a dataset of package metadata across
  npm/PyPI for trend analysis (license distribution, publish frequency,
  ecosystem health) without writing registry-API glue code yourself.

### Pricing

This actor is billed **pay-per-result**: one dataset item = one package
looked up. Enabling or disabling `includeVulnerabilities` does not change
pricing, only the amount of data returned per item.

### Limitations (please read)

- **Only npm and PyPI are supported.** No RubyGems, Maven, crates.io, Go
  modules, or other ecosystems — those would each need their own
  normalizer and haven't been built or verified.
- **Vulnerability data reflects "any version currently known affected,"
  not your exact installed version.** OSV.dev's package-level query (no
  version pinned) returns every vulnerability with an affected-version
  range that includes at least one published version — it does not tell
  you whether the *specific* version you have installed is in that range.
  Cross-reference `latestVersion` and the advisory's affected-range detail
  (visible on the linked GHSA/CVE page) if you need per-version precision.
- **`maintainerCount` and `weeklyDownloads` are always `null` for PyPI.**
  PyPI's public JSON API doesn't expose either signal — see Data sources
  above. This is a gap in what PyPI's API offers, not something this actor
  can recover without a separate paid/rate-limited service.
- **`deprecated` means different things per ecosystem.** For npm it's the
  `deprecated` field a maintainer can set on a version. For PyPI it
  reflects a release being `yanked` (pulled from install resolution) —
  PyPI has no separate "deprecated" concept, so a PyPI package that's
  simply unmaintained but not yanked will show `deprecated: null`, not
  `true`.
- **License strings are not normalized to SPDX.** npm and PyPI both allow
  free-text license fields; this actor passes through what the registry
  reports (falling back to PyPI's license classifiers when the free-text
  field is empty) rather than attempting SPDX normalization.
- **The cross-ecosystem ambiguity probe only checks existence, not
  relevance.** It answers "does this name also exist elsewhere" — it does
  not try to guess which of the two same-named packages you actually
  meant. If `nameAmbiguous` comes back `true`, treat it as "verify this
  before trusting it," not as this actor resolving the ambiguity for you.
- **The probe adds one extra request per bare-name package** (skipped
  entirely for entries using the explicit `{name, ecosystem}` form). For
  large bare-name batches this roughly doubles registry request volume,
  though it does not change pay-per-result pricing, which is billed per
  dataset row, not per request.
- **Data reflects a single point-in-time snapshot** of each registry and
  of OSV.dev at the moment the actor ran — new versions and newly
  disclosed vulnerabilities won't appear until the next run.

# Actor input Schema

## `packages` (type: `array`):

Package names to look up. Each entry is either a bare name (e.g. "express") or an object {"name": "requests", "ecosystem": "pypi"}. A bare name uses the default ecosystem below AND is checked against the other ecosystem too — if the same name exists there as a different, unrelated package (e.g. "fastapi" is a real project on PyPI and a separate, near-abandoned package on npm), the row is flagged nameAmbiguous:true with alsoFoundIn listing the other ecosystem(s), and a warning is logged. Use the {"name", "ecosystem"} object form to state the ecosystem explicitly and skip that check entirely. Leave empty to run the small built-in example set (fast, always non-empty, all entries given explicitly so no ambiguity probing runs).

## `ecosystem` (type: `string`):

Ecosystem used for any entry in "packages" given as a bare string (no explicit ecosystem). Ignored for entries that already specify one.

## `includeVulnerabilities` (type: `boolean`):

When enabled, queries OSV.dev for known vulnerabilities affecting each package and includes vulnerabilityCount + a list of vuln IDs/severities. Adds one extra request per package.

## `maxPackages` (type: `integer`):

Maximum number of packages to process in one run, after deduping by ecosystem+name.

## Actor input object example

```json
{
  "packages": [
    {
      "name": "express",
      "ecosystem": "npm"
    },
    {
      "name": "lodash",
      "ecosystem": "npm"
    },
    {
      "name": "requests",
      "ecosystem": "pypi"
    },
    {
      "name": "django",
      "ecosystem": "pypi"
    }
  ],
  "ecosystem": "npm",
  "includeVulnerabilities": true,
  "maxPackages": 500
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "packages": [
        {
            "name": "express",
            "ecosystem": "npm"
        },
        {
            "name": "lodash",
            "ecosystem": "npm"
        },
        {
            "name": "requests",
            "ecosystem": "pypi"
        },
        {
            "name": "django",
            "ecosystem": "pypi"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("studious_allergy_mig/package-intel-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "packages": [
        {
            "name": "express",
            "ecosystem": "npm",
        },
        {
            "name": "lodash",
            "ecosystem": "npm",
        },
        {
            "name": "requests",
            "ecosystem": "pypi",
        },
        {
            "name": "django",
            "ecosystem": "pypi",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("studious_allergy_mig/package-intel-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "packages": [
    {
      "name": "express",
      "ecosystem": "npm"
    },
    {
      "name": "lodash",
      "ecosystem": "npm"
    },
    {
      "name": "requests",
      "ecosystem": "pypi"
    },
    {
      "name": "django",
      "ecosystem": "pypi"
    }
  ]
}' |
apify call studious_allergy_mig/package-intel-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=studious_allergy_mig/package-intel-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/jrI6H6W0jbLmr3nFB/builds/sZqKT5u0JcsYARg4m/openapi.json