# npm Packages Scraper (`scrapyx/npm-packages-scraper`) Actor

Package metadata and download statistics from the npm registry. Resolves the latest version from dist-tags rather than the last key of `versions`, flags case-mismatched names that resolve to a different real package, and stops search before the offset where npm starts re-serving page 1.

- **URL**: https://apify.com/scrapyx/npm-packages-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## npm Packages Scraper

Package metadata and download statistics from the **npm registry**
(`registry.npmjs.org`) and the **downloads API** (`api.npmjs.org`). No key, no
login, no browser — plain HTTP.

| mode | what you get |
|---|---|
| `packages` | look up specific names: versions, dist-tags, licence, repo, maintainers, deprecation, dependencies, downloads |
| `search` | search the registry and return every match, with its search score, dependents count and download figures |

Add `includeVersions` for one row per published version, and
`downloadsFrom`/`downloadsTo` for per-day download counts.

### What this actor gets right that a naive client does not

**1. Search silently re-serves page 1 past result 5,000.** `from=9750`,
`from=20000`, `from=74657`, `from=100000` all return **page 1's 250 packages
again** — HTTP 200, still claiming `total: 74,657`, no empty page, no error,
no last-page marker. A client walking `from` up to `total` would collect the
same 250 packages about **277 times over** and report ~74,000 "results". This
actor stops before the boundary *and* compares every page against page 1, so
it cannot double-count even if the boundary moves. A measured walk returns
**5,249 unique packages — 7.03% of the claim** — and says so in
`searchDepthCapped` and `reachableFraction`.

**2. Search result order is not stable between requests.** The same offset
returns the same 250 packages in a different order each time, so a package
near a page boundary can be served twice. Deduped by name across the whole
walk; `duplicatePackagesDropped` reports how many.

**3. The last key in `versions` is not the latest version.** `versions` is
ordered by *publish time*, and maintenance releases of old majors are
published after new majors. Measured: express `dist-tags.latest` is **5.2.1**
while the last key is **4.22.2**; react's last key is
`0.0.0-experimental-f789f203-20260825`; typescript's is a nightly. Four of
five popular packages would report the wrong version from
`Object.keys(versions).pop()`. Rows carry `latestVersion` (from dist-tags),
`lastPublishedVersion`, and `lastPublishIsNotLatest`.

**4. Package names are case-sensitive, and the look-alikes are real.**
`/Express` does **not** 404 — it is a real package from 2012 with 3 versions,
latest 3.0.1, and 319 downloads a week against `express`'s 131 million. For
any name that is not already lowercase, this actor fetches the lowercase twin
too and puts the numbers side by side (`lowercaseAlternativeDownloads`,
`lowercaseAlternativeDownloadRatio` — **411,648×** in that example).

**5. The downloads range clamps to about 18 months and moves `start`.**
Asking for `2020-01-01..2026-08-10` (2,414 days) returns **547 days beginning
2025-02-10**, on an HTTP 200 with no warning. Summing the array and calling it
"since 2020" silently drops 1,867 days. The summary reports
`downloadsRangeClamped`, the days requested, the days returned and
`downloadsEffectiveStart`. (A window entirely before coverage gets HTTP 400
`"end date > start date"` — which is not what is wrong with it.)

**6. Bulk download lookups cap at 128 names and refuse scoped names outright.**
129 names → `"exceeded max bulk size of 128"`; `@types/node,express` →
`"scoped packages are not currently supported in bulk lookups"` — though
`@types/node` alone works fine. Batches are split at 128 and scoped names are
peeled out and fetched singly, reported as `bulkDownloadBatches` and
`scopedPackagesFetchedSingly`.

**7. Search carries its own download figures, and they disagree.** Measured on
the same minute: search said express had **129,453,714** weekly downloads
while the downloads API said **131,315,689** — 1.44% apart, and up to 2.30% on
other packages. Two snapshots (the search index carries its own `updated`
timestamp). Both are published, with `downloadsDisagreeWithSearch`.

**8. `score.detail` is a constant.** Across 20 results whose `score.final`
spanned 351 to 2,399, every single one reported
`{popularity: 1, quality: 1, maintenance: 1}`. It is passed through but marked
`scoreDetailIsConstant` so nobody charts it. In the same object `dependents`
is a **string** (`"107427"`), so `dependentsCount` is the coerced integer and
`dependentsRaw` keeps the original.

**9. Deprecation is per version, never per package.** `request` has **all 126
versions deprecated**, including `dist-tags.latest`, and there is no
package-level flag anywhere in the document. Rows carry `latestIsDeprecated`,
`deprecationMessage`, `deprecatedVersionCount` and `allVersionsDeprecated`.

**10. The document you fetch changes what you get.** The abbreviated registry
document is **339 KB** for express against **805 KB** for the full one, with 7
keys per version instead of 26 — it drops description, licence, repository and
maintainers from each version. Abbreviated is the default;
`includeVersionDetail` switches. In `search` mode no registry document is
fetched at all unless `fetchFullMetadata` is on, because the search payload
already carries name, version, description, licence, keywords and publisher.
`metadataSource` says which shape every row came from, so a null is never read
as an absent value.

### Input

```jsonc
{
  "mode": "packages",              // packages | search
  "packages": ["express", "@types/node"],
  "queries": ["graphql client"],   // search mode
  "includeDownloads": true,
  "downloadsPeriod": "last-week",  // last-day | last-week | last-month | last-year
  "downloadsFrom": "2026-01-01",   // ...or a per-day window instead
  "downloadsTo":   "2026-08-01",
  "includeVersions": false,
  "includeVersionDetail": false,
  "fetchFullMetadata": false,      // search mode; always on in packages mode
  "maxResultsPerQuery": 100,       // 0 = unlimited, but see the 5,000 wall
  "pageSize": 250
}
```

### Output

One `SEARCH_SUMMARY` row per query, then the data rows.

| recordType | when |
|---|---|
| `SEARCH_SUMMARY` | always, one per query |
| `PACKAGE` | both modes |
| `VERSION` | when `includeVersions` is on |
| `DOWNLOAD_DAY` | when a `downloadsFrom`/`downloadsTo` window is given |
| `ERROR` | invalid input, a 404 name, or an upstream failure — every input yields at least one row |

Every row carries `_input`, `_source`, `_scrapedAt` and `recordType`.

### Notes

- **No WAF.** All six TLS profiles tried returned byte-identical responses,
  cold, with no warmup and no proxy. A proxy is available but off by default.
- `registry.npmjs.org/robots.txt` names no AI crawler and disallows nothing on
  the paths used; `api.npmjs.org` serves no robots.txt at all (HTTP 404), which
  RFC 9309 §2.3.1.3 treats as no restrictions.
- **Registry documents are big.** Popular packages run into hundreds of KB
  each and `bytesDownloaded` is on every summary; `includeVersions` on
  typescript would emit 3,805 rows.

# Actor input Schema

## `mode` (type: `string`):

packages = look up specific package names. search = search the registry and return every match found, with its search score.

## `packages` (type: `array`):

Names to look up, e.g. express, @types/node, or a full npmjs.com/package/... URL. A trailing @version is stripped. NAMES ARE CASE-SENSITIVE: 'Express' is a real package that is not 'express' (3 versions, latest 3.0.1), so rows carry nameMatchedExactly.

## `queries` (type: `array`):

Search terms for mode='search'. Each runs as its own query with its own summary row.

## `includeDownloads` (type: `boolean`):

Fetch download statistics from api.npmjs.org alongside the registry metadata. Unscoped names are batched 128 per request; scoped names are fetched one at a time because npm refuses them in bulk lookups.

## `downloadsPeriod` (type: `string`):

Which window the download count covers.

## `downloadsFrom` (type: `string`):

Request per-day download counts over an explicit window instead of a single total. NOTE npm clamps long windows to about 18 months and moves the start date silently on an HTTP 200 — the summary reports downloadsRangeClamped and the window that actually came back.

## `downloadsTo` (type: `string`):

End of the daily-downloads window. Required whenever the start is set.

## `includeVersions` (type: `boolean`):

Emit a VERSION row for every published version, with its publish date and whether it is deprecated. Popular packages have thousands (typescript: 3,805), so this multiplies the dataset.

## `includeVersionDetail` (type: `boolean`):

The abbreviated document is used by default: 339 KB for express against 805 KB for the full one. The full document adds description, license, repository, maintainers and keywords to each VERSION row — the package-level copies are present either way.

## `fetchFullMetadata` (type: `boolean`):

Search mode only — packages mode always fetches it, since there is no search payload to build a row from. Off by default in search mode because search already returns name, version, description, licence, keywords and publisher, and a registry document costs 300 KB or more per result. Turn it on to add version counts, dist-tags, deprecation and dependencies. Rows say which shape they came from in metadataSource.

## `maxResultsPerQuery` (type: `integer`):

Search mode only. Set 0 for unlimited — but npm silently re-serves page 1 past result 5,000, so a query can never yield more than about 5,250 unique packages however high you set this. The summary reports searchDepthCapped.

## `pageSize` (type: `integer`):

Between 1 and 250. Requests above 250 are silently clamped to 250 by npm.

## `maxConcurrency` (type: `integer`):

Requests in flight at once. Registry documents are large (hundreds of KB each), so this trades memory and bandwidth for speed.

## `minRequestInterval` (type: `integer`):

Politeness pacing shared across all workers. 0 uses the built-in default.

## `proxyConfiguration` (type: `object`):

Optional and OFF by default. registry.npmjs.org and api.npmjs.org are public, keyless and showed no WAF — all six TLS profiles returned byte-identical responses.

## Actor input object example

```json
{
  "mode": "packages",
  "packages": [
    "react",
    "@types/node"
  ],
  "queries": [
    "graphql client",
    "csv parser"
  ],
  "includeDownloads": true,
  "downloadsPeriod": "last-week",
  "downloadsFrom": "2026-01-01",
  "downloadsTo": "2026-08-01",
  "includeVersions": false,
  "includeVersionDetail": false,
  "fetchFullMetadata": false,
  "maxResultsPerQuery": 100,
  "pageSize": 250,
  "maxConcurrency": 4,
  "minRequestInterval": 0,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "packages": [
        "express"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/npm-packages-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "packages": ["express"] }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/npm-packages-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "packages": [
    "express"
  ]
}' |
apify call scrapyx/npm-packages-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/npm-packages-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/IONIjZ9SD3FxFZNwa/builds/axUvGrf0yVvSWwxzy/openapi.json