npm Packages Scraper

Package metadata and download statistics from the npm registry (registry.npmjs.org) and the downloads API (api.npmjs.org). No key, no login, no browser — plain HTTP.

mode	what you get
packages	look up specific names: versions, dist-tags, licence, repo, maintainers, deprecation, dependencies, downloads
search	search the registry and return every match, with its search score, dependents count and download figures

Add includeVersions for one row per published version, and downloadsFrom/downloadsTo for per-day download counts.

What this actor gets right that a naive client does not

1. Search silently re-serves page 1 past result 5,000. from=9750, from=20000, from=74657, from=100000 all return page 1's 250 packages again — HTTP 200, still claiming total: 74,657, no empty page, no error, no last-page marker. A client walking from up to total would collect the same 250 packages about 277 times over and report ~74,000 "results". This actor stops before the boundary and compares every page against page 1, so it cannot double-count even if the boundary moves. A measured walk returns 5,249 unique packages — 7.03% of the claim — and says so in searchDepthCapped and reachableFraction.

2. Search result order is not stable between requests. The same offset returns the same 250 packages in a different order each time, so a package near a page boundary can be served twice. Deduped by name across the whole walk; duplicatePackagesDropped reports how many.

3. The last key in versions is not the latest version. versions is ordered by publish time, and maintenance releases of old majors are published after new majors. Measured: express dist-tags.latest is 5.2.1 while the last key is 4.22.2; react's last key is 0.0.0-experimental-f789f203-20260825; typescript's is a nightly. Four of five popular packages would report the wrong version from Object.keys(versions).pop(). Rows carry latestVersion (from dist-tags), lastPublishedVersion, and lastPublishIsNotLatest.

4. Package names are case-sensitive, and the look-alikes are real. /Express does not 404 — it is a real package from 2012 with 3 versions, latest 3.0.1, and 319 downloads a week against express's 131 million. For any name that is not already lowercase, this actor fetches the lowercase twin too and puts the numbers side by side (lowercaseAlternativeDownloads, lowercaseAlternativeDownloadRatio — 411,648× in that example).

5. The downloads range clamps to about 18 months and moves start. Asking for 2020-01-01..2026-08-10 (2,414 days) returns 547 days beginning 2025-02-10, on an HTTP 200 with no warning. Summing the array and calling it "since 2020" silently drops 1,867 days. The summary reports downloadsRangeClamped, the days requested, the days returned and downloadsEffectiveStart. (A window entirely before coverage gets HTTP 400 "end date > start date" — which is not what is wrong with it.)

6. Bulk download lookups cap at 128 names and refuse scoped names outright. 129 names → "exceeded max bulk size of 128"; @types/node,express → "scoped packages are not currently supported in bulk lookups" — though @types/node alone works fine. Batches are split at 128 and scoped names are peeled out and fetched singly, reported as bulkDownloadBatches and scopedPackagesFetchedSingly.

7. Search carries its own download figures, and they disagree. Measured on the same minute: search said express had 129,453,714 weekly downloads while the downloads API said 131,315,689 — 1.44% apart, and up to 2.30% on other packages. Two snapshots (the search index carries its own updated timestamp). Both are published, with downloadsDisagreeWithSearch.

8. score.detail is a constant. Across 20 results whose score.final spanned 351 to 2,399, every single one reported {popularity: 1, quality: 1, maintenance: 1}. It is passed through but marked scoreDetailIsConstant so nobody charts it. In the same object dependents is a string ("107427"), so dependentsCount is the coerced integer and dependentsRaw keeps the original.

9. Deprecation is per version, never per package. request has all 126 versions deprecated, including dist-tags.latest, and there is no package-level flag anywhere in the document. Rows carry latestIsDeprecated, deprecationMessage, deprecatedVersionCount and allVersionsDeprecated.

10. The document you fetch changes what you get. The abbreviated registry document is 339 KB for express against 805 KB for the full one, with 7 keys per version instead of 26 — it drops description, licence, repository and maintainers from each version. Abbreviated is the default; includeVersionDetail switches. In search mode no registry document is fetched at all unless fetchFullMetadata is on, because the search payload already carries name, version, description, licence, keywords and publisher. metadataSource says which shape every row came from, so a null is never read as an absent value.

Input

{
  "mode": "packages",              // packages | search
  "packages": ["express", "@types/node"],
  "queries": ["graphql client"],   // search mode
  "includeDownloads": true,
  "downloadsPeriod": "last-week",  // last-day | last-week | last-month | last-year
  "downloadsFrom": "2026-01-01",   // ...or a per-day window instead
  "downloadsTo":   "2026-08-01",
  "includeVersions": false,
  "includeVersionDetail": false,
  "fetchFullMetadata": false,      // search mode; always on in packages mode
  "maxResultsPerQuery": 100,       // 0 = unlimited, but see the 5,000 wall
  "pageSize": 250
}

Output

One SEARCH_SUMMARY row per query, then the data rows.

recordType	when
SEARCH_SUMMARY	always, one per query
PACKAGE	both modes
VERSION	when includeVersions is on
DOWNLOAD_DAY	when a downloadsFrom/downloadsTo window is given
ERROR	invalid input, a 404 name, or an upstream failure — every input yields at least one row

Every row carries _input, _source, _scrapedAt and recordType.

Notes

• No WAF. All six TLS profiles tried returned byte-identical responses, cold, with no warmup and no proxy. A proxy is available but off by default.
• registry.npmjs.org/robots.txt names no AI crawler and disallows nothing on the paths used; api.npmjs.org serves no robots.txt at all (HTTP 404), which RFC 9309 §2.3.1.3 treats as no restrictions.
• Registry documents are big. Popular packages run into hundreds of KB each and bytesDownloaded is on every summary; includeVersions on typescript would emit 3,805 rows.