crates.io Scraper: Rust Crates, Versions & Downloads

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/crates-io-scraper.md

What it does

Extract crates.io data: description, categories, keywords, licence, downloads per version, dependency lists, repository and maintainers. Track Rust ecosystem adoption. Give it crate names and it returns one structured record per crate.

Three optional toggles decide how deep each record goes:

• includeVersions: the full publish history, with each version's own download count, licence, MSRV, edition, crate size, yank status and publisher.
• includeDependencies: the dependency graph of the default (latest) version, with semver requirements, kinds and platform targets.
• includeOwners: the users and GitHub teams allowed to publish the crate.

serde returns 316 versions and 2 direct dependencies. tokio returns 195 versions and 36 dependencies. Both come back in a couple of seconds.

Input

Field	Type	Default	Notes
crates	array	-	Required. Crate names (serde) or crates.io / docs.rs / lib.rs URLs. Mixed input is fine, URLs are normalised to names automatically.
includeVersions	boolean	false	Add a versions array with one entry per published version.
includeDependencies	boolean	false	Add a dependencies array for the default version. One extra request per crate.
includeOwners	boolean	false	Add an owners array of publishers. One extra request per crate.

{
 "crates": ["serde", "https://crates.io/crates/tokio", "clap"],
 "includeVersions": true,
 "includeDependencies": true,
 "includeOwners": true
}

Which combinations make sense. All three toggles are independent and can be freely mixed. In practice:

• Adoption tracking on a schedule. Leave all three off. One request per crate, a compact record, and downloads / recentDownloads are all you diff.
• Version-level analysis. includeVersions: true alone. Still one request per crate; the version list comes back on the same response, so this costs no extra requests, only record size.
• Supply-chain and SBOM work. includeDependencies: true plus includeOwners: true. Each adds one request per crate, so a 1,000-crate run goes from 1,000 requests to 3,000.

Output

One dataset item per crate. Annotated sample, real values from a live run:

{
 "name": "serde",
 "description": "A generic serialization/deserialization framework",
 "maxVersion": "1.0.229",
 "downloads": 1238022395,
 "recentDownloads": 258045014,
 "categories": ["Encoding", "No standard library", "No dynamic allocation"],
 "keywords": ["no_std", "serde", "serialization"],
 "license": "MIT OR Apache-2.0",
 "repository": "https://github.com/serde-rs/serde",
 "documentation": "https://docs.rs/serde",
 "versionCount": 316,
 "createdAt": "2014-12-05T20:20:39.487502Z",
 "updatedAt": "2026-07-18T23:05:13.266456Z",
 "scrapedAt": "2026-08-06T11:33:38.693Z"
}

Field	Meaning
name	Canonical crate name as crates.io stores it
description	One-line crate description from Cargo.toml
maxVersion	Highest stable version, falling back to the highest version if none is stable
downloads	All-time download count
recentDownloads	Downloads in the last 90 days, the number that actually tracks current adoption
categories	Human-readable category names ("Encoding", not the encoding slug)
keywords	Author-supplied keywords
license	SPDX expression taken from the default version, crates.io stores licence per version, not per crate
repository, documentation	Source and docs URLs. documentation falls back to https://docs.rs/<name> when the author set none
versionCount	Total published versions, including yanked ones
createdAt, updatedAt	First publish and most recent publish
scrapedAt	Run timestamp

With the toggles on, three extra arrays appear:

"versions": [
 {
 "num": "1.0.229",
 "downloads": 20270930,
 "license": "MIT OR Apache-2.0",
 "yanked": false,
 "crateSize": 83669,
 "rustVersion": "1.56",
 "edition": "2021",
 "publishedBy": "dtolnay",
 "createdAt": "2026-07-18T23:05:13.266456Z"
 }
],
"dependencies": [
 {
 "crate": "serde_core",
 "req": "=1.0.229",
 "kind": "normal",
 "optional": false,
 "defaultFeatures": false,
 "features": ["result"],
 "target": null
 }
],
"owners": [
 { "login": "dtolnay", "name": "David Tolnay", "kind": "user", "url": "https://github.com/dtolnay" },
 { "login": "github:serde-rs:publish", "name": "publish", "kind": "team", "url": "https://github.com/serde-rs" }
]

kind on a dependency is normal, dev or build. target is the cfg(.) expression when the dependency is platform-conditional, otherwise null.

RUN_SUMMARY

A RUN_SUMMARY record in the key-value store carries per-run counts, per-source failures and the applied filters:

{
 "cratesRequested": 4,
 "cratesFailed": 1,
 "failures": [
 { "crate": "this-crate-does-not-exist-zzz9", "error": "crate \"this-crate-does-not-exist-zzz9\" not found (404)" }
 ],
 "cratesSaved": 3,
 "filters": { "includeVersions": true, "includeDependencies": true, "includeOwners": true },
 "finishedAt": "2026-08-06T11:33:45.691Z"
}

cratesFailed counts only crates that produced no record at all. A crate whose owners or dependency call failed is still saved, and its partial failure is listed in failures prefixed with owners: or dependencies:.

Use cases

1. Track Rust ecosystem growth. Run a fixed crate list on a schedule and diff recentDownloads and versionCount between runs. Keep the toggles off, one request per crate, the smallest possible record.

{
 "crates": ["serde", "tokio", "clap", "reqwest", "rand", "regex"]
}

2. Pick between competing crates on real adoption. Pull the candidates side by side and compare recentDownloads (current traction, not historical), updatedAt (still maintained?), license and dependency weight.

{
 "crates": ["reqwest", "ureq", "isahc", "hyper"],
 "includeDependencies": true
}

3. Monitor supply-chain surface in Rust projects. Feed the crate list from your Cargo.lock and pull dependencies plus owners for each, licence compatibility, transitive fan-out, and who can push a new version to any of them.

{
 "crates": ["tokio", "hyper", "openssl-sys", "libc"],
 "includeDependencies": true,
 "includeOwners": true
}

Limits and behaviour

• A descriptive User-Agent is mandatory. crates.io returns 403 to generic agents. The Actor always sends one that identifies itself and links back to its public page, which is what the crates.io crawler policy asks for. You cannot get a 403 from a missing agent through this Actor.
• One request per second. crates.io advertises no rate-limit headers but asks crawlers to stay at 1 req/s, so the Actor paces itself. It does not hammer the registry.
• A failing crate never aborts the run. An unknown name returns 404, is logged, and is recorded in RUN_SUMMARY.failures. The Actor only throws when every crate failed.
• Partial failures degrade gracefully. If the owners or dependencies call fails for one crate, the crate is still saved with an empty array for that field, and the failure is recorded.
• Transient errors are retried. 429 and 5xx get three attempts with linear backoff (1s, 2s). 404 and 403 fail fast, since retrying cannot help.
• Licence comes from the default version. crates.io has no crate-level licence field. Relicensed crates therefore show the current licence, not the historical one, includeVersions gives you the per-version history if you need it.
• Hyphens and underscores. crates.io treats them as interchangeable when resolving a name, so serde_json and serde-json both resolve.
• Public data only. No authentication, no personal data beyond the public publisher handles crates.io itself displays.

Defaults: 1 GB memory · 15 min timeout.

FAQ

Do I need a proxy? No. Proxy configuration is not required to run this Actor.

Do I need an account on the source? No. You supply no credentials.

Why does it need a User-Agent then? Politeness, not authentication. crates.io asks automated callers to identify themselves so it can contact abusive clients instead of blanket-blocking. Generic agents get 403; this Actor always sends a compliant one.

What happens if a source is unavailable? It is reported in RUN_SUMMARY.failures and the run continues with the remaining crates.

Can I schedule it? Yes, it is designed for scheduled runs. Diff recentDownloads, maxVersion and updatedAt between runs to detect new releases and adoption shifts.

How many crates can I pass at once? There is no hard cap. Crates are fetched sequentially at one per second, so plan the run timeout accordingly.

Does it return the crate's README or source? No. It returns registry metadata. The documentation and repository URLs point at the full text.

Why is versionCount higher than the versions I recognise? It counts yanked versions too. Each entry in the versions array carries a yanked flag so you can filter them out.

Can I integrate it with something else? Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~crates-io-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "crates": ["serde", "tokio"],
 "includeDependencies": true
 }'

JavaScript example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/crates-io-scraper').call({
 crates: ['serde', 'tokio', 'clap'],
 includeVersions: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const c of items) {
 console.log(`${c.name} v${c.maxVersion}, ${c.recentDownloads.toLocaleString()} downloads/90d`);
}