# crates.io Scraper: Rust Crates, Versions & Downloads (`arman-bd/crates-io-scraper`) Actor

Extract crates.io data: description, categories, keywords, licence, downloads per version, dependency lists, repository and maintainers. Track Rust ecosystem adoption.

- **URL**: https://apify.com/arman-bd/crates-io-scraper.md
- **Developed by:** [Arman Hossain](https://apify.com/arman-bd) (community)
- **Categories:** Developer tools, Automation, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.56 / 1,000 crate scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## crates.io Scraper: Rust Crates, Versions & Downloads

![crates.io Scraper: Description, categories, keywords, licence, downloads per version, dependency graph, repository and maintainers for any Rust crate](https://api.apify.com/v2/key-value-stores/ZQOcNAOHrIgTacAmy/records/crates-io-scraper.jpg)

**Agent skill: [SKILL.md](https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/crates-io-scraper.md)**

```
https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/crates-io-scraper.md
```

### What it does

Extract crates.io data: description, categories, keywords, licence, downloads per version, dependency lists, repository and maintainers. Track Rust ecosystem adoption. Give it crate names and it returns one structured record per crate.

Three optional toggles decide how deep each record goes:

- **`includeVersions`**: the full publish history, with each version's own download count, licence, MSRV, edition, crate size, yank status and publisher.
- **`includeDependencies`**: the dependency graph of the default (latest) version, with semver requirements, kinds and platform targets.
- **`includeOwners`**: the users and GitHub teams allowed to publish the crate.

`serde` returns 316 versions and 2 direct dependencies. `tokio` returns 195 versions and 36 dependencies. Both come back in a couple of seconds.

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `crates` | array | - | **Required.** Crate names (`serde`) or crates.io / docs.rs / lib.rs URLs. Mixed input is fine, URLs are normalised to names automatically. |
| `includeVersions` | boolean | `false` | Add a `versions` array with one entry per published version. |
| `includeDependencies` | boolean | `false` | Add a `dependencies` array for the default version. One extra request per crate. |
| `includeOwners` | boolean | `false` | Add an `owners` array of publishers. One extra request per crate. |

```json
{
 "crates": ["serde", "https://crates.io/crates/tokio", "clap"],
 "includeVersions": true,
 "includeDependencies": true,
 "includeOwners": true
}
```

**Which combinations make sense.** All three toggles are independent and can be freely mixed. In practice:

- **Adoption tracking on a schedule.** Leave all three off. One request per crate, a compact record, and `downloads` / `recentDownloads` are all you diff.
- **Version-level analysis.** `includeVersions: true` alone. Still one request per crate; the version list comes back on the same response, so this costs no extra requests, only record size.
- **Supply-chain and SBOM work.** `includeDependencies: true` plus `includeOwners: true`. Each adds one request per crate, so a 1,000-crate run goes from 1,000 requests to 3,000.

### Output

One dataset item per crate. Annotated sample, real values from a live run:

```json
{
 "name": "serde",
 "description": "A generic serialization/deserialization framework",
 "maxVersion": "1.0.229",
 "downloads": 1238022395,
 "recentDownloads": 258045014,
 "categories": ["Encoding", "No standard library", "No dynamic allocation"],
 "keywords": ["no_std", "serde", "serialization"],
 "license": "MIT OR Apache-2.0",
 "repository": "https://github.com/serde-rs/serde",
 "documentation": "https://docs.rs/serde",
 "versionCount": 316,
 "createdAt": "2014-12-05T20:20:39.487502Z",
 "updatedAt": "2026-07-18T23:05:13.266456Z",
 "scrapedAt": "2026-08-06T11:33:38.693Z"
}
```

| Field | Meaning |
|---|---|
| `name` | Canonical crate name as crates.io stores it |
| `description` | One-line crate description from `Cargo.toml` |
| `maxVersion` | Highest stable version, falling back to the highest version if none is stable |
| `downloads` | All-time download count |
| `recentDownloads` | Downloads in the last 90 days, the number that actually tracks current adoption |
| `categories` | Human-readable category names (`"Encoding"`, not the `encoding` slug) |
| `keywords` | Author-supplied keywords |
| `license` | SPDX expression taken from the **default version**, crates.io stores licence per version, not per crate |
| `repository`, `documentation` | Source and docs URLs. `documentation` falls back to `https://docs.rs/<name>` when the author set none |
| `versionCount` | Total published versions, including yanked ones |
| `createdAt`, `updatedAt` | First publish and most recent publish |
| `scrapedAt` | Run timestamp |

With the toggles on, three extra arrays appear:

```json
"versions": [
 {
 "num": "1.0.229",
 "downloads": 20270930,
 "license": "MIT OR Apache-2.0",
 "yanked": false,
 "crateSize": 83669,
 "rustVersion": "1.56",
 "edition": "2021",
 "publishedBy": "dtolnay",
 "createdAt": "2026-07-18T23:05:13.266456Z"
 }
],
"dependencies": [
 {
 "crate": "serde_core",
 "req": "=1.0.229",
 "kind": "normal",
 "optional": false,
 "defaultFeatures": false,
 "features": ["result"],
 "target": null
 }
],
"owners": [
 { "login": "dtolnay", "name": "David Tolnay", "kind": "user", "url": "https://github.com/dtolnay" },
 { "login": "github:serde-rs:publish", "name": "publish", "kind": "team", "url": "https://github.com/serde-rs" }
]
```

`kind` on a dependency is `normal`, `dev` or `build`. `target` is the `cfg(.)` expression when the dependency is platform-conditional, otherwise `null`.

#### `RUN_SUMMARY`

A `RUN_SUMMARY` record in the key-value store carries per-run counts, per-source failures and the applied filters:

```json
{
 "cratesRequested": 4,
 "cratesFailed": 1,
 "failures": [
 { "crate": "this-crate-does-not-exist-zzz9", "error": "crate \"this-crate-does-not-exist-zzz9\" not found (404)" }
 ],
 "cratesSaved": 3,
 "filters": { "includeVersions": true, "includeDependencies": true, "includeOwners": true },
 "finishedAt": "2026-08-06T11:33:45.691Z"
}
```

`cratesFailed` counts only crates that produced no record at all. A crate whose owners or dependency call failed is still saved, and its partial failure is listed in `failures` prefixed with `owners:` or `dependencies:`.

### Use cases

**1. Track Rust ecosystem growth.** Run a fixed crate list on a schedule and diff `recentDownloads` and `versionCount` between runs. Keep the toggles off, one request per crate, the smallest possible record.

```json
{
 "crates": ["serde", "tokio", "clap", "reqwest", "rand", "regex"]
}
```

**2. Pick between competing crates on real adoption.** Pull the candidates side by side and compare `recentDownloads` (current traction, not historical), `updatedAt` (still maintained?), `license` and dependency weight.

```json
{
 "crates": ["reqwest", "ureq", "isahc", "hyper"],
 "includeDependencies": true
}
```

**3. Monitor supply-chain surface in Rust projects.** Feed the crate list from your `Cargo.lock` and pull dependencies plus owners for each, licence compatibility, transitive fan-out, and who can push a new version to any of them.

```json
{
 "crates": ["tokio", "hyper", "openssl-sys", "libc"],
 "includeDependencies": true,
 "includeOwners": true
}
```

### Limits and behaviour

- **A descriptive User-Agent is mandatory.** crates.io returns `403` to generic agents. The Actor always sends one that identifies itself and links back to its public page, which is what the crates.io crawler policy asks for. You cannot get a `403` from a missing agent through this Actor.
- **One request per second.** crates.io advertises no rate-limit headers but asks crawlers to stay at 1 req/s, so the Actor paces itself. It does not hammer the registry.
- **A failing crate never aborts the run.** An unknown name returns `404`, is logged, and is recorded in `RUN_SUMMARY.failures`. The Actor only throws when *every* crate failed.
- **Partial failures degrade gracefully.** If the owners or dependencies call fails for one crate, the crate is still saved with an empty array for that field, and the failure is recorded.
- **Transient errors are retried.** `429` and `5xx` get three attempts with linear backoff (1s, 2s). `404` and `403` fail fast, since retrying cannot help.
- **Licence comes from the default version.** crates.io has no crate-level licence field. Relicensed crates therefore show the current licence, not the historical one, `includeVersions` gives you the per-version history if you need it.
- **Hyphens and underscores.** crates.io treats them as interchangeable when resolving a name, so `serde_json` and `serde-json` both resolve.
- **Public data only.** No authentication, no personal data beyond the public publisher handles crates.io itself displays.

**Defaults:** 1 GB memory · 15 min timeout.

### FAQ

**Do I need a proxy?** No. Proxy configuration is not required to run this Actor.

**Do I need an account on the source?** No. You supply no credentials.

**Why does it need a User-Agent then?** Politeness, not authentication. crates.io asks automated callers to identify themselves so it can contact abusive clients instead of blanket-blocking. Generic agents get `403`; this Actor always sends a compliant one.

**What happens if a source is unavailable?** It is reported in `RUN_SUMMARY.failures` and the run continues with the remaining crates.

**Can I schedule it?** Yes, it is designed for scheduled runs. Diff `recentDownloads`, `maxVersion` and `updatedAt` between runs to detect new releases and adoption shifts.

**How many crates can I pass at once?** There is no hard cap. Crates are fetched sequentially at one per second, so plan the run timeout accordingly.

**Does it return the crate's README or source?** No. It returns registry metadata. The `documentation` and `repository` URLs point at the full text.

**Why is `versionCount` higher than the versions I recognise?** It counts yanked versions too. Each entry in the `versions` array carries a `yanked` flag so you can filter them out.

**Can I integrate it with something else?** Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.

### API example

```bash
curl -X POST "https://api.apify.com/v2/acts/arman-bd~crates-io-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "crates": ["serde", "tokio"],
 "includeDependencies": true
 }'
```

### JavaScript example

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/crates-io-scraper').call({
 crates: ['serde', 'tokio', 'clap'],
 includeVersions: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const c of items) {
 console.log(`${c.name} v${c.maxVersion}, ${c.recentDownloads.toLocaleString()} downloads/90d`);
}
```

# Actor input Schema

## `crates` (type: `array`):

Crate names as published on crates.io. for https://crates.io/crates/serde the name is 'serde'. Full crates.io, docs.rs and lib.rs URLs are accepted and normalized automatically. Hyphens and underscores are interchangeable on crates.io.

## `includeVersions` (type: `boolean`):

Add a `versions` array with one entry per published version. version number, its own download count, licence, MSRV, edition, crate size, yank status and publisher. Large crates have hundreds of versions, so this multiplies record size.

## `includeDependencies` (type: `boolean`):

Add a `dependencies` array for the crate's default (latest) version. dependency name, semver requirement, kind (normal / dev / build), optionality, enabled features and platform target. Costs one extra request per crate.

## `includeOwners` (type: `boolean`):

Add an `owners` array listing the users and GitHub teams that can publish the crate. Costs one extra request per crate.

## Actor input object example

```json
{
  "crates": [
    "serde",
    "https://crates.io/crates/tokio"
  ],
  "includeVersions": false,
  "includeDependencies": false,
  "includeOwners": false
}
```

# Actor output Schema

## `items` (type: `string`):

Every record the run produced.

## `runsummary` (type: `string`):

The RUN_SUMMARY record from the run's key-value store.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "crates": [
        "serde",
        "tokio",
        "clap"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("arman-bd/crates-io-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "crates": [
        "serde",
        "tokio",
        "clap",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("arman-bd/crates-io-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "crates": [
    "serde",
    "tokio",
    "clap"
  ]
}' |
apify call arman-bd/crates-io-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,arman-bd/crates-io-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/N7LWf5tHDhKevVlhZ/builds/f6KtGh0mxhf53P0di/openapi.json