# AI Model Benchmark Tracker (`gochujang/ai-model-benchmark-tracker`) Actor

Track top open-source LLMs from HuggingFace by downloads, architecture, parameters, and capability. Enriched with efficiency scores, recency labels, and capability tags. No API key required.

- **URL**: https://apify.com/gochujang/ai-model-benchmark-tracker.md
- **Developed by:** [Hojun Lee](https://apify.com/gochujang) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 model trackeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does AI Model Benchmark Tracker do?

**AI Model Benchmark Tracker** pulls the top **open-source large language models** from the HuggingFace public API — sorted by monthly downloads — and enriches each record with architecture metadata, **parameter counts**, efficiency scores, deployment suitability, and capability profiles. It also produces an ecosystem-level summary that identifies the most-downloaded model, fastest-growing release, SOTA leaderboard, and the best models for **crypto and trading agent use cases**. No API key or HuggingFace account is required.

### Why use AI Model Benchmark Tracker?

- **Comprehensive LLM landscape coverage** — tracks up to 100 top `text-generation` models in a single run, spanning every major family: Llama, Mistral, Gemma, Qwen, DeepSeek, Phi, Falcon, and more, so you never have to manually browse HuggingFace to find what's trending.
- **Deep enrichment beyond raw download counts** — every model gets a `performance_tier`, `cost_efficiency_score`, `deployment_readiness`, `capability_profile`, `reasoning_vs_speed_tradeoff`, and a `context_window_tier`, turning raw metadata into actionable selection criteria.
- **Crypto and AI agent use-case fit scoring** — a dedicated `crypto_use_case_fit` field (`ideal` / `good` / `moderate` / `poor`) rates each model specifically for trading bots, on-chain analytics agents, and finance automation pipelines.
- **Open vs. closed source intelligence** — `open_vs_closed_signal` classifies every model as `fully_open`, `open_weights`, `api_only`, or `closed`, and `open_vs_closed_advantage` identifies where open-weight models are genuinely competitive with proprietary alternatives.
- **Affordable weekly monitoring** — at **$0.001 per model tracked** plus a $0.001 start fee, tracking 50 models costs just $0.051, making it practical to run weekly and keep your model selection strategy current as the LLM landscape evolves rapidly.

### How to use AI Model Benchmark Tracker

1. Go to the [Apify Store](https://apify.com/store) and open **AI Model Benchmark Tracker**.
2. Click **Try for free** to open the Actor in Apify Console.
3. Set **limit** (default 50, max 100) to control how many top models to fetch.
4. Optionally set **filterArchitecture** (e.g. `llama`, `mistral`, `gemma`) to narrow results to a specific model family.
5. Toggle **excludeQuantized** to skip GGUF/GPTQ/AWQ variants, or **instructTunedOnly** to return only chat/instruct models.
6. Set **minDownloads** to filter out low-adoption models below a downloads threshold.
7. Click **Start** — the run typically completes in under 30 seconds.
8. Download results as JSON or connect via the Apify API for dashboard pipelines.

### Input

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `limit` | integer | `50` | Number of top models to fetch by monthly downloads (max 100) |
| `filterArchitecture` | string | `""` | Filter by base model family (e.g. `llama`, `mistral`, `gemma`, `qwen`, `phi`) |
| `excludeQuantized` | boolean | `false` | When `true`, skips GGUF, GPTQ, and AWQ quantized model variants |
| `instructTunedOnly` | boolean | `false` | When `true`, returns only instruction-tuned / chat models |
| `minDownloads` | integer | `0` | Minimum 30-day download count threshold |

### Output

The dataset contains one record per model followed by a final summary record:

```json
[
  {
    "model_id": "meta-llama/Meta-Llama-3-8B-Instruct",
    "architecture": "LlamaForCausalLM",
    "parameters_b": 8.0,
    "params_tier": "medium",
    "downloads_30d": 4823901,
    "likes": 18234,
    "license": "llama3",
    "is_instruction_tuned": true,
    "is_quantized": false,
    "is_open_weight": true,
    "recency_label": "established",
    "base_model": "llama",
    "capability_tags": ["text-generation", "code", "reasoning"],
    "efficiency_score": 602987.63,
    "context_window": 131072,
    "performance_tier": "frontier",
    "cost_efficiency_score": 84.21,
    "cost_efficiency_grade": "A",
    "capability_profile": "reasoning_dominant",
    "deployment_suitability": "production_api",
    "deployment_readiness": "production_ready",
    "improvement_velocity": "incremental",
    "crypto_use_case_fit": "ideal",
    "context_efficiency_tier": "long",
    "context_window_tier": "long",
    "capability_tier": "frontier",
    "reasoning_vs_speed_tradeoff": "high_reasoning",
    "multimodal_capability": "text_only",
    "open_vs_closed_signal": "open_weights",
    "open_vs_closed_advantage": "open_source_strong",
    "benchmark_trend": "stable",
    "hf_url": "https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct",
    "scraped_at": "2025-06-01T08:00:00+00:00"
  },
  {
    "model_id": "mistralai/Mistral-7B-Instruct-v0.3",
    "architecture": "MistralForCausalLM",
    "parameters_b": 7.0,
    "params_tier": "medium",
    "downloads_30d": 2104567,
    "likes": 9812,
    "license": "apache-2.0",
    "is_instruction_tuned": true,
    "is_quantized": false,
    "is_open_weight": true,
    "recency_label": "recent",
    "base_model": "mistral",
    "capability_tags": ["text-generation", "multilingual"],
    "efficiency_score": 300652.43,
    "context_window": 32768,
    "performance_tier": "leading",
    "cost_efficiency_score": 78.55,
    "cost_efficiency_grade": "B",
    "capability_profile": "balanced",
    "deployment_suitability": "production_api",
    "deployment_readiness": "production_ready",
    "improvement_velocity": "significant",
    "crypto_use_case_fit": "good",
    "context_efficiency_tier": "standard",
    "context_window_tier": "standard",
    "capability_tier": "top_tier",
    "reasoning_vs_speed_tradeoff": "balanced",
    "multimodal_capability": "text_only",
    "open_vs_closed_signal": "fully_open",
    "open_vs_closed_advantage": "open_source_strong",
    "benchmark_trend": "improving",
    "hf_url": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3",
    "scraped_at": "2025-06-01T08:00:00+00:00"
  }
]
```

### Data fields

| Field | Description |
|-------|-------------|
| `model_id` | Full HuggingFace model identifier (e.g. `meta-llama/Meta-Llama-3-8B-Instruct`) |
| `architecture` | Transformer class name from model config (e.g. `LlamaForCausalLM`) |
| `parameters_b` | Parameter count in billions, extracted from model card, tags, or model name |
| `params_tier` | Size bucket: `tiny` (<1B), `small` (<7B), `medium` (<30B), `large` (<100B), `giant` (100B+) |
| `downloads_30d` | HuggingFace downloads over the past 30 days |
| `likes` | HuggingFace community likes count |
| `license` | Model license identifier (e.g. `llama3`, `apache-2.0`, `mit`) |
| `is_instruction_tuned` | `true` if model name contains instruct/chat/IT/DPO/RLHF markers |
| `is_quantized` | `true` if GGUF, GPTQ, AWQ, BnB, 4-bit, or 8-bit quantization detected |
| `is_open_weight` | `true` if weights are available under an open or permissive license |
| `recency_label` | `new` (<90d), `recent` (<1y), `established` (<2y), `legacy` (2y+) |
| `base_model` | Dominant model family: `llama`, `mistral`, `gemma`, `qwen`, `falcon`, `phi`, `deepseek`, `other` |
| `capability_tags` | Standardised capability list: `text-generation`, `code`, `multilingual`, `math`, `reasoning`, `vision` |
| `efficiency_score` | Downloads per billion parameters — proxy for community adoption relative to model size |
| `context_window` | Maximum context length in tokens (when available from model card or tags) |
| `performance_tier` | Download-rank tier: `frontier`, `leading`, `competitive`, `average`, `below_average` |
| `cost_efficiency_score` | 0–100 score: high downloads at small parameter count = high cost efficiency |
| `cost_efficiency_grade` | Letter grade A/B/C/D/F derived from `cost_efficiency_score` |
| `capability_profile` | Dominant use-case profile: `coding_dominant`, `reasoning_dominant`, `multimodal_dominant`, `instruction_following`, `raw_knowledge`, `balanced` |
| `deployment_suitability` | Best deployment context: `production_api`, `edge_deployment`, `fine_tuning_base`, `research_only`, `enterprise_only` |
| `deployment_readiness` | Production maturity: `production_ready`, `beta`, `research_only`, `deprecated` |
| `improvement_velocity` | Release momentum: `quantum_leap`, `significant`, `incremental`, `stagnant` |
| `crypto_use_case_fit` | Suitability for crypto/trading agents: `ideal`, `good`, `moderate`, `poor`, `not_suitable` |
| `capability_tier` | Composite capability level: `frontier`, `top_tier`, `competitive`, `mainstream`, `legacy` |
| `reasoning_vs_speed_tradeoff` | Speed/depth axis: `max_reasoning`, `high_reasoning`, `balanced`, `speed_focused`, `fastest` |
| `context_window_tier` | Context size class: `ultra_long` (>500K), `long` (>100K), `standard` (>32K), `limited`, `very_limited` |
| `multimodal_capability` | `full_multimodal`, `vision_only`, `text_only`, `specialized` |
| `open_vs_closed_signal` | Openness classification: `fully_open`, `open_weights`, `api_only`, `closed`, `unknown` |
| `open_vs_closed_advantage` | Competitive positioning: `open_source_strong`, `open_source_competitive`, `gap_closing`, `closed_dominant`, `proprietary_required` |
| `benchmark_trend` | Inferred trajectory: `improving`, `stable`, `declining`, `new_entry` |
| `hf_url` | Direct HuggingFace model page URL |
| `scraped_at` | ISO 8601 UTC timestamp of data collection |

### Cost estimation

Pricing: **$0.001 per model tracked** + **$0.001 one-time start fee** (Pay-Per-Event).

| Scenario | Models tracked | Start fee | Model fees | Total cost |
|----------|---------------|-----------|-----------|-----------|
| Small (quick scan) | 10 | $0.001 | $0.010 | **$0.011** |
| Medium (default) | 50 | $0.001 | $0.050 | **$0.051** |
| Large (full top 100) | 100 | $0.001 | $0.100 | **$0.101** |

A weekly scheduled run tracking 50 models costs approximately **$0.20/month** — well within the free tier for most Apify plans.

### FAQ

**Does this Actor require a HuggingFace API token?**
No. It uses the HuggingFace public `/api/models` endpoint, which is freely accessible without authentication. No HuggingFace account is needed.

**How is `parameters_b` determined when it isn't explicitly listed?**
The Actor checks three sources in order: (1) the model card metadata fields `parameter_count` / `num_parameters`, (2) HuggingFace tags like `7B` or `13B`, and (3) a regex pattern applied to the model ID itself (e.g. `llama-3-70b-instruct` → 70B). If none of these yield a result, `parameters_b` will be `null`.

**Can I track models other than text-generation?**
Currently the Actor filters for the `text-generation` pipeline tag to stay focused on the LLM landscape relevant to agents, chatbots, and code generation. Support for additional pipeline types such as `image-to-text` or `text-to-image` may be added in future versions.

**Disclaimer:** Download counts and derived scores reflect community adoption signals on HuggingFace and do not constitute endorsements of any specific model. Performance tiers are proxies based on download rank, not formal benchmark evaluations (e.g. MMLU, HumanEval). Always evaluate models on your own tasks and data before deploying to production systems.

Keywords: open source LLM tracker, HuggingFace model benchmark, AI model comparison, LLM download rankings, best open source language model, LLM efficiency score, AI model leaderboard, Llama Mistral Gemma comparison, LLM deployment readiness, crypto AI agent model selection

***

### Related actors

- [HuggingFace Model Tracker](https://apify.com/gochujang/huggingface-model-tracker) — Live trending and upload activity for models benchmarked here
- [Semantic Scholar Papers](https://apify.com/gochujang/semantic-scholar-papers) — Research papers behind the models in this benchmark tracker
- [AI Token Sector Tracker](https://apify.com/gochujang/ai-token-sector-tracker) — AI crypto token performance alongside AI model release activity

### Feedback

If this actor powers your AI research, a review helps others find it: [Leave a review on Apify Store](https://apify.com/gochujang/ai-model-benchmark-tracker#reviews)

***

**Keywords:** LLM benchmark, open source AI models, HuggingFace models, language model tracker, AI model downloads, model parameters, AI efficiency score, LLM comparison, model capability, AI research tracker, open weight models, instruction tuned, quantized models, AI deployment, model performance ranking

# Actor input Schema

## `limit` (type: `integer`):

How many top models to fetch from HuggingFace (sorted by downloads). Max 100.

## `filterArchitecture` (type: `string`):

Only include models matching this base family (e.g. 'llama', 'mistral', 'gemma'). Leave empty for all.

## `excludeQuantized` (type: `boolean`):

If true, skip quantized model variants.

## `instructTunedOnly` (type: `boolean`):

If true, only return instruction-tuned / chat models.

## `minDownloads` (type: `integer`):

Skip models with fewer than this many downloads in the past 30 days.

## Actor input object example

```json
{
  "limit": 50,
  "filterArchitecture": "",
  "excludeQuantized": false,
  "instructTunedOnly": false,
  "minDownloads": 0
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("gochujang/ai-model-benchmark-tracker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("gochujang/ai-model-benchmark-tracker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call gochujang/ai-model-benchmark-tracker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gochujang/ai-model-benchmark-tracker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gknYjmjTQ4klX0oMO/builds/4n0OhAbbjEIUDMg5S/openapi.json