# Hugging Face Hub Scraper (`scrapyx/huggingface-hub-scraper`) Actor
Models, datasets and Spaces from the Hugging Face Hub API. Avoids the Firefox TLS profiles this host CAPTCHAs as an HTTP 405, and reports its 401 'Invalid username or password' for what it really is - a repository that does not exist.
- **URL**: https://apify.com/scrapyx/huggingface-hub-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $1.40 / 1,000 results
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.
Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event
## What's an Apify Actor?
Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## Hugging Face Hub Scraper
Models, datasets and Spaces from the **public Hugging Face Hub API** — no API
key, no token, no login, no browser.
### Modes
| Mode | What you get |
| --- | --- |
| `search` | Browse and filter the Hub — by free text, owner/organisation, tag, sorted by downloads, likes, trending score or date. Follows the cursor across as many pages as you ask for. |
| `repos` | The **full** per-repository document for repositories you name (`owner/name` or a `huggingface.co` URL), including the file list, config, safetensors index, linked Spaces and storage usage. |
Set `repoType` to `models`, `datasets` or `spaces`. The three share one API
shape but **not** one field set, so nothing here assumes a common one.
### Six upstream quirks it corrects
#### 1. There *is* a WAF, and it challenges **only Firefox** — as an HTTP 405
The Hub sits behind AWS WAF on CloudFront. Running the full profile ladder
against the same routes, on the same IP, minutes apart:
| TLS profile | Result |
| --- | --- |
| chrome124, chrome131, chrome136, chrome99\_android | **200** |
| safari17\_0, safari18\_0 | **200** |
| edge99, edge101 | **200** |
| **firefox133, firefox135** | **405**, 4 times out of 4 each |
The Firefox responses were a 2,125-byte `Human Verification`
page carrying `server: CloudFront` and `x-amzn-waf-action: captcha` — on a
**GET**. So the status code says *Method Not Allowed*, which points at the HTTP
verb and says nothing about a challenge. A client that reads only the status
learns exactly the wrong lesson and starts debugging its request method.
This actor's profile pool is **Firefox-free on purpose**, and a Firefox profile
is rejected even if one is passed in. Its response classifier reads the
`x-amzn-waf-action` header **before** it interprets any status code.
`wafChallengesSeen` is published on every run and should always be `0`; above
zero means the WAF's rules have widened past what was measured.
#### 2. Everything that does not exist is **HTTP 401 "Invalid username or password."**
Measured on anonymous requests that carry no credentials and need none:
```
/api/models/nosuchmodel-xyz-12345 -> 401 {"error":"Invalid username or password."}
/api/models/nosuchorg-xyz/nosuchmodel -> 401 (identical)
/api/datasets/nosuchdataset-xyz-12345 -> 401 (identical)
/api/spaces/nosuchspace-xyz/nope -> 401 (identical)
/api/nosuchroute -> 401 (identical)
```
Nothing 404s. And the message is actively misleading — it points at
authentication for a request that never offered any, so the natural reaction is
to go hunting for a token that was never needed. This actor reports a 401 as
**"does not exist (or is private)"**, lists the ids in `repoIdsNotFound`, and
never as an auth problem.
#### 3. A **gated** repository returns 200 with full metadata
`meta-llama/Llama-2-7b-hf` is gated behind a manual access request — and its
API document comes back complete, 772,549 downloads and all. So *gated* does
not mean unreadable, and a 401 does not mean gated. The two are unrelated.
#### 4. `gated` is a **string** on gated repos and a **boolean** on open ones
Same field, two types:
```
meta-llama/Llama-2-7b-hf gated: "manual" (a string)
google-bert/bert-base-uncased gated: false (a boolean)
```
`gated == True` is therefore false for *every* gated repo, and anything typing
that column as boolean chokes on the string. Both are published: `gatedRaw`
verbatim, and a real boolean `isGated` beside it. On **Spaces list rows the
field is absent entirely** — `gatedRaw` is `null` there, which is a third state
again, so `isGated` is the only column safe to filter on.
#### 5. Unknown **values** are honest; unknown **parameter names** are not
| Sent | Answer |
| --- | --- |
| `sort=bogus` | **400** `✖ Invalid sort parameter: bogus` — honest |
| `filter=nosuchtag-xyz` | 200 with **0** results — honest |
| `author=nosuchorg-xyz` | 200 with **0** results — honest |
| `nosuchparam=x` | **200 with the normal unfiltered results** |
So a typo in a *value* tells you, and a typo in a *parameter name* silently
answers a broader question. Every parameter this actor sends is composed from a
fixed set and never passed through from input; `sort` is validated locally so
the run fails before spending a request.
#### 6. `limit` silently caps at 1000, and there is **no total of any kind**
`limit=10000` returns exactly **1,000** rows, HTTP 200, no error and no echo of
what you asked for. Beyond that the cursor in the `Link` header is the only way
forward — a base64 blob encoding a range query, so it must be followed verbatim
rather than reconstructed from a page number.
And the API publishes **no result count at all** — no `totalCount` field, no
`X-Total-Count` header, nothing. The only honest figure is what the walk
actually collected, so that is what `reposReturned` reports, with
`upstreamProvidesNoTotal: true` stated on every summary so nobody goes looking
for the number that isn't there.
### Output
Every run emits one `SEARCH_SUMMARY` row plus one `REPO` row per repository
(and an `ERROR` row per repository that could not be fetched).
`REPO` rows carry the upstream object **verbatim**, plus normalised twins:
`repoId`, `repoUrl`, `owner`, `repoName`, `downloads`, `likes`,
`trendingScore`, `tags`, `createdAt`, `lastModified`, `isPrivate`, `isDisabled`,
`gatedRaw`/`isGated`, `fileCount`, `sha`, `usedStorageBytes`, and
`metadataSource` (`list` or `detail`). Type-specific fields survive untouched:
`pipelineTag`/`libraryName` on models, `description` on datasets,
`sdk`/`subdomain` on Spaces.
`SEARCH_SUMMARY` reports the walk honestly: `reposReturned`, `requestsMade`,
`bytesDownloaded`, `pagesFetched`, `pageSizes`, `duplicateReposDropped`,
`stopReason`, `pageCapHit`, `wafChallengesSeen`, `gatedRepos`, `privateRepos`,
`metadataFromDetailRoute`, `repoIdsNotFound`, `requestedLimit`,
`maxLimitBeforeSilentCap` and `upstreamProvidesNoTotal`.
### Cost and pacing
`fetchFullDetail` costs **one extra request per result** — 4,566 bytes for one
model's document against ~435 bytes per list row — in exchange for `cardData`,
`config`, `safetensors`, linked Spaces, storage usage and the full file list.
`repos` mode always fetches it, because that document *is* the job.
`cardData` (the README front matter) is the largest field in the payload and is
dropped unless you set `includeCardData`.
### Limits
- **No result count exists upstream.** Nothing in this actor can tell you how
many repositories match a query before walking it.
- Search pages are sequential — the cursor is a chain, so `maxConcurrency`
applies to detail fetches only.
- Private repositories are invisible to an anonymous client and are
indistinguishable from ones that never existed: both are the same 401.
- A proxy is **optional and off by default**. The WAF here challenges TLS
fingerprints, not IPs, and the pool avoids the ones it challenges.
# Actor input Schema
## `mode` (type: `string`):
search = browse and filter the Hub. repos = full detail for specific repositories you name.
## `repoType` (type: `string`):
models, datasets or spaces. The three share one API shape but NOT one field set — models carry pipeline\_tag and library\_name, datasets carry description, spaces carry sdk and subdomain.
## `search` (type: `string`):
Free-text search over repository names.
## `author` (type: `string`):
Only repositories owned by this user or org, e.g. 'google' or 'meta-llama'. An owner that does not exist honestly returns zero results.
## `filterTag` (type: `string`):
A Hub tag, e.g. 'text-classification', 'pytorch', 'license:apache-2.0'. An unknown tag honestly returns zero results rather than the unfiltered list.
## `sort` (type: `string`):
Verified live on all three repository types.
## `direction` (type: `string`):
-1 = descending (most first), 1 = ascending.
## `repoIds` (type: `array`):
For mode='repos'. 'owner/name' (e.g. google-bert/bert-base-uncased) or a huggingface.co URL. NOTE a repository that does not exist answers HTTP 401 'Invalid username or password.' — not a 404, and not an actual auth problem, since these requests carry no credentials.
## `maxResults` (type: `integer`):
Set 0 for unlimited. This API provides NO result count of any kind — no totalCount, no header — so the only honest figure is what the walk actually collected, reported as reposReturned.
## `pageSize` (type: `integer`):
Between 1 and 1000. Hugging Face silently caps anything above 1000 — a request for 10,000 returned exactly 1,000 with no error.
## `fetchFullDetail` (type: `boolean`):
Search mode only — repos mode always fetches it. Adds cardData, config, safetensors, linked Spaces, storage usage and the full file list, at one extra request per result (4,566 bytes for one model against ~435 per list row).
## `includeCardData` (type: `boolean`):
cardData is the README front matter and is the largest field in the payload, so it is dropped by default.
## `maxConcurrency` (type: `integer`):
Detail fetches in flight at once. Search pages are sequential because the cursor is a chain.
## `minRequestInterval` (type: `integer`):
Politeness pacing shared across all workers. 0 uses the built-in default.
## `proxyConfiguration` (type: `object`):
Optional and OFF by default. There IS an AWS WAF on this host, but it challenges only Firefox TLS fingerprints — which this actor's profile pool excludes — so eight other profiles run clean without a proxy.
## Actor input object example
```json
{
"mode": "search",
"repoType": "models",
"search": "llama",
"author": "google",
"filterTag": "text-generation",
"sort": "downloads",
"direction": "-1",
"repoIds": [
"google-bert/bert-base-uncased",
"https://huggingface.co/datasets/squad"
],
"maxResults": 200,
"pageSize": 100,
"fetchFullDetail": false,
"includeCardData": false,
"maxConcurrency": 4,
"minRequestInterval": 0,
"proxyConfiguration": {
"useApifyProxy": false
}
}
```
# Actor output Schema
## `items` (type: `string`):
One row per scraped record. See the dataset's default view for field definitions.
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {};
// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/huggingface-hub-scraper").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {}
# Run the Actor and wait for it to finish
run = client.actor("scrapyx/huggingface-hub-scraper").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{}' |
apify call scrapyx/huggingface-hub-scraper --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"type": "http",
"url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/huggingface-hub-scraper"
}
}
}
```
The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).
## OpenAPI specification
Download the OpenAPI definition: https://api.apify.com/v2/actors/xJAQbN1MyM3vALQlp/builds/Z273v7cNi9zyEfVl9/openapi.json