Hugging Face Hub Scraper

Models, datasets and Spaces from the public Hugging Face Hub API — no API key, no token, no login, no browser.

Modes

Mode	What you get
search	Browse and filter the Hub — by free text, owner/organisation, tag, sorted by downloads, likes, trending score or date. Follows the cursor across as many pages as you ask for.
repos	The full per-repository document for repositories you name (owner/name or a huggingface.co URL), including the file list, config, safetensors index, linked Spaces and storage usage.

Set repoType to models, datasets or spaces. The three share one API shape but not one field set, so nothing here assumes a common one.

Six upstream quirks it corrects

1. There is a WAF, and it challenges only Firefox — as an HTTP 405

The Hub sits behind AWS WAF on CloudFront. Running the full profile ladder against the same routes, on the same IP, minutes apart:

TLS profile	Result
chrome124, chrome131, chrome136, chrome99_android	200
safari17_0, safari18_0	200
edge99, edge101	200
firefox133, firefox135	405, 4 times out of 4 each

The Firefox responses were a 2,125-byte <title>Human Verification</title> page carrying server: CloudFront and x-amzn-waf-action: captcha — on a GET. So the status code says Method Not Allowed, which points at the HTTP verb and says nothing about a challenge. A client that reads only the status learns exactly the wrong lesson and starts debugging its request method.

This actor's profile pool is Firefox-free on purpose, and a Firefox profile is rejected even if one is passed in. Its response classifier reads the x-amzn-waf-action header before it interprets any status code. wafChallengesSeen is published on every run and should always be 0; above zero means the WAF's rules have widened past what was measured.

2. Everything that does not exist is HTTP 401 "Invalid username or password."

Measured on anonymous requests that carry no credentials and need none:

/api/models/nosuchmodel-xyz-12345       -> 401 {"error":"Invalid username or password."}
/api/models/nosuchorg-xyz/nosuchmodel   -> 401  (identical)
/api/datasets/nosuchdataset-xyz-12345   -> 401  (identical)
/api/spaces/nosuchspace-xyz/nope        -> 401  (identical)
/api/nosuchroute                        -> 401  (identical)

Nothing 404s. And the message is actively misleading — it points at authentication for a request that never offered any, so the natural reaction is to go hunting for a token that was never needed. This actor reports a 401 as "does not exist (or is private)", lists the ids in repoIdsNotFound, and never as an auth problem.

3. A gated repository returns 200 with full metadata

meta-llama/Llama-2-7b-hf is gated behind a manual access request — and its API document comes back complete, 772,549 downloads and all. So gated does not mean unreadable, and a 401 does not mean gated. The two are unrelated.

4. gated is a string on gated repos and a boolean on open ones

Same field, two types:

meta-llama/Llama-2-7b-hf      gated: "manual"   (a string)
google-bert/bert-base-uncased gated: false      (a boolean)

gated == True is therefore false for every gated repo, and anything typing that column as boolean chokes on the string. Both are published: gatedRaw verbatim, and a real boolean isGated beside it. On Spaces list rows the field is absent entirely — gatedRaw is null there, which is a third state again, so isGated is the only column safe to filter on.

5. Unknown values are honest; unknown parameter names are not

Sent	Answer
sort=bogus	400 ✖ Invalid sort parameter: bogus — honest
filter=nosuchtag-xyz	200 with 0 results — honest
author=nosuchorg-xyz	200 with 0 results — honest
nosuchparam=x	200 with the normal unfiltered results

So a typo in a value tells you, and a typo in a parameter name silently answers a broader question. Every parameter this actor sends is composed from a fixed set and never passed through from input; sort is validated locally so the run fails before spending a request.

6. limit silently caps at 1000, and there is no total of any kind

limit=10000 returns exactly 1,000 rows, HTTP 200, no error and no echo of what you asked for. Beyond that the cursor in the Link header is the only way forward — a base64 blob encoding a range query, so it must be followed verbatim rather than reconstructed from a page number.

And the API publishes no result count at all — no totalCount field, no X-Total-Count header, nothing. The only honest figure is what the walk actually collected, so that is what reposReturned reports, with upstreamProvidesNoTotal: true stated on every summary so nobody goes looking for the number that isn't there.

Output

Every run emits one SEARCH_SUMMARY row plus one REPO row per repository (and an ERROR row per repository that could not be fetched).

REPO rows carry the upstream object verbatim, plus normalised twins: repoId, repoUrl, owner, repoName, downloads, likes, trendingScore, tags, createdAt, lastModified, isPrivate, isDisabled, gatedRaw/isGated, fileCount, sha, usedStorageBytes, and metadataSource (list or detail). Type-specific fields survive untouched: pipelineTag/libraryName on models, description on datasets, sdk/subdomain on Spaces.

SEARCH_SUMMARY reports the walk honestly: reposReturned, requestsMade, bytesDownloaded, pagesFetched, pageSizes, duplicateReposDropped, stopReason, pageCapHit, wafChallengesSeen, gatedRepos, privateRepos, metadataFromDetailRoute, repoIdsNotFound, requestedLimit, maxLimitBeforeSilentCap and upstreamProvidesNoTotal.

Cost and pacing

fetchFullDetail costs one extra request per result — 4,566 bytes for one model's document against ~435 bytes per list row — in exchange for cardData, config, safetensors, linked Spaces, storage usage and the full file list. repos mode always fetches it, because that document is the job.

cardData (the README front matter) is the largest field in the payload and is dropped unless you set includeCardData.

Limits

• No result count exists upstream. Nothing in this actor can tell you how many repositories match a query before walking it.
• Search pages are sequential — the cursor is a chain, so maxConcurrency applies to detail fetches only.
• Private repositories are invisible to an anonymous client and are indistinguishable from ones that never existed: both are the same 401.
• A proxy is optional and off by default. The WAF here challenges TLS fingerprints, not IPs, and the pool avoids the ones it challenges.