PyPI Scraper

A PyPI scraper for package metadata straight from PyPI's own JSON API: current version, licence, declared dependencies, supported Python versions, the full release history with dates, and which releases were yanked after publication. No key, no login, no scraping of the website. Give it a list of package names, or a name pattern to match across the whole index of over 900,000 projects (907,502 when last counted, 7 October 2026).

What this answers that a package page does not

"What does this actually depend on?" requires_dist is the real dependency list with version specifiers and environment markers, exactly as the maintainer declared it. That is what a supply-chain review or a licence audit needs, and reading it off the rendered page is guesswork.

"Is this project still alive?" Every row carries last_release_at and days_since_last_release. Sort by it and an unmaintained dependency stands out immediately. The run summary counts how many of your packages have had no release in over two years.

"Are we pinned to a release that was pulled?" yanked_versions lists every version whose files were all yanked. Maintainers yank releases for broken builds, licence mistakes and security problems, and a pinned dependency on one is a finding that appears nowhere in a naive scrape. A version with even one live file is not counted, because it is still installable.

Two ways to use this PyPI scraper

By name, the fast path. One API call per package, ten at a time:

packages: ["httpx", "requests", "django"]

By pattern, when you do not know the names:

name_pattern: "django-rest*"

A pattern with no wildcard is treated as "contains", so boto matches every project with boto in the name. Matches are returned shortest-name-first, which puts the canonical project ahead of its forks and plugins.

About the pattern search

PyPI publishes no search API. The /search/ page is HTML, hands about 3 KB to anything that is not a browser, and the maintainers ask people not to scrape it. So this Actor does not.

Instead a pattern is matched against PyPI's simple index, the official machine-readable list of every project, requested with the documented JSON accept header. It is one download of the full index followed by local filtering. That costs a few seconds more at the start of a run and asks a service funded by donations for exactly one page instead of many.

If you already know the names, pass packages and skip the index entirely.

Input

Field	Type	Default	What it does
packages	list of strings	["httpx", "requests", "django"]	Project names to look up. A string is split on whitespace, commas and semicolons. When this list is non-empty it wins and name_pattern is ignored.
name_pattern	string	— (example django-rest*)	Glob or substring matched against the full simple index when packages is empty.
limit	integer (1–1000 after coercion)	50	How many projects to consider: the first limit names in input order, or the first limit pattern matches shortest-name-first. 0 falls back to 50; values above 1000 are clamped to 1000.
retries	integer (1–6 after coercion)	3	Attempts per upstream request with backoff. 0 falls back to 3; values above 6 are clamped to 6.
cursor	string	—	Continuation cursor from a previous response; repeat the original input with this cursor to collect the rest of a large job.

Give either a list of names or a pattern. An empty input is rejected with a message rather than walking the index for no reason. Names that are not valid project names are reported per name and do not produce a row.

A verified lookup input is:

{"packages":["httpx","requests","django"]}

Output

One dataset row per project found. The run summary and any failures land in the key-value store records summary and errors.

Every row carries 30 stable fields plus fetched_at (ISO 8601 UTC). The fields fall into these groups:

• Identity: name, version, summary, keywords, url, package_url
• Legal: license, license_classifiers (the trove classifiers, which are often more reliable than the free-text field)
• People: author, author_email, maintainer
• Links: home_page, project_urls (docs, changelog, source, issues)
• Compatibility: requires_python, python_versions, development_status, classifiers
• Dependencies: requires_dist, dependency_count
• History: release_count, first_release_at, last_release_at, days_since_last_release
• Yanking: yanked, yanked_reason, yanked_versions
• Current release files: latest_files, latest_size_bytes, has_wheel
• Collection: position, fetched_at

has_wheel is worth a note: a project shipping only a source distribution has to compile on install, which is the difference between a fast CI run and a slow one that needs a toolchain.

A row as delivered for itsdangerous, verbatim from a real run (project_urls and classifiers shortened — the full row is in the repository under examples/regular.json):

{
  "name": "itsdangerous",
  "version": "2.2.0",
  "summary": "Safely pass data to untrusted environments and back.",
  "license": null,
  "license_classifiers": ["OSI Approved :: BSD License"],
  "requires_python": ">=3.8",
  "requires_dist": null,
  "dependency_count": 0,
  "python_versions": [],
  "development_status": "5 - Production/Stable",
  "release_count": 27,
  "first_release_at": "2011-06-24 02:53:45",
  "last_release_at": "2024-04-16 21:28:15",
  "days_since_last_release": 904,
  "yanked": false,
  "yanked_versions": null,
  "latest_files": 2,
  "latest_size_bytes": 70644,
  "has_wheel": true,
  "url": "https://pypi.org/project/itsdangerous/",
  "position": 3,
  "fetched_at": "2026-10-07T00:28:04.031Z"
}

Two things in that row are worth reading twice: license is null while license_classifiers carries OSI Approved :: BSD License, and python_versions is empty while requires_python says >=3.8. Neither is a gap in the data — it is what the maintainer declared, and it is exactly why every one of those fields comes back.

A missing package is said, not implied

A name that does not exist on PyPI returns a not_found error naming it, rather than a row of nulls. When you are auditing a requirements file, "this package does not exist" and "this package exists but has no dependencies" are entirely different findings, and a scraper that blurs them is worse than useless.

Transport failures are separated too. A 404 is definitive and reported as not_found; a 500 or a timeout is retried with backoff and only then reported as fetch_failed. You always know which of the two you are looking at. Soft outcomes (nothing found, a missing name) finish as a successful run with an empty or partial dataset, the same as before.

Pricing

Pricing follows the same event model this Actor has always used:

Event	When it is charged	Amount
run_start	Once per run, including runs that return nothing	0.001 USD
package	One per delivered project row	0.001 USD

A run that finds nothing is charged only the start fee. A run that returns three packages is charged the start fee plus three package events. Free-plan runs receive at most 10 rows per run and are charged only for the rows they receive; upgrade to remove the cap. There are no hidden platform events in this model beyond what the marketplace itself bills.

Typical uses

• Dependency and licence audit. Feed your requirements file in, get every declared licence and dependency out, then filter on the licences your legal team cares about.
• Maintenance review. Sort your dependency tree by days_since_last_release and see what has been abandoned under you.
• Yanked-release check. Cross-reference your pinned versions against yanked_versions.
• Ecosystem research. Match a pattern like django-* or *-airflow-* and measure how a plugin ecosystem is doing: how many projects, how many still releasing, which Python versions they have moved to.

PyPI scraper vs the alternatives

Four realistic ways exist to get Python package metadata at scale. Every one of them leaves you holding something this actor already handles.

PyPI's own JSON API, by hand. pypi.org/pypi/<name>/json is the very source this actor reads, and for one package once, curl is the right tool. What you then own is everything around it: retries with backoff, a consistent set of error codes, concurrency and rate limits, normalising the nulls every maintainer left empty, and paging a long list through a job that may be cut short. That is the code this actor ships, plus a dataset you can page with cursor and a summary record that already counts what you would otherwise write a script for.

Scraping the website, or its search page. PyPI's search is an HTML page that hands a non-browser client about 3 KB, and the maintainers ask people not to scrape it. This actor never touches it: projects come from the documented JSON API, and name discovery comes from the simple index with the documented JSON accept header. Two public endpoints, one download of the index, no browser needed.

Store actors that count downloads. Download statistics live in a separate statistics service, and no row here carries a download count. If your question is "how many times is this installed", this is the wrong actor. If it is "what does it depend on, what licence did they declare, is it still releasing, and did anything get yanked", that is precisely this actor — and it is the part a download chart never shows you.

Reading your requirements file in an editor. That answers one package at a time, from the file you already have, and it never tells you that a pinned version was yanked after publication, or that the project under you stopped releasing two years ago. Feed the same file to this actor instead and you get those answers as columns you can sort, for every entry at once.

Notes

PyPI metadata is only as good as what maintainers declare. license is a free text field and is frequently empty even when license_classifiers is populated, which is why both are returned. author and home_page are increasingly left blank in favour of project_urls, so that is checked as a fallback for the homepage.

FAQ

Does this PyPI scraper need an API key?

No. Both endpoints it reads are public and key-free: the per-project JSON API and the simple index. You only need whatever credentials your own Apify or marketplace account already uses to start a run.

What is the difference between license and license_classifiers?

license is the free-text metadata field a maintainer filled in. license_classifiers are the trove classifiers under License ::. Either one can be empty while the other is populated, so both come back on every row and you can pick the one your policy trusts.

Why does a name pattern take longer than a name list?

A name list is one request per package. A pattern has to see every project name on PyPI first, because there is no search API to ask. That single index download is the price of discovering names you did not know; if you already know them, pass packages and skip it.

What happens when a package does not exist?

You get a not_found error naming the package, and no row of nulls. Other names in the same run still return. The run itself succeeds, so a single missing name does not throw away the rest of your audit.

How do I collect more than one request's worth of projects?

Set limit up to 1000 and follow the cursor from each response, repeating the original input. Rows arrive in stable name order and position numbers them across the whole job, so concatenating the pages is the same table you would have got in one go.

Is a yanked release the same as a deprecated package?

No. Yanking pulls specific release files after publication; it is not a deprecation mark on the project. That is why yanked, yanked_reason and yanked_versions stay separate fields and are not folded into a deprecation flag.

Does it return download counts?

No. Downloads are published by a separate statistics service, not by the project metadata this actor reads, so there is no download field on any row — not a zero, not a null pretending to be one. What you get instead is the supply side: how many releases exist, when the last one landed, how many dependencies were declared, and which versions were pulled.