PyPI Package Scraper

What does PyPI Package Scraper do?

PyPI Package Scraper extracts detailed metadata, version history, and download statistics from the Python Package Index (PyPI) — the official repository of over 500,000 Python packages. It uses the official PyPI JSON API and pypistats.org to deliver structured data without requiring a browser or HTML parsing.

You can search for packages by keyword, look up a specific package's full metadata and version history, or browse trending packages sorted by download volume. All data is returned as clean JSON in your Apify dataset, ready for analysis, integration, or export.

The Apify platform handles scheduling, scaling, and monitoring — no infrastructure to manage. Results are accessible via API, webhook, or downloadable in JSON, CSV, HTML, or Excel.

Why use PyPI Package Scraper?

Data-driven decision making: Evaluate which packages to use in your project by comparing download numbers, license types, Python version requirements, and maintenance recency side by side.

Dependency auditing: Automate identification of dependency chain metadata — author, repository URL, license — across your entire project for compliance or reporting.

Market intelligence: Track the PyPI ecosystem. Monitor download trends for competing packages, spot rising stars, and identify packages with stale maintenance records.

CI/CD integration: Fetch a package's latest version programmatically inside your build pipeline to alert on outdated dependencies or license changes.

Data enrichment: Enrich your own dataset with authoritative PyPI metadata — package name, summary, author, home page, repository URL, keywords.

How to use PyPI Package Scraper

1. Open the Actor — Go to the PyPI Package Scraper page on Apify Console and click Try or Run.
2. Configure input — Choose a mode and fill in the fields. You can run a search, fetch a single package, or get trending packages.
3. Run the Actor — Hit the start button. The scraper fetches data from PyPI in seconds using concurrent HTTP requests.
4. Get your data — Results appear in the Output tab as a table. Download in JSON, CSV, HTML, Excel, or access via the Apify API.

If you prefer to run programmatically, use the Apify API or one of the SDKs (Python, JavaScript).

Input

The Actor accepts a JSON object with the following fields:

Field	Type	Default	Description
mode	string	"search"	One of search, package, or trending
query	string	""	Keyword search term (required for search mode)
package_name	string	""	Exact PyPI package name (required for package mode)
max_results	integer	25	Maximum results (1–250)
include_downloads	boolean	true	Fetch daily download counts from pypistats.org

Example input (search mode)

{
    "mode": "search",
    "query": "async http client",
    "max_results": 10,
    "include_downloads": true
}

Example input (package mode)

{
    "mode": "package",
    "package_name": "httpx",
    "include_downloads": true
}

Example input (trending mode)

{
    "mode": "trending",
    "max_results": 20,
    "include_downloads": true
}

Output

Each result in the dataset contains these fields:

{
    "name": "httpx",
    "version": "0.28.1",
    "description": "The next generation HTTP client.",
    "author": "Tom Christie",
    "license": "BSD-3-Clause",
    "home_page": "https://www.python-httpx.org",
    "repository": "https://github.com/encode/httpx",
    "keywords": ["http", "async", "client", "rest", "api"],
    "requires_python": ">=3.8",
    "last_modified": "2024-12-15T10:30:00",
    "downloads": 1256789,
    "versions": ["0.28.0", "0.28.1"]
}

The versions array is included only in package mode. The downloads field appears only when include_downloads is true.

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel from the Apify Console.

Data fields

Field	Type	Description
name	string	Package name
version	string	Latest version string
description	string	One-line summary from setup.py/pyproject.toml
author	string	Package author
license	string	SPDX license identifier
home_page	string	Project home page URL
repository	string	Source code repository URL
keywords	array of strings	Classifier keywords
requires_python	string	Minimum Python version constraint
last_modified	string	ISO date of most recent file upload
downloads	integer	Last-day download count (optional)
versions	array of strings	All published version strings (package mode only)

Pricing / Cost estimation

The Actor uses Apify's pay-per-event (PPE) pricing model:

• 10 results are free per run, regardless of mode.
• After the first 10, each additional result costs $0.002 (one-fifth of a cent).
• The Actor start event costs $0.00005.

A typical search run returning 25 results costs approximately 15 × $0.002 = $0.03plus negligible start cost. A trending run with 50 results costs roughly 40 × $0.002 = $0.08.

Running locally with apify run does not incur any charges. Costs are only generated when running on the Apify platform.

Use Cases

Open-source maintainers

Track download trends for your own packages. Run the scraper weekly on Apify's scheduler and compare daily download counts over time. Detect drops that might indicate a packaging issue or a competing library gaining traction.

Security and compliance teams

Audit your organization's Python dependencies at scale. Run package mode for each dependency in your requirements.txt to collect license identifiers, author names, and repository URLs. Cross-reference licenses against your approved list and flag unmaintained packages (those with old last_modified dates).

Data scientists and analysts

Enrich package recommendation models with real PyPI metadata. Collect keyword tags, download volumes, and Python version constraints across hundreds of packages to identify clusters, popularity trends, and compatibility patterns. Export to CSV for analysis in pandas, R, or your BI tool of choice.

DevOps and platform engineers

Integrate into your CI/CD pipeline via the Apify API. On every deploy, check your critical dependencies' latest versions and repository URLs. Alert on packages whose latest version is more than 12 months old. Feed results into your dependency update bot (Renovate, Dependabot).

Content and documentation teams

Generate up-to-date package reference tables for tutorials, comparison articles, or internal wikis. Search mode returns summary metadata that's ready to embed in documentation. The deterministic output structure makes it easy to diff changes between runs.

Research and education

Study the Python ecosystem quantitatively. Pull metadata for hundreds of packages via trending mode and analyze license prevalence, author diversity, description vocabulary, and version velocity across the top packages. All data is structured and repeatable — perfect for longitudinal studies.

Tips and advanced options

• Maximize throughput — The Actor uses up to 10 concurrent HTTP requests. Running with max_results up to 50 usually completes in under 30 seconds.
• Minimize cost — Set include_downloads: false to skip the pypistats.org API call. This cuts execution time roughly in half and reduces the number of API roundtrips.
• Package mode for depth — Use package mode when you need the full version history. The versions array includes every release, not just the latest.
• Search pagination — PyPI search returns a single results page. For broader coverage, use a lower max_results and vary your query terms across runs.
• Retries — The Actor automatically retries failed requests with exponential backoff. Transient network errors on PyPI's side are handled transparently.

FAQ, disclaimers, and support

Is it legal to scrape PyPI? Yes. PyPI's JSON API is designed for programmatic access and has no rate-limiting restrictions for reasonable use. pypistats.org also provides a public API. Both APIs are free and require no authentication.

Can I use this for commercial purposes? Yes. All PyPI data is publicly available. The Apify platform itself requires a paid plan for production use above the free tier.

Rate limits — PyPI does not enforce strict rate limits but may throttle aggressive requests. The Actor uses a concurrency limit of 10 connections and retries with backoff to stay within reasonable bounds.

Known limitations — Search mode uses PyPI's web search results page, which returns at most 20 results per query. For exhaustive package discovery, consider iterating the simple index directly. The pypistats.org API reports download counts with a ~24-hour delay.

Need help? Open an issue on the GitHub repository for bugs or feature requests. For custom scraping solutions or enterprise support, contact the Apify team through the platform.

Disclaimer — This Actor is provided as-is. The author is not responsible for how you use the scraped data. Always respect PyPI's terms of service and applicable laws.