Github Marketplace Scraper

This Apify actor scrapes listing details from GitHub Marketplace using a search query-based approach. It extracts relevant metadata, developer details, pricing information, and more.

Features

• Crawls GitHub Marketplace search results for application listings
• Extracts application metadata such as name, description, and logo
• Retrieves developer information including name and support email
• Captures pricing plans for each listing
• Collects number of installs and category information

Use Cases

• Competitive analysis of GitHub Marketplace applications
• Market research on GitHub ecosystem apps
• Aggregating and analyzing developer and pricing data
• Monitoring changes in listings over time

How It Works

1. Search Query Crawling: The scraper loads search results from GitHub Marketplace based on a predefined query.
2. Data Extraction: For each listing, it retrieves relevant metadata using Cheerio.
3. Data Storage: Extracted data is pushed to the Apify dataset for further processing.

Input Schema

This scraper does not require user input as it automatically crawls GitHub Marketplace using a predefined search query.

Example Output

{
  "app_name": "Sample App",
  "developer_name": "Sample Developer",
  "description": "This is a sample GitHub Marketplace listing.",
  "description_long": "Detailed description of the app.",
  "logo_url": "https://example.com/logo.png",
  "number_of_installs": 5000,
  "privacy_policy": "https://example.com/privacy",
  "support_email": "support@example.com",
  "categories": [
    {"name": "DevOps"},
    {"name": "Security"}
  ],
  "plans": [
    {"name": "Basic", "price": "$10/month"},
    {"name": "Pro", "price": "$50/month"}
  ],
  "app_url": "https://github.com/marketplace/sample-app"
}

Need More Features?

If you'd like to add new data fields to this scraper or need a custom scraper for another purpose, feel free to file an issue or get in touch! We are open to customizing the scraper to suit your needs.

Why Choose This Scraper?

• Efficient: Handles large-scale scraping with robust session and concurrency management.
• Customizable: Easily tailor input options for specific scraping requirements.
• Reliable: Implements error-handling mechanisms to minimize interruptions.

Proxies and Anti-blocking

This scraper uses the Apify Proxy or your custom proxy settings to reduce the risk of being blocked. It supports automatic session management for uninterrupted scraping.

Resources

• Apify SDK Documentation
• Crawlee Documentation

Get Started

• Clone this scraper or use it directly on the Apify platform.
• Customize the input settings as needed.
• Run the scraper and export the data in JSON, CSV, or Excel format.

Unlock valuable insights from the Github Marketplace with this powerful scraper. Get started today!