Indie Game Market Radar

A powerful, multi-platform web scraper that extracts comprehensive metadata from indie game marketplaces including itch.io, Steam, GOG, GameJolt, and Epic Games Store. Perfect for market research, trend analysis, price tracking, and building indie game discovery tools.

🎮 Features

• Multi-Platform Support: Automatically detects and scrapes from multiple indie game platforms

  • itch.io
  • Steam (indie games)
  • GOG (indie section)
  • GameJolt
  • Epic Games Store (indie games)

• Rich Metadata Extraction:

  • Game title and URL
  • Developer/Publisher information
  • Price (including currency detection)
  • Ratings and reviews count
  • Tags and genres
  • Release date
  • Game description
  • Platform detection
  • Screenshot URLs

• Smart Filtering: Filter games by price range, tags, platform, and more

• Intelligent Crawling: Automatically discovers game pages and listing pages

• Data Normalization: Consistent data structure across all platforms

• Proxy Support: Built-in proxy rotation to prevent blocking

🚀 Quick Start

Input

The Actor accepts the following input parameters:

{
  "startUrls": [
    {
      "url": "https://itch.io/games/tag-indie"
    }
  ],
  "maxRequestsPerCrawl": 100,
  "extractOnlyGamePages": false,
  "minPrice": null,
  "maxPrice": null,
  "platforms": ["all"],
  "tags": []
}

Input Parameters

• startUrls (array, required): List of URLs to start crawling from. Can include:
  • Game listing pages (e.g., https://itch.io/games/tag-indie)
  • Individual game pages (e.g., https://store.steampowered.com/app/123456)
  • Search result pages
• maxRequestsPerCrawl (integer, default: 100): Maximum number of pages to crawl
• extractOnlyGamePages (boolean, default: false): If true, only extracts data from game detail pages, skipping listing pages
• minPrice (number, optional): Minimum price filter (in USD)
• maxPrice (number, optional): Maximum price filter (in USD)
• platforms (array, default: ["all"]): Filter by platform. Options: ["itch.io", "steam", "gog", "gamejolt", "epic", "all"]
• tags (array, optional): Filter games by tags (platform-specific)

Output

The Actor outputs structured data to the Apify dataset. Each game entry includes:

{
  "title": "Hollow Knight",
  "url": "https://store.steampowered.com/app/367520",
  "platform": "steam",
  "developer": "Team Cherry",
  "publisher": "Team Cherry",
  "price": 14.99,
  "currency": "USD",
  "originalPrice": "$14.99",
  "discount": null,
  "rating": 4.8,
  "reviewsCount": 125000,
  "tags": ["Metroidvania", "Indie", "Action", "Platformer"],
  "genres": ["Action", "Adventure"],
  "releaseDate": "2017-02-24",
  "description": "Forge your own path in Hollow Knight! An epic action adventure...",
  "screenshots": [
    "https://cdn.akamai.steamstatic.com/steam/apps/367520/ss_123.jpg"
  ],
  "thumbnail": "https://cdn.akamai.steamstatic.com/steam/apps/367520/header.jpg"
}

📊 Use Cases

1. Market Research: Track indie game trends, pricing strategies, and market saturation
2. Competitive Analysis: Monitor competitor games and their metadata
3. Price Tracking: Build price history databases for indie games
4. Discovery Tools: Create indie game recommendation engines
5. Data Aggregation: Build comprehensive indie game databases
6. Trend Analysis: Identify emerging genres, tags, and pricing patterns

🛠️ Installation & Usage

Local Development

# Install dependencies
npm install

# Run the Actor
npm start

# Format code
npm run format

# Lint code
npm run lint

Apify Platform

1. Push your code to Apify platform
2. Configure input in the Apify Console
3. Run the Actor
4. Download results from the Dataset

📋 Supported Platforms

itch.io

• Extracts: title, developer, price, tags, description, ratings
• Supports: game pages, tag pages, search results

Steam

• Extracts: title, developer, publisher, price, tags, genres, release date, ratings, reviews
• Supports: app pages, indie game listings, search results

GOG

• Extracts: title, developer, price, tags, description, release date
• Supports: game pages, indie game listings

GameJolt

• Extracts: title, developer, tags, description, ratings
• Supports: game pages, tag pages

Epic Games Store

• Extracts: title, developer, price, tags, description
• Supports: game pages, indie game listings

🔧 Configuration

Proxy Configuration

The Actor automatically uses Apify's proxy configuration. No additional setup required.

Rate Limiting

The Actor respects robots.txt and includes built-in rate limiting to avoid overwhelming target sites.

📈 Output Schema

The dataset includes the following fields:

Field	Type	Description
title	string	Game title
url	string	Game page URL
platform	string	Detected platform (itch.io, steam, gog, etc.)
developer	string	Developer name
publisher	string	Publisher name (if different from developer)
price	number	Price in USD
currency	string	Currency code (USD, EUR, etc.)
originalPrice	string	Original price string from page
discount	number	Discount percentage (if on sale)
rating	number	Average rating (0-5 scale)
reviewsCount	number	Number of reviews
tags	array	Game tags
genres	array	Game genres
releaseDate	string	Release date (ISO format)
description	string	Game description
screenshots	array	Array of screenshot URLs
thumbnail	string	Thumbnail image URL

🤝 Contributing

This Actor is part of the Apify $1M Challenge. Contributions and improvements are welcome!

📝 License

ISC

🔗 Resources

• Apify Documentation
• Crawlee Documentation
• Apify Store
• Apify Challenge

💡 Tips

• Start with listing pages to discover many games quickly
• Use extractOnlyGamePages: true if you only want detailed game data
• Filter by platform if you're only interested in specific marketplaces
• Adjust maxRequestsPerCrawl based on your needs and rate limits

🐛 Troubleshooting

Issue: No games extracted

• Solution: Check that your start URLs are valid game listing or game detail pages

Issue: Missing metadata fields

• Solution: Some platforms may not have all fields available. The Actor extracts what's available.

Issue: Rate limiting

• Solution: Reduce maxRequestsPerCrawl or use Apify's proxy rotation

📊 Example Output

View the extracted data in the Apify Console's Dataset tab. The data is also available via the Apify API for integration into your applications.

---

Built with ❤️ for the indie game community