Product Hunt Scraper

A web scraper that extracts comprehensive product information from Product Hunt using Apify and Crawlee.

Quick Start

1. Input: Configure your scraping parameters in the input field
2. Run: Click "Start" to begin scraping
3. Output: Download results from the Dataset tab

Input Configuration

Note: If you provide a Start Date (and/or End Date), the Start URLs field will be ignored. Only one method (date range OR Start URLs) will be used per run.

Basic Configuration

{
  "startUrls": ["https://www.producthunt.com/"],
  "maxRequestRetries": 3,
  "maxConcurrency": 5,
  "maxRequestsPerCrawl": 100,
  "scrapeComingSoon": true
}

Daily Leaderboard Scraping

{
  "startUrls": ["https://www.producthunt.com/leaderboard/daily/2025/7/5/all"],
  "scrapeDailyLeaderboard": true,
  "maxRequestsPerCrawl": 50
}

Input Parameters

Parameter	Type	Default	Description
startUrls	Array	["https://www.producthunt.com/"]	Starting URLs for scraping
maxRequestRetries	Number	3	Maximum retry attempts for failed requests
maxConcurrency	Number	5	Number of concurrent requests
maxRequestsPerCrawl	Number	100	Maximum pages to crawl
scrapeComingSoon	Boolean	true	Whether to scrape "Coming Soon" products
scrapeDailyLeaderboard	Boolean	false	Whether to scrape daily leaderboard
sortByDate	Boolean	false	Whether to sort output by date
sortOrder	String	"desc"	Sort order: "asc" (oldest first) or "desc" (newest first)
startDate	String	null	Start date for daily leaderboard range (format: YYYY-MM-DD or YYYY/MM/DD)
endDate	String	null	End date for daily leaderboard range (defaults to yesterday if not specified)

Input Method Exclusivity

• If you provide a Start Date (and/or End Date), the Start URLs field will be ignored.
• If you do not provide a Start Date, the scraper will use the Start URLs.
• If neither is provided, the scraper will default to scraping the daily leaderboard for yesterday (today - 1 day).

Examples

Default (no input, scrapes yesterday's leaderboard):

{}

// Will scrape the daily leaderboard for yesterday

Date range only (Start URLs ignored):

{
  "startDate": "2025-07-01",
  "endDate": "2025-07-05"
}

Start URLs only (date fields empty):

{
  "startUrls": [
    "https://www.producthunt.com/leaderboard/daily/2025/7/4/all"
  ]
}

Both provided (date range takes precedence):

{
  "startDate": "2025-07-01",
  "endDate": "2025-07-01",
  "startUrls": [
    "https://www.producthunt.com/leaderboard/daily/2025/7/4/all"
  ]
}
// Only 2025-07-01 will be scraped

Output Format

Regular Products

{
  "name": "Product Name",
  "tagline": "Product tagline",
  "description": "Detailed description",
  "upvotes": 1234,
  "categories": ["SaaS", "Productivity"],
  "launchDate": "Launch date",
  "imageUrl": "https://example.com/image.jpg",
  "productUrl": "https://product-website.com",
  "socialLinks": [
    {
      "platform": "twitter",
      "url": "https://twitter.com/product"
    }
  ],
  "scrapedAt": "2024-01-01T12:00:00.000Z",
  "companyWebsite": "https://company.com",
  "productHuntUrl": "https://www.producthunt.com/products/product-name",
  "makers": [
    { "username": "maker1", "name": "Maker One", "roles": ["Maker"] },
    { "username": "maker2", "name": "Maker Two", "roles": ["Maker"] }
  ],
  "hunter": { "username": "hunter1", "name": "Hunter Name" }
}

Daily Leaderboard Products

{
  "name": "Product Name",
  "tagline": "Product description",
  "categories": ["SaaS", "Productivity"],
  "upvotes": "1234",
  "launchDate": "July 5, 2025",
  "imageUrl": "https://example.com/image.jpg",
  "productUrl": "https://www.producthunt.com/products/product-name",
  "scrapedFrom": "daily-leaderboard",
  "scrapedAt": "2024-01-01T12:00:00.000Z",
  "productHuntUrl": "https://www.producthunt.com/leaderboard/daily/2025/7/5/all",
  "makers": [
    { "username": "maker1", "name": "Maker One", "roles": ["Maker"] }
  ],
  "hunter": { "username": "hunter1", "name": "Hunter Name" }
}

Team Extraction Output

• makers: Array of all team members with the "Maker" role. Each object contains:
  • username: Product Hunt username (string)
  • name: Display name (string)
  • roles: Array of roles (e.g. ["Maker"] or ["Hunter", "Maker"])
• hunter: Object with the first team member who has the "Hunter" role, with:
  • username: Product Hunt username (string)
  • name: Display name (string)
  • If no hunter is found, this field is null.

Usage Examples

Scrape Today's Products

{
  "startUrls": ["https://www.producthunt.com/"],
  "maxRequestsPerCrawl": 50
}

Scrape Daily Leaderboard

{
  "startUrls": ["https://www.producthunt.com/leaderboard/daily/2025/7/5/all"],
  "scrapeDailyLeaderboard": true
}

Scrape Coming Soon Products

{
  "startUrls": ["https://www.producthunt.com/coming-soon"],
  "scrapeComingSoon": true
}

Scrape Specific Categories

{
  "startUrls": [
    "https://www.producthunt.com/categories/developer-tools",
    "https://www.producthunt.com/categories/productivity"
  ]
}

Scrape with Date Sorting

{
  "startUrls": ["https://www.producthunt.com/"],
  "sortByDate": true,
  "sortOrder": "desc"
}

Scrape Daily Leaderboard with Date Sorting (Oldest First)

{
  "startUrls": ["https://www.producthunt.com/leaderboard/daily/2025/7/5/all"],
  "scrapeDailyLeaderboard": true,
  "sortByDate": true,
  "sortOrder": "asc"
}

Scrape Daily Leaderboard Date Range

{
  "startDate": "2025-07-01",
  "endDate": "2025-07-05",
  "scrapeDailyLeaderboard": true,
  "maxRequestsPerCrawl": 200
}

Scrape Daily Leaderboard from Date to Yesterday

{
  "startDate": "2025-07-01",
  "scrapeDailyLeaderboard": true,
  "sortByDate": true,
  "sortOrder": "desc"
}

Combine Custom URLs with Date Range

{
  "startUrls": [
    "https://www.producthunt.com/leaderboard/daily/2025/7/5/all",
    "https://www.producthunt.com/leaderboard/daily/2025/7/4/all"
  ],
  "startDate": "2025-07-01",
  "endDate": "2025-07-03",
  "scrapeDailyLeaderboard": true
}

Data Fields

Field	Description	Available For
name	Product name	All products
tagline	Short product description	All products
description	Detailed product description	Regular products
upvotes	Number of upvotes	All products
categories	Array of product categories	All products
maker	Product creator name	All products
launchDate	Product launch date	All products
imageUrl	Product image URL	All products
productUrl	Direct product website link	All products
pricing	Pricing information	Regular products
metaKeywords	Meta keywords	Regular products
socialLinks	Social media links	Regular products
scrapedFrom	Data source identifier	All products
scrapedAt	Timestamp of scraping	All products
sourceUrl	Original Product Hunt URL	All products

Performance Tips

• Concurrency: Increase maxConcurrency for faster scraping (be mindful of rate limits)
• Retries: Higher maxRequestRetries values improve reliability but slow down scraping
• Request Limits: Adjust maxRequestsPerCrawl based on your needs
• Proxy: The actor uses Camoufox proxy for anti-detection
• Sorting: Enable sortByDate to get chronologically ordered results (adds processing time for large datasets)
• Date Ranges: Large date ranges will generate many URLs; increase maxRequestsPerCrawl accordingly

Troubleshooting

Common Issues

1. No products found: Product Hunt may have changed their HTML structure
2. Rate limiting: Reduce maxConcurrency or add delays
3. Missing data: Some products may not have all fields available
4. Daily leaderboard issues: Check if the URL format is correct

Debug Mode

Enable debug logging by checking the actor logs in the Apify console.

Legal Notice

• Respect Product Hunt's robots.txt and terms of service
• Use reasonable request rates
• Use scraped data responsibly and in accordance with applicable laws
• This scraper is for educational and research purposes

Support

For issues and questions:

1. Check the troubleshooting section
2. Review the actor logs
3. Contact Apify support

---

Note: Always respect website terms of service and use data responsibly.

Related Actors

• CNN Top Headlines Scraper Actor: Scrape the latest top news headlines and full article details from CNN.