LTD Deal Aggregator

Collect and normalize lifetime deal (LTD) listings from popular marketplaces into one clean dataset.

🎯 Overview

This Apify actor scrapes lifetime deal listings from multiple sources including AppSumo, DealMirror, and StackSocial. It normalizes the data into a consistent format, supports incremental runs, and can send webhook notifications for new deals.

🚀 Features

• Multi-source scraping: AppSumo, DealMirror, StackSocial (PitchGround coming in Phase 2)
• Adapter pattern architecture: Easy to add new sources
• Smart filtering: Keywords, categories, price range, discount percentage
• Incremental mode: Only return new deals since last run
• Webhook notifications: Get notified when new deals are found
• Deduplication: Automatically removes duplicate deals
• Detailed scraping: Optional deep scraping of individual deal pages

📋 Use Cases

• Newsletter creators: Build daily LTD deal newsletters
• Affiliates: Track fresh deals with metadata for promotion
• Agencies: Scout tools for clients
• Founders: Monitor competitors and pricing trends
• Automation pipelines: Integrate with webhooks for automated workflows

🔧 Input Configuration

Source Selection

• sources (array): Select which marketplaces to scrape
  • Options: appsumo, dealmirror, stacksocial, pitchground
  • Default: ["appsumo", "dealmirror", "stacksocial"]

Deal Filtering

• mode (string): Filter deals by status

  • Options: live_now, new_arrivals, ended_recently
  • Default: live_now

• publishedLastNDays (number): Only include deals published in the last N days

  • Default: 30

• keywords (string): Include deals containing these keywords (comma-separated)

• excludeKeywords (string): Exclude deals containing these keywords (comma-separated)

• categories (array): Filter by categories

  • Options: AI, SaaS, DevTools, Marketing, Productivity, Design, Analytics, Sales, Customer Service, HR, Finance, Education, Health, Other

• minDiscountPercent (number): Minimum discount percentage

  • Default: 0

• minPrice / maxPrice (number): Price range filter

  • Defaults: 0 / 10000

• pricingType (string): Filter by pricing type

  • Options: lifetime, annual, subscription, any
  • Default: lifetime

Crawl Controls

• includeDetails (boolean): Scrape individual deal pages for detailed information

  • Default: true

• maxItems (number): Maximum number of deals to collect

  • Default: 300

• maxPages (number): Maximum number of pages to crawl per source

  • Default: 20

Monitoring & Webhooks

• incremental (boolean): Only return new deals since last run

  • Default: true

• notifyOnlyNew (boolean): Send webhook notifications only for new deals

  • Default: true

• webhookUrl (string): URL to send notifications to

• webhookSecret (string): Secret for webhook authentication

• webhookBatchSize (number): Number of deals per webhook batch

  • Default: 50

Performance Settings

• useProxy (boolean): Use proxy for requests

  • Default: false

• maxConcurrency (number): Maximum number of concurrent requests

  • Default: 10

• requestRetries (number): Number of times to retry failed requests

  • Default: 4

• timeoutSecs (number): Timeout for individual requests

  • Default: 30

• minDelayMs / maxDelayMs (number): Delay between requests

  • Defaults: 1000 / 3000

📊 Output Schema

Each deal in the dataset contains:

{
  "source": "appsumo",
  "sourceDealId": "product-name-lifetime-deal",
  "title": "Product Name",
  "tagline": "Short product description",
  "dealUrl": "https://appsumo.com/products/...",
  "imageUrl": "https://...",
  "categories": ["AI", "SaaS"],
  "pricing": {
    "listPrice": 299,
    "dealPrice": 49,
    "currency": "USD",
    "discountPercent": 84,
    "isLifetime": true
  },
  "socialProof": {
    "rating": 4.5,
    "reviewCount": 120
  },
  "availability": {
    "status": "live",
    "expiresAt": "2024-12-31T23:59:59.000Z"
  },
  "description": "Detailed product description...",
  "features": ["Feature 1", "Feature 2"],
  "dealTiers": [
    {
      "tierName": "Tier 1",
      "price": 49,
      "limits": "Up to 5 users"
    }
  ],
  "productWebsite": "https://product.com",
  "companyName": "Company Inc",
  "hash": "abc123...",
  "scrapedAt": "2024-12-27T10:00:00.000Z"
}

🏗️ Architecture

Adapter Pattern

The actor uses an adapter pattern to support multiple sources:

src/
├── main.js                    # Main orchestration logic
└── adapters/
    ├── appsumoAdapter.js      # AppSumo scraping logic
    ├── dealmirrorAdapter.js   # DealMirror scraping logic
    └── stacksocialAdapter.js  # StackSocial scraping logic

Adapter Interface

Each adapter implements:

• buildStartRequests(input) → Request[]
• parseListPage(ctx) → { deals, nextPages }
• parseDetailPage(ctx, stub) → DealNormalized
• matchesFilters(deal) → boolean

🔄 Incremental Mode

When incremental is enabled:

1. The actor loads previously seen deal hashes from state
2. Only new deals (not in previous state) are returned
3. State is saved after each run for next execution

This is perfect for scheduled runs to monitor new deals.

🔔 Webhook Notifications

Configure webhooks to receive real-time notifications:

{
  "deals": [...],
  "count": 50,
  "timestamp": "2024-12-27T10:00:00.000Z",
  "signature": "sha256_hash"
}

The signature is generated using: SHA256(JSON.stringify(payload) + webhookSecret)

📈 Example Use Cases

Daily Newsletter

{
  "sources": ["appsumo", "dealmirror", "stacksocial"],
  "mode": "new_arrivals",
  "publishedLastNDays": 1,
  "incremental": true,
  "webhookUrl": "https://your-api.com/webhook"
}

AI Tools Monitor

{
  "sources": ["appsumo", "dealmirror"],
  "keywords": "AI, artificial intelligence, machine learning",
  "categories": ["AI"],
  "minDiscountPercent": 50,
  "includeDetails": true
}

Competitor Tracking

{
  "sources": ["appsumo"],
  "keywords": "project management, productivity",
  "maxPrice": 100,
  "includeDetails": true
}

🛠️ Development

Local Setup

# Install dependencies
npm install

# Run locally
npm start

Adding a New Source

1. Create a new adapter in src/adapters/
2. Implement the adapter interface
3. Register in ADAPTER_REGISTRY in main.js
4. Add to INPUT_SCHEMA enum

💰 Pricing (Pay Per Event)

This actor uses pay-per-event (PPE) pricing for transparent, usage-based costs:

Pricing Events

• Actor Start: $0.00005 per run (~$0.05 per 1,000 runs)

  • Covers startup costs
  • Apify provides 5 seconds of free compute per run
  • Charged automatically once per run

• Deal Found: $0.002 per deal (~$2.00 per 1,000 deals)

  • Charged for each deal added to your dataset
  • Covers scraping, parsing, normalization, and deduplication
  • Only charged for unique deals (duplicates are free)

Example Costs

Deals Scraped	Actor Starts	Total Cost
100 deals	1 run	$0.20
500 deals	1 run	$1.00
1,000 deals	1 run	$2.00
5,000 deals	1 run	$10.00

Note: You only pay for successful results. Failed requests and duplicate deals are not charged.

Set Your Spending Limit

Control costs by setting a maximum spending limit per run in the Apify Console. The actor will automatically stop when your limit is reached.

📝 Roadmap

Phase 2 (Enrichment)

• PitchGround adapter
• Price history tracking
• AI-powered tagging
• Affiliate link templates

Phase 3 (Monitoring)

• Price drop detection
• Deal status changes
• Slack/Discord integrations
• Historical snapshots

🤝 Support

For issues or feature requests, please contact support or open an issue.

📄 License

ISC