Shopify App Store Crawler

An Apify Actor that crawls the Shopify App Store to extract app details, search rankings, category rankings, and autocomplete suggestions.

Features

• App Details: Extract comprehensive metadata from individual app pages including pricing, reviews, developer info, features, and more
• Search Rankings: Track app rankings for specific search keywords
• Category Rankings: Monitor app positions within category pages
• Autocomplete Suggestions: Capture search autocomplete suggestions for keywords

Input

The Actor accepts the following input parameters:

Parameter	Type	Description
appHandles	string[]	App handles to crawl (e.g., ["omnisend", "klaviyo"])
searchKeywords	string[]	Keywords for search ranking extraction
autoCompleteKeywords	string[]	Keywords for autocomplete suggestions
categoryUrls	string[]	Category slugs (e.g., ["store-management/inventory-management"])
trackAppsForRanking	string[]	Filter ranking results to specific apps (optional)
startUrls	object[]	Direct URLs to crawl (labels auto-detected)
maxConcurrency	integer	Max parallel requests (default: 3)
maxRequestsPerCrawl	integer	Limit total requests (0 = unlimited)
proxyConfiguration	object	Proxy settings

Example Input

{
    "appHandles": ["omnisend", "klaviyo", "mailchimp"],
    "searchKeywords": ["email marketing", "sms marketing"],
    "autoCompleteKeywords": ["email", "marketing"],
    "categoryUrls": ["store-management/inventory-management"],
    "trackAppsForRanking": ["omnisend"],
    "maxConcurrency": 3
}

Output

Results are stored in the default dataset with a type field to distinguish between output types:

App Details (type: "app")

{
    "type": "app",
    "scrapedAt": "2024-01-15T10:30:00Z",
    "appName": "Omnisend Email Marketing & SMS",
    "appHandle": "omnisend",
    "url": "https://apps.shopify.com/omnisend",
    "rating": 4.7,
    "reviewCount": 5432,
    "builtForShopify": true,
    "description": "...",
    "pricing": [...],
    "developer": {...},
    "features": [...],
    "reviews": [...]
}

Search Ranking (type: "search_ranking")

{
    "type": "search_ranking",
    "scrapedAt": "2024-01-15T10:31:00Z",
    "appHandle": "omnisend",
    "rankType": "search",
    "rankValue": "email marketing",
    "rank": 3,
    "results": [
        { "appName": "Klaviyo", "appHandle": "klaviyo", "rank": 1 },
        { "appName": "Mailchimp", "appHandle": "mailchimp", "rank": 2 },
        { "appName": "Omnisend", "appHandle": "omnisend", "rank": 3 }
    ]
}

Category Ranking (type: "category_ranking")

{
    "type": "category_ranking",
    "scrapedAt": "2024-01-15T10:32:00Z",
    "appHandle": "omnisend",
    "rankType": "category",
    "rankValue": "store-management/inventory-management",
    "rank": 5,
    "results": [...]
}

Autocomplete (type: "autocomplete")

{
    "type": "autocomplete",
    "scrapedAt": "2024-01-15T10:33:00Z",
    "keyword": "email",
    "results": [
        { "name": "email marketing", "target": "https://apps.shopify.com/search?q=email+marketing" },
        { "name": "email popup", "target": "https://apps.shopify.com/search?q=email+popup" }
    ]
}

Local Development

Prerequisites

• Node.js 18+
• pnpm

Setup

# Install dependencies
pnpm install

# Run in development mode
pnpm start

# Or use Apify CLI
apify run

Build

# Compile TypeScript
pnpm run build

# Lint code
pnpm run lint

# Format code
pnpm run format

Deploy to Apify

Option 1: Link Git Repository

1. Go to Actor creation page
2. Click Link Git Repository
3. Connect your repository

Option 2: Push from Local

# Login to Apify
apify login

# Deploy Actor
apify push

Rate Limiting

The crawler is configured with conservative rate limiting to avoid being blocked:

• Max concurrency: 3 parallel requests
• Same domain delay: 2 seconds between requests
• Max requests per minute: 30
• Automatic retry with backoff on 429 errors

Tech Stack

• Crawlee - Web scraping framework
• Playwright - Browser automation
• Apify SDK - Actor platform integration
• Zod - Schema validation
• TypeScript

Resources

• Crawlee Documentation
• Apify Platform Documentation
• PlaywrightCrawler API

License

ISC