Kalshi Market Data Scraper

This Actor scrapes comprehensive market data from the Kalshi API including:

• Events (prediction market categories)
• Markets (individual betting markets with enhanced price analytics)
• Orderbooks (current bid/ask prices and liquidity depth)
• Recent trades (for historical price calculations)

Enhanced Price Analytics

The scraper now calculates comprehensive price metrics to match professional trading datasets:

Core Price Data

• last_price - Current market price
• yes_bid, yes_ask, no_bid, no_ask - Current orderbook prices
• bid_ask_spread - Difference between best bid and ask

Historical Price Analysis

• prev_day_price - Price 24 hours ago
• prev_hour_price - Price 1 hour ago
• prev_week_price - Price 7 days ago
• day_change - Absolute price change over 24 hours
• hour_change - Absolute price change over 1 hour
• day_change_pct - Percentage price change over 24 hours
• hour_change_pct - Percentage price change over 1 hour

Volume & Liquidity Metrics

• volume - Total contract volume
• volume_24h - Trading volume in last 24 hours
• recent_volume - Trading volume in last hour
• dollar_volume - Total volume in dollar terms
• dollar_recent_volume - Recent volume in dollar terms
• dollar_open_interest - Open interest in dollar terms
• open_interest - Total open contracts
• liquidity - Total orderbook depth

Market Mechanics

• settlement_timer_seconds - Time until settlement
• can_close_early - Whether market can close before expiration
• close_unconfirmed - Early closure status
• risk_limit_cents - Position risk limits
• min_tick_size - Minimum price increment
• mutually_exclusive - Whether market is mutually exclusive
• settle_details - Settlement mechanism details
• description_context - Additional market context

Configuration

Input Parameters

{
  "environment": "production",
  "includeEvents": true,
  "includeMarkets": true,
  "includeOrderbooks": true,
  "includeTrades": true,
  "maxEvents": 0,
  "maxMarketsPerEvent": 0,
  "enhancedPriceMetrics": true,
  "eventStatusFilter": "open"
}

• environment: "demo" or "production"
• includeEvents: Scrape event data (default: true)
• includeMarkets: Scrape market data (default: true)
• includeOrderbooks: Scrape orderbook data for price calculations (default: true)
• includeTrades: Scrape trade history for price analytics (default: true)
• maxEvents: Limit number of events (0 = no limit)
• maxMarketsPerEvent: Limit markets per event (0 = no limit)
• eventStatusFilter: Filter events by status - "open" (default), "closed", "settled", or "all"

Event Status Filter Options

• "open" (default): Only collect active events where trading is still happening
• "closed": Only collect events where trading has ended but results aren't final
• "settled": Only collect events with final results determined
• "all": Collect all events regardless of status

The default "open" filter is recommended for most use cases as it focuses on actively tradeable markets.

Credentials

Set up your Kalshi API credentials in Apify secure storage:

• KALSHI_API_KEY: Your Kalshi Key ID
• KALSHI_PRIVATE_KEY: Your RSA private key in PEM format

Output Format

The scraper outputs comprehensive JSON data that can be converted to CSV with all price analytics:

{
  "type": "market",
  "scraped_at": "2025-01-17T10:30:00Z",
  "market_info": {
    "ticker": "EXAMPLE-24Q1",
    "last_price": 0.52,
    "yes_bid": 0.51,
    "yes_ask": 0.53,
    "bid_ask_spread": 0.02,
    "prev_day_price": 0.48,
    "day_change": 0.04,
    "day_change_pct": 8.33,
    "volume_24h": 1250,
    "dollar_volume": 650,
    "liquidity": 5000,
    "open_interest": 2500,
    "can_close_early": true,
    "settlement_timer_seconds": 86400,
    // ... additional comprehensive fields
  },
  "orderbook": {
    "yes": [...],
    "no": [...]
  },
  "recent_trades": [...]
}

Comprehensive Data Collection

This enhanced version collects the same comprehensive data as professional trading platforms:

1. Real-time pricing with bid/ask spreads
2. Historical price analysis with change calculations
3. Volume analytics across multiple timeframes
4. Liquidity metrics from orderbook depth
5. Market mechanics and settlement details
6. Trade history for price discovery analysis

The output matches the structure of professional Kalshi datasets with 40+ data fields per market.

Usage

For comprehensive data collection, use the included input_comprehensive.json configuration which enables all enhanced features.

The scraper automatically calculates derived metrics from orderbook and trade data, providing the complete picture needed for market analysis and trading strategies.

Features

• Comprehensive Data Collection: Events, markets, orderbooks, and recent trades
• Secure Credential Management: Uses Apify secure storage for API credentials
• Environment Support: Works with both Kalshi demo and production environments
• Robust Error Handling: Automatic retries with exponential backoff
• Rate Limiting: Smart delays to respect API limits
• Statistics Tracking: Detailed progress and performance metrics
• Flexible Configuration: Customizable data collection options

Setup

1. Kalshi API Credentials

Get your API credentials from Kalshi Account Profile:

1. Go to the API Keys section
2. Create a new API key if you don't have one
3. Note down your Key ID and download the Private Key file
4. The private key should be in PEM format starting with -----BEGIN RSA PRIVATE KEY-----

2. Secure Storage Setup (Recommended)

For security, store your credentials in Apify's secure storage instead of hardcoding them:

Option A: Use the Helper Script

1. Put your credentials in config.env:

KALSHI_API_KEY=your-api-key-id
KALSHI_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
your-private-key-content-here
-----END RSA PRIVATE KEY-----"

2. Run the setup script:

$python3 setup_secure_storage.py

Option B: Manual Setup

In your Apify console or via API, set these key-value pairs:

• KALSHI_API_KEY: Your Kalshi API Key ID
• KALSHI_PRIVATE_KEY: Your RSA private key in PEM format

3. Local Development Setup

For local testing and development:

1. Clone this repository
2. Install dependencies:

$pip install -r requirements.txt

3. Set up your credentials in config.env:

# Copy the template
cp config.env.template config.env

# Edit with your credentials
nano config.env

4. Run locally:

$python3 load_from_env.py

Performance Notes

• Events + Markets only: ~2-5 minutes for 400+ events
• With Orderbooks: Significantly longer due to API rate limits
• With Trades: Additional time depending on market activity
• Rate Limiting: Built-in delays prevent API throttling

Troubleshooting

Authentication Issues

❌ Authentication failed: Invalid signature

• Verify your private key is in correct PEM format
• Ensure no extra whitespace or formatting issues
• Check that you're using the correct API Key ID

API Rate Limits

⚠️ Rate limit encountered, waiting...

• The scraper automatically handles rate limits
• Consider reducing maxEvents or maxMarketsPerEvent for faster runs
• Disable orderbooks/trades for much faster scraping

Local Development Issues

❌ No credentials found in any source!

• Make sure config.env file exists with proper credentials
• Verify the file format matches the template
• Check that environment variables are set correctly

Missing Data

⚠️ Failed to fetch orderbook for market XYZ

• Some markets may not have orderbook data available
• Network timeouts are automatically retried
• Check market status (some may be closed)

API Documentation

For more details about the Kalshi API:

• Kalshi API Documentation
• Trading API Reference

Support

If you encounter issues:

1. Check the troubleshooting section above
2. Verify your API credentials and permissions
3. Review the Apify Actor logs for detailed error messages
4. Ensure you're using the correct environment (demo vs production)

License

This project is open source. See the Apify platform terms for usage guidelines.

Changelog

v0.2.0

• Added retry logic with exponential backoff
• Enhanced error handling and recovery
• Added runtime statistics tracking
• Improved rate limiting
• Added environment variable support
• Enhanced logging and progress reporting

v0.1.0

• Initial release
• Support for events, markets, orderbooks, and trades
• RSA-based authentication
• Demo and production environment support
• Configurable data collection parameters