🏇 FEI Scraper - Equestrian Data Extraction Platform

LICENSE

Extract comprehensive equestrian data from the Fédération Equestre Internationale (FEI) database with legal compliance and hybrid extraction strategy

📋 Overview

The FEI Scraper is a sophisticated web scraping tool designed to ethically extract structured data from the official FEI database. It implements a hybrid extraction strategy that combines fragile index generation with robust data extraction, ensuring high-quality results while respecting FEI's Terms of Service.

⚖️ Legal Compliance

⚠️ IMPORTANT: Personal Non-Commercial Use Only

This scraper is designed strictly for:

• ✅ Personal research and analysis
• ✅ Educational purposes
• ✅ Academic studies
• ✅ Non-profit equestrian research

NOT PERMITTED:

• ❌ Commercial exploitation
• ❌ Data resale
• ❌ Competing services
• ❌ Bulk harvesting

🏗️ Architecture

Hybrid Extraction Strategy

Phase 1: Index Generation (Fragile)

• Generates master IDs for persons, horses, and events
• Uses sophisticated form manipulation and navigation
• Handles FEI's complex ASP.NET Web Forms architecture
• Implements proper ViewState management

Phase 2: Data Extraction (Stable)

• Extracts detailed data using generated IDs
• Leverages FEI Clear URLs (defacto API endpoints)
• Parses structured HTML content using Cheerio
• Validates and sanitizes extracted data

Core Components

• FEIIndexGenerator: Handles index generation with rate limiting
• FEIDataExtractor: Performs robust data extraction
• RateLimiter: Respects FEI's server constraints
• DataValidator: Ensures data quality and consistency

🚀 Features

Entity Coverage

• 🏃 Persons: Athletes, trainers, officials with detailed profiles
• 🐴 Horses: Competition horses with breeding and performance data
• 🏆 Events: Competitions, championships, and results
• 📊 Rankings: Longines Rankings and performance metrics

Technical Features

• Rate Limiting: Configurable delays (5-10 seconds recommended)
• Error Handling: Robust recovery from network issues
• Data Validation: Comprehensive schema validation
• Proxy Support: Apify Proxy integration
• Logging: Detailed execution logs for debugging

Supported Disciplines

• Jumping (J), Dressage (D), Eventing (E)
• Driving (DR), Endurance (END), Reining (RE)
• Vaulting (V), Para-Dressage (PD), Para-Driving (PDR)

🛠️ Installation & Setup

Prerequisites

• Node.js 22+
• Apify CLI (for deployment)
• Git

Local Development

# Clone repository
git clone https://github.com/your-username/fei-scraper.git
cd fei-scraper

# Install dependencies
npm install

# Run locally
npm start

Apify Deployment

# Install Apify CLI
npm install -g apify-cli

# Login to Apify
apify login --token "YOUR_API_TOKEN"

# Deploy to Apify
apify push

⚙️ Configuration

Input Parameters

Parameter	Type	Default	Description
mode	string	"both"	Extraction mode: "index", "data", or "both"
entityTypes	array	["persons","horses","events"]	Entities to extract
personFeiIds	array	[]	Specific Person FEI IDs (optional)
horseFeiIds	array	[]	Specific Horse FEI IDs (optional)
eventIds	array	[]	Specific Event IDs (optional)
nationalFederations	array	[]	Federation codes (optional)
disciplines	array	["J","D","E","DR","END","RE","V","PD","PDR"]	FEI disciplines
dateFrom	string	"2020-01-01"	Start date (YYYY-MM-DD)
dateTo	string	"2025-12-31"	End date (YYYY-MM-DD)
requestDelay	integer	7	Delay between requests (seconds)
maxRequestsPerCrawl	integer	1000	Maximum requests
includeMajorResults	boolean	false	Extract major championships only

Sample Configuration

{
  "mode": "data",
  "entityTypes": ["persons", "horses"],
  "requestDelay": 7,
  "maxRequestsPerCrawl": 50,
  "personFeiIds": [
    "10000837",
    "10011463",
    "10000001"
  ],
  "horseFeiIds": [
    "GBR43241",
    "FRA45550"
  ]
}

📊 Output Schema

Dataset Structure

The scraper generates structured datasets with the following views:

Overview View

• Summary statistics
• Execution metadata
• Data quality metrics

Persons View

• Personal information (name, FEI ID, nationality, gender)
• Registration details
• Competition records
• Ranking information

Horses View

• Horse details (name, FEI ID, breed, age)
• Ownership history
• Competition results
• Performance metrics

Events View

• Event information (name, date, location)
• Competition details
• Results and rankings
• Participating athletes/horses

Rankings View

• Longines Rankings data
• Performance scores
• Historical trends

Sample Output

{
  "url": "https://data.fei.org/Person/Performance.aspx?personfeiid=10000837",
  "timestamp": "2025-11-11T21:42:18.343Z",
  "entityType": "person",
  "pageTitle": "Athlete Performance",
  "personData": {
    "name": "MAHER, Ben",
    "feiId": "10000837",
    "nationality": "GBR - Great Britain",
    "gender": "Male",
    "registration": "Jumping 2025"
  },
  "performances": [
    {
      "date": "09/11/2025 15:00",
      "event": "Verona",
      "result": "ITA",
      "horse": "DALLAS VEGAS BATILLY",
      "competition": "CSI5*-W",
      "position": "7",
      "score": "4(4+0)/75.74"
    }
  ],
  "dataQuality": {
    "hasPersonName": true,
    "hasPerformances": true,
    "totalPerformances": 1
  }
}

🔒 Security

Data Protection

• No personal data storage beyond FEI public data
• Secure API token handling
• Encrypted communication (HTTPS)

Access Control

• Private Actor by default
• API token authentication
• Usage monitoring and logging

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Fédération Equestre Internationale for maintaining the comprehensive database
• Apify for the excellent scraping platform
• Contributors for improving the tool

---

Run with recommended settings

{ "mode": "data", "entityTypes": ["persons"], "requestDelay": 7, "maxRequestsPerCrawl": 50, "personFeiIds": ["10000837"] }

Your FEI Scraper is now ready to extract comprehensive equestrian data! 🏇✨