Sherdog API - Fighter Search & Scraper

A powerful web scraper built with Crawlee and Cheerio that searches for and extracts comprehensive fighter profile data from Sherdog, the world's largest MMA database.

🥊 Features

This scraper searches for fighters based on various criteria and extracts detailed information from their Sherdog profiles.

Search Capabilities

• Search by Name - Find fighters by partial name matches
• Filter by Weight Class - Search within specific weight divisions
• Filter by Association - Find fighters from specific gyms/teams
• Pagination Support - Automatically searches multiple pages for results
• Results Limiting - Control the number of fighters to scrape

Extracted Data

• Basic Information - Name, nickname, country, association, weight class
• Fight Record - Overall record, wins/losses breakdown by method
• Physical Measurements - Height and weight in metric or imperial units
• Personal Information - Age, birth date, profile image URL
• Fight History - Complete fight-by-fight breakdown with opponent, event, method, referee, round, and time details
• Upcoming fight - If there is a upcoming fight the details about it will be provided

📋 Input Configuration

The scraper accepts the following input parameters:

{
  "searchTerm": "McGregor",
  "weightClass": "6",
  "association": "SBG Ireland",
  "resultsLimit": 50,
  "units": "metric"
}

Parameters

Parameter	Type	Required	Default	Description
searchTerm	String	Yes*	-	Fighter name to search for
weightClass	String	Yes*	-	Weight class filter (see table below)
association	String	Yes*	-	Gym/team name to filter by
resultsLimit	Number	No	10	Maximum number of fighters to scrape
units	String	No	"metric"	Measurement units: "metric" or "imperial"

*At least one of searchTerm, weightClass, or association must be provided.

Weight Class Codes

Weight Class	Code
Super Heavyweight	"1"
Heavyweight	"2"
Light Heavyweight	"3"
Middleweight	"4"
Welterweight	"5"
Lightweight	"6"
Featherweight	"7"
Bantamweight	"9"
Flyweight	"10"
Catchweight	"11"
Strawweight	"13"
Atomweight	"15"
Minimumweight	"17"

📊 Output Format

Each scraped fighter profile returns a structured JSON object:

{
  "profileUrl": "https://www.sherdog.com/fighter/Conor-McGregor-29688",
  "name": "Conor McGregor",
  "nickname": "The Notorious",
  "country": "Ireland",
  "record": "22-6-0",
  "wins": 22,
  "koWins": 19,
  "subWins": 1,
  "decWins": 2,
  "losses": 6,
  "koLosses": 1,
  "subLosses": 4,
  "decLosses": 1,
  "draws": 0,
  "age": 35,
  "birthDate": "1988-07-14",
  "height": "175 cm",
  "weight": "70 kg",
  "association": "SBG Ireland",
  "weightClass": "Lightweight",
  "fighterImageUrl": "https://sherdog.com/image_crop/200/300/_images/fighter/20140714000000/Conor_McGregor.JPG",
  "fightHistory": [
    {
      "result": "win",
      "opponentName": "Donald Cerrone",
      "opponentSherdogUrl": "https://www.sherdog.com/fighter/Donald-Cerrone-12345",
      "eventName": "UFC 246 - McGregor vs. Cerrone",
      "eventSherdogUrl": "https://www.sherdog.com/events/UFC-246-McGregor-vs-Cerrone-78901",
      "eventDate": "Jan / 18 / 2020",
      "method": "TKO (Head Kick and Punches)",
      "refereeName": "Herb Dean",
      "refereeSherdogUrl": "https://www.sherdog.com/referee/Herb-Dean-5",
      "round": 1,
      "time": "0:40"
    },
    {
      "result": "loss",
      "opponentName": "Khabib Nurmagomedov",
      "opponentSherdogUrl": "https://www.sherdog.com/fighter/Khabib-Nurmagomedov-23456",
      "eventName": "UFC 229 - Khabib vs. McGregor",
      "eventSherdogUrl": "https://www.sherdog.com/events/UFC-229-Khabib-vs-McGregor-67890",
      "eventDate": "Oct / 06 / 2018",
      "method": "Submission (Rear-Naked Choke)",
      "refereeName": "Herb Dean",
      "refereeSherdogUrl": "https://www.sherdog.com/referee/Herb-Dean-5",
      "round": 4,
      "time": "3:03"
    }
  ]
}

🌐 How It Works

1. Search Phase: The scraper starts by searching Sherdog's fighter database using your criteria
2. Discovery: It finds fighter profile URLs from search results across multiple pages
3. Scraping: For each discovered fighter, it extracts detailed profile information including complete fight history
4. Output: All fighter data is saved to the dataset in structured JSON format

📊 Fight History Data

The scraper now extracts comprehensive fight-by-fight history for each fighter, including:

Fight History Fields

• Result - Win, loss, draw, or no contest
• Opponent - Name and Sherdog profile URL
• Event - Event name, URL, and date
• Method - How the fight ended (TKO, Submission, Decision, etc.)
• Referee - Referee name and profile URL (when available)
• Round - Which round the fight ended
• Time - Time within the round when the fight ended

Example Fight History Entry

{
  "result": "win",
  "opponentName": "Donald Cerrone",
  "opponentSherdogUrl": "https://www.sherdog.com/fighter/Donald-Cerrone-12345",
  "eventName": "UFC 246 - McGregor vs. Cerrone",
  "eventSherdogUrl": "https://www.sherdog.com/events/UFC-246-McGregor-vs-Cerrone-78901",
  "eventDate": "Jan / 18 / 2020",
  "method": "TKO (Head Kick and Punches)",
  "refereeName": "Herb Dean",
  "refereeSherdogUrl": "https://www.sherdog.com/referee/Herb-Dean-5",
  "round": 1,
  "time": "0:40"
}

📈 Usage Examples

Search MMA fighters by Name

{
  "searchTerm": "McGregor",
  "resultsLimit": 10,
  "units": "metric"
}

Search MMA fighters by Weight Class

{
  "weightClass": "6",
  "resultsLimit": 100,
  "units": "imperial"
}

Search MMA fighters by Association/Gym

{
  "association": "SBG Ireland",
  "resultsLimit": 50,
  "units": "metric"
}

Combined Search

{
  "searchTerm": "Silva",
  "weightClass": "4",
  "association": "Chute Boxe",
  "resultsLimit": 25,
  "units": "metric"
}

⚙️ Configuration Options

Units Configuration

• Metric: Heights in cm, weights in kg
• Imperial: Heights in ft/in, weights in lbs

Results Limiting

• Set resultsLimit to control how many fighters to scrape
• Useful for testing or when you only need a subset of results
• Leave undefined to scrape all available fighters

🔍 Search Behavior

• Partial Matching: Search terms work with partial name matches
• Case Insensitive: Searches are not case-sensitive
• Pagination: Automatically searches multiple pages until results limit is reached
• Deduplication: Each fighter profile is only scraped once

📦 Output Dataset

The scraper saves all fighter data to the default dataset. Each item contains:

• Complete fighter profile information
• Original search criteria used
• Timestamp of when the data was scraped

🚀 Performance

• Concurrent Scraping: Multiple fighter profiles are scraped simultaneously
• Proxy Support: Built-in proxy rotation for reliable scraping
• Error Handling: Graceful handling of network errors and missing data
• Rate Limiting: Respectful scraping with built-in delays