SWE-bench Leaderboard Scraper

Collect AI coding agent benchmark results from SWE-bench leaderboards. Pick a mode, set item limits, and export clean JSON, CSV, or Excel data seamlessly.

What it does

This scraper processes SWE-bench leaderboard metrics and flattens each board's results into structured dataset rows. You can extract data across all boards or focus on specific ones like Verified, Lite, Multilingual, and others. Each record captures key performance indicators such as agent details, evaluation model, cost, resolved score percentage, and related benchmark parameters.

Modes

• All leaderboards: Extracts and flattens benchmark metrics across all boards (or a selected list of target boards) up to your configured maximum item limit.

Input example

{
  "mode": "all",
  "leaderboards": ["Verified", "Lite"],
  "maxItems": 50,
  "requestDelaySecs": 1,
  "requestTimeoutSecs": 30
}

Output example

{
  "leaderboard": "Multilingual",
  "agent": "mini-SWE-agent",
  "agent_org": "SWE-agent",
  "checked": true,
  "cost": 105.55,
  "date": "2026-02-13",
  "folder": "20260213_mini-v2.0.0a0_gemini-3-flash",
  "model_display": "Gemini 3 Flash",
  "model_org": "Google DeepMind",
  "name": "Gemini 3 Flash",
  "resolved": 72.7,
  "scraped_at": "2026-10-26T00:00:00+00:00"
}

The overview output table organizes fields including leaderboard, agent, agent_org, checked status, cost, evaluation date, folder identifier, and execution timestamps.

Pricing and usage limits

• Pay-as-you-go / Compute units: Compute usage depends on the number of leaderboards selected, your maxItems cap, and network response speed.
• Custom limits: Set maxItems in the input configuration to strictly control dataset size and resource consumption per run.
• Scheduled runs: Running small batch updates periodically consumes minimal compute resources while maintaining up-to-date benchmark tracking.

Use cases

• Benchmark tracking: Gather updated SWE-bench scores for AI coding agents and models directly into structured spreadsheets.
• Competitive analysis: Side-by-side evaluation of Lite and Verified benchmark performances across leading LLMs.
• Model monitoring: Track resolve rate improvements and compute cost efficiency trends over time.
• Data analysis: Export datasets directly to CSV or Excel formats for python analysis notebooks and visual dashboards.
• Research reporting: Keep track of structured evaluation records across various agent architectures.

FAQ

Who is this for?

Researchers, developers, and AI analysts looking to aggregate and compare software engineering agent benchmarks without manual record-keeping.

What export formats are supported?

JSON, CSV, Excel, XML, and other standard Apify export formats.

Can I run it on a schedule?

Yes. Use Apify schedules to automatically refresh your dataset on an automated hourly, daily, or weekly interval.

What happens when a data request encounters an error?

The execution handles errors gracefully by recording an error status row and continuing the rest of the extraction job.

How do I filter specific boards?

Specify desired names such as Verified or Lite in the leaderboards input list. Leaving the array empty processes all available leaderboards automatically.

Related tools & custom pipelines

If you plan to map, analyze, or join these benchmark datasets, BigiByte provides full data scraping solutions tailored to your technical requirements.

Need extra open-data extractions, automated benchmark monitors, or specialized data pipelines? BigiByte builds custom scrapers and APIs: https://bigibyte.com

Trademark Notice

SWE-bench and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the subject of evaluation and does not imply affiliation, sponsorship, endorsement, or partnership.

Disclaimer

This scraper is an independent automation tool developed by BigiByte and is not affiliated with, endorsed by, sponsored by, or officially associated with SWE-bench or its maintainers. Users are responsible for ensuring their usage complies with relevant terms, guidelines, and policies.

Is web scraping legal?

This actor extracts publicly accessible data without bypassing paywalls, or technical security layers. While collecting public web data is widely permitted, compliance depends on how data is gathered and used:

• Personal data: When collecting identifiers like emails, phone numbers, or social handles, ensure compliance with applicable data protection laws such as GDPR or CCPA.
• Copyrighted content: Public text, images, and visual assets are often protected by intellectual property rights. Avoid directly duplicating or re-distributing full creative works.
• Platform terms & rate limits: Practice responsible data extraction by avoiding aggressive traffic spikes, adhering to rate limits, and respecting published platform policies.
• Disclaimer: This summary serves informational purposes only and does not constitute formal legal counsel. Seek professional legal advice for specific compliance queries.

For a detailed guide, see Apify's legal overview: Is web scraping legal?