Software-engineering benchmark
SWE-bench Pro
Repository-level issues that require more than generating a short function.
Key information
- Focus
- Resolving software-engineering tasks in professional repositories.
- Measure
- Resolve rate on a named split and agent configuration.
- Version / scope
- SWE-bench Pro; public and commercial evaluations separate
- Update pattern
- Leaderboard updates and task fixes are release-driven; no fixed schedule was confirmed.
- Publisher update
- Not consistently disclosed
- Editorial verification
- 15 October 2026
In plain English
What makes it different?
SWE-bench Pro is Scale AI's independent benchmark for difficult repository-level work. It is not the same evaluation as SWE-bench Verified. Public, held-out and private commercial task sets serve different purposes and should not be collapsed into one comparison.
Sources and attribution are listed below.
What it measures
Resolving software-engineering tasks in professional repositories.
Strengths
- Tests changes across realistic codebases rather than isolated snippets.
- Combines outcome checks with human work on task and environment quality.
Limitations
- A passing test suite cannot capture every aspect of maintainability or product quality.
- Public task exposure and closed commercial tasks create different contamination and reproducibility constraints.
How the evaluation works
Tasks come from repository changes and are prepared with reproducible environments and human-reviewed requirements.
A submitted patch must satisfy the tests for the requested change while retaining existing behaviour. The published resolve rate depends on the selected tasks and agent setup.
Configuration matters
Separate public from commercial results. Record scaffold, model effort, token budget, attempts and evaluator revision; distinguish Pro from the original SWE-bench and its Verified subset.
Primary methods: Scale research paper ↗Original release explanation ↗
Results at the publisher
Scores stay with the original source
We have not documented the permissions needed to reproduce this leaderboard here. You can still explore its methods and visit the publisher for results.
Versions & freshness
Verified 15 October 2026The paper was introduced on 19 September 2025. The live source does not supply one common version or evaluation date for every listed run; those dates remain unknown here.
Official Scale leaderboard, research paper and public repository accessible. Leaderboard updates and task fixes are release-driven; no fixed schedule was confirmed.
Our review date is not an evaluation date. Individual run dates are not consistently available; they remain unknown here. No automatic result collection is enabled.
Publication & reuse
MIT-licensed evaluation software is documented. It does not establish a grant for Scale's hosted leaderboard results, private repositories or all third-party task code.
Repository licences, dataset/task licences and published result rights must be assessed separately.
View the scope of the reuse decision
- Result fields
- No result fields cleared
- Public result display
- not enabled
- Storage
- not enabled
- Historical copies
- not enabled
- Updates
- not enabled
- Commercial reuse
- not enabled
- Derived ratings
- No scoped grant documented; not enabled.
- Attribution
- Credit the named publisher and link the primary sources. Attribution alone does not grant reproduction rights.
This decision does not clear mixed-source scientific recommendations. Publisher content, software, datasets and model licences are separate. No partnership is implied.
Primary sources & attribution
This guide is original scor.es commentary about Scale AI Research Team's work. The links below support the identity, methods, versions and reuse review.
- Official public leaderboard and protocol ↗Checked 15 October 2026
- Scale research paper ↗Checked 15 October 2026
- Original release explanation ↗Checked 15 October 2026
- Evaluation repository licence ↗Checked 15 October 2026
Editorial edition 2026-10-15.1 · reuse review directory-rights-1. Re-check due 14 November 2026.