OpenAI no longer uses SWE-bench Verified to measure frontier coding capabilities because models have saturated the benchmark faster than anticipated. Primary source states that top models now exceed previous performance ceilings without targeted training on the evaluation set (https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/).

The 2024 SWE-bench paper by Jimenez et al. introduced the benchmark to test LLM agents on real GitHub issues in large repositories, with initial resolution rates below 5% for early models (https://arxiv.org/abs/2310.06770). OpenAI's o1 model family and Anthropic's Claude 3.5 Sonnet, which scored 49% on Verified, illustrate the rapid gains; related coverage from Anthropic's announcement similarly noted high performance but did not address the benchmark's decreasing ability to differentiate capabilities (https://www.anthropic.com/news/claude-3-5-sonnet).

Mainstream reporting emphasized absolute score increases while missing the core implication: static benchmarks saturate within 12-18 months, as previously observed with HumanEval. This pattern reveals an overlooked gap in measuring sustained agentic coding behaviors such as iterative debugging across evolving codebases and novel architectural decisions not present in existing public datasets.