Seq@K protocol shifts rankings for 9 of 16 frontier models across 7 benchmarks

The paper introduces Seq@K to measure whether a model solves a task within K attempts when given natural-language feedback on prior errors. Unlike Pass@K or multi-turn agent benchmarks, the metric isolates refinement from feedback without requiring long-horizon planning. Authors evaluate 16 models on seven existing benchmarks and report that optimal model selections change, Pareto frontiers admit cheaper configurations, and performance gaps widen or shrink depending on the protocol.

Data show model trajectories under Seq@K expose distinct behaviors: some models correct issues raised in feedback at rates above 40 percent while others retain original answers in over 70 percent of cases. Feedback specificity alters outcomes; generic error messages produce smaller gains than targeted critiques. These patterns appear consistently across coding, math, and reasoning suites.

Prior Pass@K leaderboards therefore understate requirements for agentic loops that rely on iterative correction. The protocol reveals that efficiency gains from model choice can increase by up to 18 percent when feedback is modeled explicitly. Practitioners must therefore benchmark the full feedback loop rather than isolated first-attempt accuracy.

Future leaderboards that omit Seq@K will mis-rank models for deployment in conversational agents and tool-use pipelines where sequential refinement is the dominant interaction mode.