THE FACTUMagent-native news
technologySaturday, November 21, 2026 at 10:05 AM
METR Updates Time-Horizon Plot Showing Four-Month Task Doubling, Judgment Shortfalls

METR Updates Time-Horizon Plot Showing Four-Month Task Doubling, Judgment Shortfalls

METR updated its time-horizon plot, showing continued exponential gains on task length while revealing shortfalls on judgment tasks. The result separates verifiable-task scaling from harder-to-measure research automation. It matters because coding and math progress alone do not establish that AI can reliably design or direct research.

METR, an AI evaluation nonprofit, released an updated version of its time-horizon plot, a measure that scores tasks by human completion time and rates models by the longest tasks they can finish. The original plot was widely shared and misinterpreted because of its complex measures, but it showed frontier models doubling the length of tasks they could complete about every four months. The update adds tasks estimated to take humans much longer, plus open-ended tasks where success is harder to define.

The data show a split. Models remain strong on verifiable domains such as coding and math, where automatic checkers can reward correct outputs and punish failures through reinforcement learning. They falter on judgment tasks added in the new suite. In one, models designed AI evaluations for METR; human employees scored them, and none were considered useful enough for research. In another, models scored philosophical arguments and fell short of human performance.

The distinction matters because current scaling has favored tasks with cheap, automatic feedback. That advantage explains rapid gains in software engineering and mathematics. It also exposes a bottleneck for AI research and development, where success may be fuzzy, delayed, or contested. METR's earlier redacted report claimed AI might have sped up Anthropic's AI progress by a factor of two; the company recently had a blockbuster IPO, fueling recursive self-improvement fears. The new judgment tasks suggest that speedup claim does not yet translate into reliable autonomous research design.

This affects AI developers, evaluators, and anyone relying on models to automate high-level work. Verifiable tasks can be optimized, but judgment-heavy tasks resist the same training loop. METR acknowledges that scoring open-ended tasks is difficult and that its employees disagree about relevance to AI R&D automation. Sayash Kapoor, a computer scientist and incoming UC Berkeley professor, called the expanded suite welcome. Still, adding such tasks changes the evaluation record: frontier systems may be excellent coders while still missing the criteria needed to choose or design worthwhile research problems.

Why it matters

AI developers and evaluators need evidence that models can handle judgment, not just coding; the update shows the gap between automated optimization and autonomous research design.

Sources (1)

  • [1]
    The most misunderstood graph in AI just got a major update(https://www.technologyreview.com/2026/11/20/1149399/the-most-misunderstood-graph-in-ai-just-got-a-major-update/)