Metrics Reveal Distinct Exploration-Exploitation Failures in Frontier LM Agents
New policy-agnostic metrics distinguish exploration and exploitation errors in LM agents; SOTA models exhibit unique failure modes that improve markedly with reasoning and harness tweaks, offering a foundational diagnostic for agent reliability.
A new paper introduces controllable partially observable grid environments paired with unknown task DAGs to systematically quantify exploration and exploitation errors from agent actions alone (Park et al., arXiv:2604.13151).
Frontier LM agents including GPT-4o, Claude-3.5-Sonnet and o1-preview all underperform on the benchmark, displaying model-specific failure modes; reasoning models solve tasks more effectively and both error types drop sharply with minimal prompt harness engineering, confirming the authors' claim that these errors are measurable and mitigable without internal policy access. Prior benchmarks such as WebArena measured only final success rates and missed this decomposition (Zhou et al., arXiv:2309.07864), while the ReAct framework provided interleaving of reasoning and action yet supplied no quantitative diagnostics for when agents explore insufficiently versus exploit known information poorly (Yao et al., arXiv:2210.03629).
Coverage of LM agent reliability has focused on aggregate accuracy while under-analyzing the foundational exploration-exploitation tradeoff now made observable; the metric therefore supplies a missing diagnostic layer for autonomous coding agents that loop in local optima and embodied systems that overlook unseen map cells, patterns repeatedly observed yet unquantified in SWE-bench and robotics evaluations.
o1-preview: Reasoning models cut both exploration and exploitation errors by better chaining partial observations to the task DAG, showing that test-time compute and structured harnesses can diagnose and correct failures missed by success-rate-only benchmarks.
Sources (3)
- [1]Exploration and Exploitation Errors Are Measurable for Language Model Agents(https://arxiv.org/abs/2604.13151)
- [2]ReAct: Synergizing Reasoning and Acting in Language Models(https://arxiv.org/abs/2210.03629)
- [3]WebArena: A Realistic Web Environment for Building Autonomous Agents(https://arxiv.org/abs/2309.07864)