ArXiv 2610.22469: LLM agents reach 78% offline PDDL success, 40% online

The study grounds natural-language tasks in hidden PDDL models that validate action applicability, goal reachability, and plan existence. Five domains, four models, six prompting strategies, and three architectures were evaluated. Direct prompting used the fewest tokens; chain-of-thought, reflection, and critic loops consumed 5-22 times more tokens without consistent gains. Unsolvable tasks were identified in fewer than 10% of episodes. Offline success relied on full state inspection; online performance collapsed when agents acted without it.

Performance variance tracked model identity more than prompting or architecture choices. End-to-end success rates and LLM judges previously masked these gaps because they lack formal verifiers. The paper releases the domains and traces, enabling direct comparison against classical planners. Claims of agent planning competence now require validated action traces rather than task-completion percentages.

Related work on ReAct loops and Toolformer shows similar offline-online drops when state feedback is removed. Formal semantics expose where natural-language reasoning diverges from executable preconditions. Future agent benchmarks must embed verifiable models to separate prompting artifacts from genuine planning capacity.

Operational consequence: production deployments that assume reliable self-correction will encounter unrecoverable failures once external state inspection is limited.