Sutton 2024 Post Flags One-Step Model Error Accumulation in RL World Models

Sutton identifies the one-step trap as the assumption that accurate short-term dynamics suffice for long-term prediction via rollout. His cited works quantify the failure: compounding discrepancies in stochastic environments turn small per-step MSE into divergent trajectories after 10-20 steps. The 1999 AI journal paper on options and the 2011 AAMAS Horde architecture already demonstrated that temporally extended predictions via GVFs reduce this divergence on real sensorimotor data. The trap connects to evaluation failures where short-horizon benchmarks like Atari 100k or MuJoCo 1k-step rollouts mask long-term model collapse. Incremental progress in one-step predictors appears strong on these metrics yet fails when deployed in option-based or multi-step planning loops, a pattern repeated across POMDP solvers and model-based control papers since the 1990s. Operationally this implies research groups must replace pure one-step world models with option-conditioned GVFs before scaling planning depth. Without the shift, compute budgets allocated to longer rollouts deliver diminishing returns once horizon exceeds the model's verified accuracy length. Next steps include integrating the 2023 reward-respecting subtasks framework into existing model-based agents to test whether abstract models close the gap on tasks requiring 50-plus step foresight.