StoreBench: DeepSeek-V4-Pro reaches 49% pass rate on live-commerce tasks versus 97% scripted heuristic
StoreBench measures LLM agents on long-horizon commerce operations against calibrated heuristics and human baselines. Top model reaches 49% pass rate; GRPO training yields rapid gains on held-out tasks. Environment enforces production constraints absent from prior static suites.
The arXiv paper introduces a production-grade commerce backend where agents control 29 merchant tools under fixed action budgets. Customers, suppliers, and market shocks operate continuously. Episodes replay identically from action sequences, with rewards hardened against known hacks and thresholds calibrated to scripted baselines. Eleven scenarios run over three world seeds at matched reasoning effort.
Data show no model exceeds the heuristic anchor. Human experts using identical tools record a mean composite score of 0.708 versus the best model's 0.700. Under a full simulated year with the Claude Code harness, several models exhibit large gains. GRPO post-training on Qwen3.5-27B using five disjoint tasks lifts held-out composite from 0.136 to 0.373.
Prior static agent benchmarks such as WebArena and ToolBench lack continuous economic feedback and latency-independent timing. StoreBench enforces both. The withheld full suite and training split release limit contamination while exposing verification tooling.
Next evaluations will track whether post-training runs close the gap to scripted policies above 70% mean pass rate within twelve months.
DeepSeek-V4-Pro: mean pass rate on StoreBench held-out scenarios exceeds 65% by October 2027 under matched compute
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2610.10942)
- [2]Supporting Source(https://arxiv.org/abs/2307.13854)