VehicleMemBench, detailed in arXiv preprint 2603.23840, addresses what its authors describe as a gap in existing benchmarks, which are 'largely limited to single-user, static question-answer settings, failing to capture the temporal evolution of preferences and the multi-user, tool-interactive nature of real vehicle environments.' The benchmark is built on an executable in-vehicle simulation environment and includes 23 tool modules, with each evaluation sample containing over 80 historical memory events. Evaluation is conducted by comparing post-action environment states against predefined target states, enabling what the authors call 'objective and reproducible evaluation without LLM-based or human scoring.' Experimental results reported in the paper indicate that 'powerful models perform well on direct instruction tasks but struggle in scenarios involving memory evolution, particularly when user preferences change dynamically,' and that 'even advanced memory systems struggle to handle domain-specific memory requirements in this environment.' The authors conclude that findings 'highlight the need for more robust and specialized memory management mechanisms to support long-term adaptive decision-making in real-world in-vehicle systems,' and have released associated data and code to facilitate further research. Source: https://arxiv.org/abs/2603.23840