LongMedBench records 19.72 visits and 44.91 events per patient across 335 MIMIC-IV cases, exposing LLM failures on implicit temporal reasoning

LongMedBench converts MIMIC-IV admission records into time-series event streams for multi-session agent interactions. The benchmark supplies three evaluation suites that isolate fact retrieval, temporal ordering, and sequential treatment choice over an average horizon of 19.72 visits. Construction follows a documented pipeline that preserves clinical note order and lab-result timestamps without synthetic augmentation.

Experiments on the 335-patient cohort demonstrate that retrieval-augmented agents raise fact-based QA accuracy while long-horizon decision accuracy remains bounded by immediate context length. Models exhibit measurable drops when required to infer elapsed intervals from narrative cues rather than numeric fields. Decision-making scores stay below retrieval scores across all tested agent-memory configurations.

These results quantify a concrete gap between current retrieval mechanisms and the longitudinal evidence aggregation required for realistic clinical trajectories. Operational deployment of medical agents therefore requires explicit temporal state tracking and horizon-aware memory architectures before safety-critical use. Subsequent benchmarks must report per-horizon error curves rather than aggregate accuracy alone.

MIMIC-IV release 2.2 and the original LongMedBench pipeline together establish a reproducible testbed for measuring progress on this limitation.