Astro2Method Shows LLMs Hit 90% on Easy Astronomy ML Choices but Stall at 60% on Hard Inference Tasks

The benchmark comprises 500 expert-reviewed binary questions drawn from papers published between January 2024 and June 2026. Easy items supplied full performance comparisons; Hard items supplied only partial justification, forcing models to infer why one algorithm outperformed another. Direct prompting already saturated Easy performance, yet no retrieval-augmented or agentic method (ReAct, Self-Ask, CRITIC) produced consistent gains on Hard items unless the exact source paper was retrieved.

Web-search agents improved only when they surfaced the original paper; otherwise accuracy remained flat. Supplying the masked pipeline description from the source paper raised Hard accuracy to levels matching or exceeding web search, indicating that experimental detail rather than broader domain knowledge drives success. This pattern reveals that current LLMs largely memorize surface associations rather than reconstruct causal method-selection logic from related literature.

The study underscores a persistent gap between pattern matching and genuine scientific reasoning. Earlier benchmarks such as SciBench and GPQA tested factual recall or simple multiple-choice questions; Astro2Method isolates the harder step of choosing among competing methods when ground-truth comparisons are incomplete. Results align with concurrent findings that retrieval helps only when the retrieved text contains the decisive experimental contrast.

Future progress will require benchmarks that penalize exact-paper retrieval and reward inference across multiple studies. Larger context windows or astronomy-specific fine-tuning may close part of the gap, but the data suggest that architectural advances in causal reasoning will be necessary before LLMs can reliably guide method selection without human oversight.