LLM Agents Match Published Astronomy Results by Chance, Not Reasoning Reconstruction

The arXiv study by Wang et al. introduced an end-to-end reproduction framework separating execution failures from methodological underspecification. Agents received full papers but had to infer implicit choices such as parallax zero-point corrections and sky masking. In the Astrophysical Journal case study, exhaustive enumeration of twelve predefined paths showed that the single matching result emerged only after all runs completed, without the published value serving as any optimization target. This demonstrates that outcome matching alone cannot confirm reconstruction of causal reasoning chains.

The decisive +0.02 mas parallax correction was explicitly stated in the source paper yet remained unrecognized by agents until post-hoc analysis exposed its effect size. This pattern aligns with documented LLM limitations in causal inference across scientific domains, where retrieval succeeds but relevance weighting fails. Broader literature on reproducibility, including the 2023 Nature Human Behaviour survey of 1,500 researchers, shows similar ambiguity rates in 70% of observational studies, suggesting the LLM bottleneck mirrors human specification gaps.

Future validation requires mandatory explicit dependency graphs in papers plus blinded multi-agent tournaments that penalize exhaustive search strategies. Without these, claims of AI-driven scientific discovery risk conflating surface reproduction with genuine knowledge transfer.

Deployment of such frameworks could accelerate error detection in upcoming large surveys like LSST but demands community standards for implicit-knowledge annotation within two years.