Vision-Language Models Echo Human Consensus on Ultra-Faint Dwarfs Yet Expose Limits in Scientific Uncertainty

A June 2026 arXiv preprint (not yet peer-reviewed) tests whether zero-shot vision-language models can replicate human identification of ultra-faint dwarf galaxy candidates from multi-panel diagnostic images drawn from wide-field surveys. The study pits model outputs against aggregated labels produced by a large-scale citizen science campaign, finding that VLMs track overall human calibration on clearer cases while diverging sharply on individual ambiguous examples. Uncertainty quantification via self-reported scores or repeated sampling proved unreliable, a limitation that echoes earlier findings in medical imaging VLMs where calibration fails under domain shift. This pattern suggests that current models capture statistical regularities in training data rather than the causal reasoning astronomers apply when weighing low-surface-brightness features against artifacts. Related work on LSST precursor data (Drlica-Wagner et al., 2024, ApJ) and GPT-4V evaluations in exoplanet vetting (Morgan et al., 2025, AJ) indicates the same gap: aggregate agreement masks instance-level fragility that could flood future discovery pipelines with false positives. The preprint therefore reframes the question from replacement to complementarity, highlighting the need for hybrid workflows that route edge cases to human experts.