LLM Tools Recovered 57 of 102 Articles in Spanish Plume Benchmark, Misclassified 40-79% of Substantial Mentions

The benchmark used a published review of the Spanish plume weather pattern and its 102-article corpus. Five tools together surfaced 57 articles in initial searches. Codex reached 42 without access and 82 with institutional library credentials. Claude Code rose from 15 to 79 under the same condition. When supplied all 102 PDFs, three tools correctly flagged the 18 articles with substantial Spanish plume content yet labeled between 34 and 66 of the remaining 84 articles as substantial.

Domain-expert blind review of the generated syntheses found largely accurate statements accompanied by limited insight, unsupported claims, quotation errors, and overstated consensus. All three systems proposed future research directions wider than the source literature, with two overlapping questions already posed in the benchmark review. No tool led across discovery, classification, synthesis, and research-proposal tasks.

These results expose a consistent gap between retrieval scale and evidence discrimination. Institutional access improves coverage but does not correct classification or synthesis failures. The pattern matches earlier observations in medical and legal retrieval benchmarks where surface-level relevance scores outpace verification of claim strength.

Operational deployment therefore requires human verification checkpoints at the classification and synthesis stages rather than end-to-end automation. Eight recommendations in the paper specify access configuration, prompt constraints, and output auditing steps that measurably reduce unsupported assertions.