A April 2026 preprint (arXiv:2604.14188, not yet peer-reviewed) by Xingyang Yu and collaborators introduces a novel benchmark for testing large language models in highly abstract theoretical physics. Rather than simple right-or-wrong scoring, the team built an expert-curated dataset of exactly 12 questions covering core topics in quantum field theory (QFT) and string theory. They then applied a five-level grading rubric that separately evaluates statement correctness, awareness of key concepts, presence of a reasoning chain, reconstruction of tacit (unspoken) steps, and whether the model enriches the discussion beyond standard answers.

Methodology note: The small sample of 12 questions enables deep expert grading but severely limits statistical generalization; results should be viewed as exploratory rather than definitive. The authors report that frontier LLMs achieve near-ceiling performance on explicit derivations that stay inside stable conceptual frames. Performance collapses, however, when models must reconstruct omitted intermediate steps or reorganize their representational choices to satisfy global consistency constraints. The dominant failure mode is 'instability in representation selection' – the model simply picks the wrong conceptual lens for the implicit tensions in the problem.

This work goes far beyond a routine LLM benchmark. It opens a meta-scientific window onto the hidden assumptions that have powered a century of progress in fundamental physics. Theoretical physicists routinely rely on tacit knowledge – intuitions about which degrees of freedom to integrate out, which duality frame is natural, or which infinities can be safely ignored – that is rarely written down. The preprint makes these layers visible by watching exactly where LLMs break.

Synthesizing this with Google’s 2022 Minerva paper (arXiv:2206.14858), which demonstrated that LLMs can handle quantitative scientific reasoning yet still fail at multi-step conceptual planning, and with Michael Polanyi’s classic 1966 treatise 'The Tacit Dimension' (which argued that 'we can know more than we can tell'), a clearer pattern emerges. Breakthroughs such as the renormalization group in QFT or Maldacena’s AdS/CFT correspondence succeeded precisely because humans intuited the right representational shift before formal proof existed. LLMs, trained on vast explicit literature, reproduce the published steps but cannot yet surface or justify the unspoken selection criteria that chose those steps.

What much early coverage of LLMs in physics has missed is this inversion: the models’ failures are not merely shortcomings; they function as diagnostic probes. By systematically cataloging where representation instability occurs, researchers can map the exact tacit boundaries that human physicists cross during discovery. Historical parallels abound – Feynman’s diagrams made tacit perturbative intuition explicit; Polchinski’s D-branes crystallized unspoken dualities. The current LLM struggles suggest we may soon use AI to stress-test long-accepted but rarely examined assumptions in the string landscape or effective field theory.

Limitations abound: the grading rubric itself contains subjective elements, the dataset is tiny and curated by a narrow group of experts, and contemporary models may improve rapidly. Nevertheless, the preprint correctly identifies highly abstract theoretical physics as an unusually sensitive litmus test for epistemic evaluation paradigms. If AI can be trained to articulate the previously unspoken selection rules, it could accelerate discovery by making explicit the very intuitions that have driven progress for decades. This is less about whether machines will replace theorists and more about whether they can illuminate the hidden scaffolding on which our deepest physical theories still rest.