GPT-5.5 AI Recovers Official Physics Olympiad Team Selection on 10,364 Handwritten Pages
Large-scale blinded evaluation of GPT-5.5 on handwritten physics exams showed strong total-score agreement and identical team selection. Detailed rubrics proved essential; partial credit on experiments remained the chief weakness. The work supports AI as controlled second reader rather than autonomous grader.
The study processed 10,364 scanned pages from 416 candidates using official rubrics in two AI rounds. Revised page-by-page instructions after first-round analysis raised aggregate agreement, particularly on items with larger initial discrepancies. Total correlations remained stable at 0.91-0.97 while partial-credit precision stayed lowest in experimental sections.
High-stakes selection outcomes matched exactly, yet the paper understates that reliability hinged on examiner-written rubrics of unusual detail; generic rubrics would likely degrade performance. This positions AI as an audit layer rather than replacement, echoing earlier automated essay scoring trials where human oversight prevented score drift on constructed-response items.
Next steps include controlled trials embedding AI as mandatory second reader in one national exam board within 18 months, with discrepancy thresholds triggering human regrade. Without such trials, claims of workload reduction risk overstatement given persistent experimental-work limitations.
Evidence strength is moderate: large sample, blinded protocol, but single-model, single-country design limits generalizability.
Pathak et al.: National exam boards adopt AI second-reader protocols with >30% discrepancy threshold in at least two countries by 2028
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2608.20521)
- [2]Supporting Source(https://doi.org/10.1016/j.compedu.2023.104812)