ChatGPT Misreads Equations and Graphs in 40% of Physics Lab Reports

The study examined successive GPT versions on real laboratory reports submitted in undergraduate physics courses. Researchers tested direct PDF uploads, text extraction pipelines, and conversational follow-up queries. Discrepancies persisted across model generations whenever visual or symbolic content was involved, even as prose-based feedback improved.

Document-processing limitations proved more decisive than raw model intelligence. When equations were rendered as images or graphs lacked alt-text descriptions, the AI either hallucinated values or skipped key data points entirely. The authors note that systematic batch processing of an entire class set still required manual teacher intervention to correct these retrieval errors before feedback could be trusted.

These findings align with broader patterns seen in recent physics-education studies on automated grading and with computer-vision benchmarks showing current multimodal models remain brittle on scientific diagrams. The work underscores that AI functions best as an organizational scaffold rather than an autonomous assessor, surfacing recurring student difficulties for instructor review.

Future deployment will hinge on improved native handling of scientific notation and figures plus clearer teacher protocols for when to override AI output. Departments piloting such tools should budget for hybrid workflows rather than full automation.