Well-specified prompts boost o4-mini rotational mechanics solutions while MAPS reflection raises student critique alignment

The arXiv preprint by Nikhil Borse and colleagues at the University of Minnesota adapted a problem-classification framework to vary prompt specificity on a rotational-mechanics task. OpenAI’s o4-mini produced more complete solutions under well-specified prompts; underspecified and multimodal versions revealed gaps in physics reasoning and correctness that the MAPS rubric quantified. Student groups that first solved a parallel problem without guidance rarely flagged these flaws, while MAPS-guided groups aligned closer to expert criteria.

This work extends earlier findings on AI limitations in physics (e.g., 2023 studies showing GPT-4’s persistent algebraic errors) by demonstrating that structured critique training can mitigate uncritical acceptance. It also highlights how prompt engineering functions as an implicit benchmark for model reasoning quality, an angle largely absent from current AI-education literature focused on answer accuracy alone.

Future classroom integration will require larger samples across institutions and longitudinal tracking to determine whether MAPS-guided critique transfers to unaided problem solving or other AI tools. Without such data, claims about scalable preparation remain provisional.