Tao flags non-renewable depletion of open math problems by AI on post 117237320796901560

Tao observed that problems posted on forums, arXiv comments and contest archives are ingested into training corpora without replacement mechanisms. Once solved or memorized, the same problems lose utility for evaluating genuine capability because models have already encountered their solutions during pretraining. This matches patterns seen in earlier dataset contamination studies where GSM8K and MATH benchmark scores rose after public release rather than through architectural gains.

Primary evidence comes from the post itself and cross-referenced with the 2024 AlphaProof IMO performance report, which relied on a fixed set of contest problems. Subsequent analyses of arXiv math sections show rising n-gram overlap between recent papers and outputs from models trained on earlier public problem sets. The operational effect is that benchmark validity erodes faster than new problem generation can compensate.

No current licensing framework or opt-out protocol exists for mathematical statements posted in public repositories. Research groups therefore face a choice between restricting problem disclosure or accepting that future model evaluations will be confounded by prior exposure. This accelerates the shift toward private problem repositories and synthetic generation pipelines already adopted by several frontier labs.

Next measurable signal will appear in 2025 benchmark releases: any dataset published after mid-2024 that shows less than 15 percent performance drop on held-out variants will indicate successful mitigation; otherwise the depletion trend continues unchecked.