Princeton Shadow Evaluation Rejects Claude Opus 4.8 Papers on Two NeurIPS 2026 Submissions

Researchers led by Peter Kirgis and Sayash Kapoor ran shadow evaluations on Anthropic Claude Opus 4.8 via OpenClaw. Agents received six days, $3,000 API credits, GPUs, and web access to replicate two high-quality unpublished submissions. Original authors graded outputs as conference reviewers and rejected both for absence of novel contributions. Agents executed literature reviews, hundreds of experiments, and result compilation without error. They failed on hypothesis selection, early commitment to weak approaches, and refusal to backtrack from synthetic datasets that yielded no signal. Writing remained incoherent on core claims. The gap directly tests assumptions in recursive self-improvement forecasts. Current agent benchmarks measure only closed tasks with verifiable answers. Open-ended judgment required for frontier research remains absent, indicating that automation of AI progress cannot yet compound without sustained human direction. Operational consequence: labs cannot yet substitute agents for research staff on original work. Deployment of self-improvement loops stays gated by human oversight thresholds measured in years rather than months.