arXiv:2610.35942 derives break-even defect cost from time saved and excess escaped defects
arXiv:2610.35942 replaces stopwatch measures of AI time savings with a ledger that nets time against excess defects. Re-analysis of METR data and a new SEC-brief experiment show that over-trust produces faster but worse output once defect costs are included. The break-even defect cost supplies the missing scalar for deciding when further model improvement ceases to justify reduced checking effort.
The model-free quantities separate stopwatch time from ledger time that subtracts defects let through. In the baseline checking model, over-trust reduces observed minutes at a higher total-minute cost once defect rates are counted. Randomized studies identify only bounds; re-analysis of METR's 246 issues yields either 19% slowdown or 24% speedup depending on the estimand chosen. GDPval's review-every-output workflow limits GPT-5 speedup to 3.6x from review time alone. Consultants outside the frontier posted higher error rates yet lower elapsed time. The pre-specified SEC-brief experiment confirmed the model's switching prediction: reviewers given the true drafter error rate checked more, while those told a 10x lower rate checked 7 of 9 fewer figures and passed one wrong figure in 22.5% of cases. The paper replaces point estimates with explicit bounds on the published record. It isolates the decision threshold where additional model quality stops paying for itself in defect-adjusted time. Future deployments can test the same ledger metric against internal review logs rather than self-reported stopwatch times.
METR: within 12 months, at least two public benchmarks will report both time saved and excess escaped defects rather than stopwatch deltas alone.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2610.35942)
- [2]Supporting Source(https://metr.org/blog/2025-developer-productivity)