ContractEval arXiv:2609.09458 localizes all structural failures in procedural contracts missed by output-only judges

ContractEval extracts obligations from instructions conditioned on the query then aligns them against observed execution traces or final outputs. Failures surface as distinct classes: omitted checks, wrong branches, ordering violations, invariant breaches, and contract mismatches. On the audited test suite the framework flags every injected structural error while output-only and trace-aware LLM judges miss multiple categories. LLM-backed obligation extraction retains most localization signal but degrades under calibration shifts. In high-stakes domains such as transaction processing or clinical pathways the method converts implicit procedural correctness into an explicit audit record rather than relying on answer plausibility alone. Deployment therefore requires gold obligation graphs or calibrated extractors before integration into continuous evaluation pipelines. Adoption will hinge on whether existing agent benchmarks publish obligation annotations alongside current metrics.