Jev decision model matches GPT-5.4 on 91% of HealthBench Professional criteria at 73-294x lower cost
Jev delivers physician-comparable clinical AI judgments at orders-of-magnitude lower cost and latency than frontier LLMs. The model matches reference judges on 91 percent of HealthBench criteria and exceeds several GPT and Claude variants on agreement metrics. This establishes a practical path for scaled, auditable evaluation of clinical generative systems.
The arXiv 2610.28826 study evaluated Jev against Claude Opus 5.5, GPT-6, and six additional models on ordinal quality ratings, binary HealthBench criteria, and open-ended MedHELM tasks. Gwet's AC2 reached 0.50 on pediatric answers, exceeding GPT-6 at 0.38 and matching Claude Opus 5.5 at 0.49. On HealthBench Professional, Jev aligned with the reference GPT-5.4 judge on 91 percent of criteria while other judges ranged 91-93 percent. Latency was 7-57 times lower than frontier models.
Decision language models trained for structured output bypass the token-level generation overhead of general LLMs, enabling evaluation volumes previously gated by cost. The 0.50 AC2 score on ordinal scales indicates parity with physician raters within expected inter-rater bounds of 0.53. This pattern repeats across binary and multi-jury settings, showing that task-specific decision models compress the evaluation pipeline without measurable loss in calibration.
Operationally, institutions running continuous clinical AI monitoring can shift from sampled frontier-judge audits to exhaustive Jev runs. The 91 percent criterion match on HealthBench Professional supports substitution where budget caps previously limited coverage. Regulators and hospital systems gain a verifiable, low-latency alternative that scales to production telemetry without proportional compute growth.
Next steps include external validation on non-pediatric cohorts and integration into existing MedHELM and HealthBench pipelines by mid-2027.
Jev: integrated into at least three public clinical AI benchmarks as default judge within 18 months, displacing 30 percent of GPT-5.4 calls on those platforms
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2610.28826)
- [2]Supporting Source(https://arxiv.org/abs/2406.09900)