Nine frontier LLMs record 42.1% collective failure on 2,005-item Oncology Decision Boundary Benchmark
The ODBB benchmark reveals a shared 42.1% failure rate across nine frontier LLMs on guideline-conformant oncology decisions. Failures cluster in pathway selection rather than factual recall. Clinical deployment now requires boundary-detection architectures that route uncertain cases to clinicians.
The Oncology Decision Boundary Benchmark tested 1,586 NCCN guideline items and 419 colorectal cancer cases across nine models. A deterministic scorer mapped outputs to 14 failure modes with oncologist validation (weighted κ 0.939 and 0.790). Pooled results showed 35.7% NCCN failure and 66.4% case failure; two decisiveness-tuned models generated unsafe commitments at three-to-five times the rate of the remaining seven without accuracy gains. In 3–9% of items models identified the correct step yet withheld commitment.
Existing medical LLM benchmarks measure factual recall. ODBB isolates meta-judgment under guideline branching and uncertainty. The 42.1% shared blind spot indicates architectural limits rather than data deficits. Model scale and release timing did not close the gap, confirming that single-model deployment cannot satisfy clinical safety thresholds.
Operational deployment therefore shifts from model selection to boundary detection and clinician routing. Architectures must flag competence boundaries in real time and escalate rather than default to any single LLM output. This constraint applies equally to closed and open-weight systems evaluated here.
Future releases will require explicit competence-boundary modules before any clinical decision-support claim can be substantiated.
AXIOM: No single frontier LLM will exceed 65% pooled accuracy on ODBB v2 within 12 months absent explicit boundary-detection layers.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2608.28592)
- [2]Supporting Source(https://www.nccn.org/guidelines)