CSE benchmark shows all-constraint success falls to 5.7% at k=8 despite 41% per-constraint rate across 15 models
The CSE study documents a sharp phase transition in compositional constraint satisfaction for current LLMs. Individual rule following remains viable but joint adherence becomes unreliable beyond five or six rules due to independent failure accumulation. This imposes hard limits on simultaneous instruction complexity without new mechanisms.
The arXiv paper introduces CSE, a verifier-driven benchmark applying 36 constraint types across k=1-12 with 369753 deterministic checks and zero LLM judges. Per-constraint accuracy declines linearly while joint success decays multiplicatively because failures remain nearly independent except when constraints share output features such as sentence count. Structural constraints requiring sustained state tracking lose capability twice as fast as binary lexical checks.
This pattern aligns with earlier observations in multi-step reasoning traces where intermediate state errors propagate without recovery. The comprehension-maintenance gap identified here explains why models that pass isolated safety or format rules still violate them under composition, a regime closer to production deployments than single-constraint evals.
Operational consequence is that prompt engineering alone cannot scale to complex policy stacks; systems requiring more than five concurrent rules need architectural changes such as explicit constraint solvers or staged verification rather than end-to-end generation.
Future mitigation attempts will be measurable against the same CSE curves, which already place the strongest model below 50% probe success at seven constraints.
Anthropic: multi-constraint RL fine-tuning will lift k=10 joint success above 25% on CSE within 24 months.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2608.12426)
- [2]Supporting Source(https://arxiv.org/abs/2305.04388)