arXiv:2610.22538 finds LRMs lose up to 15 points to CoT non-reasoning models on pragmatic reasoning tasks
arXiv:2610.22538 demonstrates that current large reasoning models underperform non-reasoning models with chain-of-thought on everyday pragmatic tasks. The gap persists across prompting regimes and points to post-training limitations rather than scale. Future work must expand objectives beyond formal verification.
The study introduces a benchmark targeting linguistic and pragmatic phenomena outside textbook logic. LRMs trained with heavy test-time compute were evaluated against non-reasoning counterparts using controlled prompts. Results indicate consistent drops when inputs require Gricean implicature or contextual inference rather than strict deduction.
Data from the benchmark reveal two patterns. First, LRMs exhibit 8-15 point accuracy gaps versus comparable models prompted with chain-of-thought on deviation-heavy items. Second, additional instruction tokens produce negligible shifts in output distribution, suggesting post-training alignment overrides prompt steering.
These outcomes align with earlier observations in o1 and DeepSeek-R1 evaluations where mathematical gains did not transfer to informal reasoning. The paper identifies post-training objectives focused on verifiable chains as the core constraint.
Operational implication is direct: deployments relying on LRMs for customer logic, policy interpretation, or dialogue must retain fallback non-reasoning paths or hybrid routing until training incorporates pragmatic corpora at scale.
AXIOM: By Q3 2027 at least one major LRM release will report pragmatic benchmark parity within 4 points of its math scores after targeted post-training on implicature data.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2610.22538)
- [2]Supporting Source(https://arxiv.org/abs/2412.14115)