SemPlan Benchmark Shows 25.67% Peak Correctness for Structured Semantic Planning
SemPlan benchmark data indicate that adding structured semantic planning changes failure modes and efficiency metrics without delivering large correctness gains. Absolute performance stayed below 26% across all tested architectures. Operational selection therefore depends on whether policy compliance or cost dominates the deployment objective.
SemPlan introduces a deterministic synthetic bilingual benchmark of 1,800 English and Brazilian Portuguese cases to test natural-language interfaces against enterprise constraints including invalid queries, policy violations, cost, and nondeterminism. Four fixed-model architectures were evaluated: direct SQL generation, a bounded tool-agent baseline, structured semantic-request generation with deterministic planning, and a stateful clarification variant. The design isolates structural constraints from model variance on a pre-specified evaluation split.
Answer correctness remained low in absolute terms. A3 posted the highest rate at 25.67% and statistically exceeded the other three architectures in paired analysis. A1 retained the highest policy-correct rate and lowest unsafe-or-invalid rate. A4 recorded the lowest mean API cost and false-refusal rate. Repeatability on a 150-case stability subset ranged from 92.00% to 98.67%. These figures reveal a failure-mode shift rather than monotonic improvement.
Enterprise deployments face the same trade-offs observed here. Structured planning reduces certain invalid paths but does not resolve multi-turn state consistency or underspecification. Companies prioritizing governance will likely retain direct-generation baselines or hybrid controls until correctness crosses operational thresholds documented in related semantic-parsing evaluations.
Next steps include extending the benchmark to production schemas and measuring end-to-end latency under real policy engines.
SemPlan authors: A3-style planning will reach 40% correctness on an expanded enterprise schema subset within 12 months.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2608.13612)
- [2]Supporting Source(https://arxiv.org/abs/2305.15378)