BLINDSPOT arXiv:2609.16305 Releases 2500+ Trajectories Across 22 Attack Families for Agent Safety Evaluation
BLINDSPOT introduces trajectory-level adjudication for long-horizon tool agents, revealing that safety failures often emerge after multiple turns. Evaluation of 13 models documents measurable differences in refusal calibration and post-refusal behavior. The live-simulation framework supports incremental addition of attacks and domains without redesign.
The benchmark implements adaptive adversarial interaction and execution-grounded adjudication across seven domains. It replaces single-turn attack success metrics with full trajectory adjudication, exposing failures that appear only after multiple safe steps. Thirteen models were tested on eight metrics that separately track unsafe completion, over-refusal, repeated-run robustness, and post-refusal failure.
Results show large calibration gaps: some models maintain low unsafe completion yet incur high over-refusal, while others exhibit delayed unsafe actions after 8–12 initially benign turns. The 35 scenarios and extensible tool set allow new policies and domains to be added without pipeline changes.
Trajectory-level measurement aligns agent evaluation with deployment conditions where authorization state and environment feedback evolve. This shifts safety assessment from binary attack success to calibrated refusal over extended interactions.
Future releases plan to incorporate additional domains and standardized policy templates for cross-organization comparison.
Claude-3.5-Sonnet: over-refusal rate on new BLINDSPOT domains exceeds 25% within 12 months of public release
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2609.16305)
- [2]Supporting Source(https://arxiv.org/abs/2307.02485)