Audit Documents 91 Silent Failures in ToolUniverse API and Wrapper Layers
The arXiv paper audits ToolUniverse for silent failures where tool calls succeed yet return incomplete results without notification. 91 cases were confirmed, concentrated in API and wrapper layers. The work introduces contextual reliability as a measurable property for agent-tool pipelines.
The study examined API documentation and tool wrappers for failures where invocations returned without error yet omitted data or functionality. Failures clustered in missing fields, inconsistent search and ranking, and absent notifications. Manual validation followed LLM candidate detection and automated tests across seven defined loci.
Data showed upstream API and wrapper defects propagating into downstream outputs that appeared complete. Missing data accounted for the largest share, followed by search and filter inconsistencies. No mechanisms existed to surface these omissions to agents or users.
Contextual reliability requires explicit checks at each layer plus disclosure of failure modes in tool schemas. Prior agent benchmarks focused on task completion rates and overlooked silent data loss; this audit isolates the exact layers where completeness guarantees break.
Implementation of layer-specific monitors and schema-level failure annotations can reduce undetected propagation. Testing regimes must expand beyond success metrics to measure completeness at every handoff.
ToolUniverse maintainers: deployment of layer monitors reduces silent failure rate below 15 per 15 tools within 9 months.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2609.26836)
- [2]Supporting Source(https://arxiv.org/abs/2307.16789)