Needle2 Ships 45M-Parameter Model as 14MB CQ2-Bit Binary Running at 500 Tokens/s on Raspberry Pi 5

Cactus Compute released Needle2 under Apache 2.0 with weights on Hugging Face. The model targets tool calling, device control, and structured extraction on hardware without GPUs or NPUs. It ships as a dependency-free C++ binary that runs on ESP32-S3 microcontrollers, sub-$200 Android phones, Meta Quest 3S, and Raspberry Pi 5. The release includes a grammar-constrained inference engine that enforces schema outputs and returns an empty call on low confidence.

On the google/mobile-actions split of 961 rows, Needle2 matches or exceeds FunctionGemma 270M, LFM2.5 230M, and Apple FM while using 5-70x fewer parameters at 2-bit precision instead of f16. Decode reaches 500 tokens/s on Pi 5, 400-1500 tokens/s on VR headsets, and 300-700 tokens/s on Samsung A-series devices. Session RAM stays at 28MB peak. The model was trained end-to-end against Cactus Quants rather than post-hoc quantized.

The architecture bets that device APIs already expose typed functions, so the task reduces to mapping utterances onto those functions without open-ended generation. This framing enables the 45M parameter count and lossless 2-bit training. Future deployments will likely appear first in smart-home firmware and low-cost robot controllers where escalation to cloud models occurs only below a learned threshold.

Adoption depends on integration into existing device SDKs rather than standalone downloads.