AI Signal 452
Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
Needle2 is a 45-million-parameter agentic LLM compressed to a 14 MB file that runs on low-cost phones, wearables, smart home devices and robots without a GPU or NPU.
It brings on-device AI to the vast majority of edge hardware that costs under $200, enabling private, low-latency control of devices via function calls. Developers can fine-tune the model on a workstation in minutes to a few hours and ship a binary that works across Cortex-M, x86 and WebAssembly targets. The system’s confidence-score driven edge-cloud fallback keeps most routine actions local while safely escalating uncertain requests.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
The model delivers 45MParams, 800+ tok/s Pi5 prefill, 500+ tok/s Pi5 decode, CQ2-bit compression, a 14 MB file size and 28 MB session RAM.
Deployment is a single dependency-free C++ binary that probes the CPU at startup, selects kernels, and bundles the model, tokenizer and schema-compiled byte-level grammar, running from microcontrollers to desktops with no install step.
Edge-cloud collaboration uses a learned confidence score and a schema-enforced function-call interface: high-confidence requests are executed locally, low-confidence ones trigger cloud escalation, and the grammar guarantees structured, refusal-safe outputs.
THE READ
What the cluster adds up to.
Needle2 is built around a Simple Attention Network where each block applies RMS-normalized flattening of four residual streams, an orthonormal Walsh-Hadamard transform, hashed n-gram tables for key-value retrieval, and Sinkhorn-iterated doubly-stochastic routing. Learned gates, sandwich-normed residuals and engram sites at two layers shape the computation, while decoding is constrained by a byte-level grammar generated from the declared function schemas. The model was pretrained on a proprietary 115-billion-token corpus and post-trained on 38 billion tokens with compact reasoning traces.
Instead of post-hoc quantization, Needle2 trains against Cactus Quants from pretraining through post-training, covering weights, activations and the KV cache. This lossless 2-bit approach yields a 14 MB file for the 45 million-parameter model and a 28 MB session RAM footprint, allowing execution on devices that provide only a few hundred megabytes of memory and lack dedicated AI accelerators. The approach avoids the accuracy loss typical of quantizing larger models after training.
The deliverable is a single C++ binary with no external dependencies. At startup the binary probes the host CPU, selects appropriate kernels, and loads the embedded model, tokenizer and grammar compiler. This artifact runs unchanged on Cortex-M microcontrollers, x86 laptops and WebAssembly environments, eliminating installation or download steps. Developers can fine-tune the model on a Mac or PC using the supplied repository and Python package, completing the process in minutes to a few hours to adapt the agent to a specific device’s tool vocabulary.
Interaction follows a function-call paradigm: natural language is mapped to a typed device function via a schema-driven contract, and every response includes a confidence score. When the score exceeds a developer-set threshold the action is executed locally; otherwise the system either re-asks or escalates to the cloud, keeping most routine requests private, instantaneous and offline. The Pebble Index Ring exemplifies this by running Needle2 locally to turn spoken requests into actions without any network connection.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER