TECH Signal 383
Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
This demonstrates a practical path to running very large MoE models on consumer hardware that could never hold them in memory, by exploiting the fact that only about 3B of the 80B parameters are active per token. The trade-off is explicit: these models converse like large ones but recall facts like small ones, which is a useful constraint for engineers choosing deployment targets.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Swiftlet keeps only dense weights (attention, routers, shared experts, embeddings) resident in memory and fetches routed experts via single pread calls from repacked fixed-stride containers on SSD, avoiding mmap page-cache thrash.
Seventy-five percent of layers use Gated DeltaNet linear attention with a fixed-size recurrent state, eliminating the growing KV cache that constrains long-context inference in standard transformers.
The runtime ships as a Swift package, CLI, OpenAI-compatible loopback server, and is embedded in the Priv AI iOS app, requiring only Apple Silicon with macOS 14+ or iOS 17+ and sufficient SSD space for the model container.
THE CLUSTER