AI Signal 460
FreeToken Unlocks Frontier MoE Inference on Consumer Hardware via Dynamic Co-Execution
↗
Researchers from UC Berkeley and MIT released FreeToken, an open-source inference engine that uses dynamic co-scheduling to run large Mixture-of-Experts models on consumer GPUs by overlapping PCIe weight streaming with computation.
For engineers self-hosting models, FreeToken reduces the need for datacenter-grade interconnects by splitting token computation between CPU and GPU based on real-time bandwidth. It also handles dynamic context changes from agentic workloads via semantic anchor checkpointing, avoiding full recomputation. However, it requires specific NVIDIA GPUs and may not match datacenter throughput for high-concurrency serving.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
FreeToken implements a q* policy that dynamically splits MoE token computation between CPU cores and GPU tensor cores based on interconnect throughput.
It uses fast weight format, full-layer double buffering, and an elastic memory manager to overlap PCIe weight streaming with active computation.
Benchmarks show FreeToken running a 35B model at ~39 tokens/sec on an 8GB RTX 4060 laptop and serving a 284B model on an RTX 5090.
THE CLUSTER