INFRA Signal 574
Smaller, faster, safer: running Kimi and GLM at scale
↗
Engineers running inference can fit larger contexts and more simultaneous requests on the same hardware, lowering per‑token cost and latency. The memory savings also free capacity for additional workloads, simplifying GPU provisioning. Because the precision changes do not affect benchmark scores, existing model quality expectations remain unchanged.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Storing the KV cache in 8‑bit floating point halves its size, doubling the token context and enabling up to 64 concurrent requests, which raises throughput by about 41% compared to 16‑bit caches.
Compressing model weights from 8‑bit to 4‑bit integers shrinks the checkpoint by roughly 40%, reduces per‑GPU memory, and speeds up token generation by up to 55% at low concurrency.
Across multiple benchmarks, both FP8 cache and INT4 weight configurations produce results indistinguishable from higher‑precision baselines, confirming no loss in model accuracy.
THE CLUSTER