TECH Signal 414 2 feeds carried it
Smaller, faster, safer: running Kimi and GLM at scale
↗
Cloudflare reduces GPU memory use for large models by quantizing KV caches to FP8 and compressing weights to INT4, keeping accuracy while boosting throughput and lowering cost.
Engineers can serve longer contexts and more concurrent requests on the same hardware, which translates into cheaper inference for high-capacity models like Kimi and GLM. The techniques preserve benchmark scores, so model quality does not suffer, but they introduce phase-specific performance differences that must be managed in deployment pipelines.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Switching the KV cache from BF16 to FP8 halves its size, doubling the token window and allowing up to twice as many concurrent decode requests with about 30% lower cost per token.
Compressing model weights from 8-bit to 4-bit INT4 cuts the checkpoint size by roughly 40%, reducing per-GPU memory and accelerating decode by up to 55% at low concurrency, though prefill becomes slower due to extra dequantization work.
Across a suite of accuracy benchmarks, the FP8 cache and INT4 weights produce results indistinguishable from the higher-precision baselines, so deployments can adopt the optimizations without sacrificing answer quality.
THE READ
What the cluster adds up to.
Running inference for frontier models such as Kimi and GLM on Cloudflare's edge GPUs is limited by the amount of memory needed for both the model weights and the attention KV cache. To push more requests onto a single GPU, the team layered three optimizations: KV-cache quantization, weight compression, and integrity checks for shared caches. These changes are applied on top of an existing split between prefill and decode pipelines, which already improves GPU utilization. The KV cache, which stores attention keys and values for each processed token, is normally kept in 16-bit BF16 format. By moving it to an 8-bit floating-point format (FP8), the cache size is halved, enabling the model to retain roughly twice as many tokens in memory. Benchmarks show that while BF16 is marginally faster per token, it runs out of memory at 32 concurrent requests, whereas FP8 continues to 64 requests, delivering about 41% higher peak throughput and roughly 30% lower cost per token. The trade-off is a small per-token conversion overhead in the attention kernel. Model weights for GLM were compressed from 8-bit floating point to 4-bit integers (INT4), shrinking the checkpoint by about 40% and r
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER