TECH Signal 539
DeepSeek V4 Flash on a Single AMD MI300X
A community repo supplies the patches and Docker-Compose stack needed to serve DeepSeek-V4-Flash on a single AMD MI300X GPU.
The setup proves that the MI300X’s 192 GiB of HBM3 can hold the entire 304-billion-parameter model without PCIe weight streaming, offering a cost-effective alternative to Nvidia-focused deployments. It also documents the FP8 format and MoE routing fixes required for reliable inference on this older AMD architecture. Engineers can now benchmark multi-stream serving on a single card, simplifying hardware provisioning for production LLM services.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
The repository bundles ROCm-nightly vLLM, FP8-specific patches, and AITER tuning tables that make the DeepSeek-V4-Flash checkpoint run correctly on a single MI300X.
Measured throughput reaches about 168 tokens/s for a single decode stream, 7.9 to 8.5 K tokens/s for prefills, and sustains 8-stream and 64-stream bursts with no out-of-memory errors.
Deployment requires an MI300X card, ~235 GiB RAM for a CPU KV tier, ~500 GiB disk for the model cache, and a Docker-Compose environment pinned to specific image digests.
THE CLUSTER