Nvidia Groq 3 LPU benchmarks show 4x token throughput on Gemma 4 31B model

Nvidia’s Groq 3 LPU benchmarks highlight a fundamental shift in inference hardware design. The chips prioritize memory bandwidth over capacity, using on-die SRAM to achieve 150 TB/s, orders of magnitude faster than HBM4. This architecture eliminates DRAM bottlenecks but introduces a new constraint: each LPU holds only 500 MB of memory. For context, Nvidia’s Rubin GPUs offer 288 GB of HBM, making the Groq 3’s 500 MB per chip a stark trade-off. The LPX rack compensates by aggregating up to 256 LPUs, but this requires Ethernet-based distribution, adding latency and complexity to the system.

The benchmark results, 3,400 tokens per second on Gemma 4 31B, are impressive but narrowly scoped. The model fits neatly into a single LPX rack at FP8 precision, avoiding the need for multi-rack scaling. This is a best-case scenario: dense models with parameters that can be evenly distributed across LPUs. Pipeline parallelism likely drives the performance, but the lack of transparency from Nvidia about the exact distribution method leaves questions about efficiency at higher concurrency levels. Data parallelism could improve throughput, but only if the model can be replicated across LPUs without hitting memory limits.

The real test for Groq 3 LPUs will be their performance on larger or more complex models. MoE architectures, like DeepSeek V3, activate only a subset of parameters per token, introducing overheads that dense models avoid. The Gemma 4 31B benchmark sidesteps this issue, but MoE models are increasingly common in production. Additionally, the 500 MB per-LPU limit means even moderately sized models may require multi-rack setups, where Ethernet latency and synchronization could erode performance gains. Engineers must assess whether their workloads align with the LPU’s strengths or if traditional GPUs remain the safer choice.

Nvidia’s $20 billion bet on Groq’s technology hinges on the assumption that faster inference will drive smarter AI agents. The benchmark suggests that for specific use cases, like real-time code assistants or agents requiring high token throughput, the LPU architecture could justify its premium. However, the lack of data on power efficiency, thermal performance, or cost per token leaves critical gaps in the evaluation. Early adopters like Nebius will provide real-world validation, but until then, the LPU’s advantages remain theoretical for most workloads.