Transformer-only ASIC Sohu reportedly delivers 62,500 tokens/sec per chip vs H100’s 700 tokens/sec at batch 1

Etched AI’s Sohu chip is a transformer-only ASIC that replaces programmable compute units with fixed-function silicon. By hard-coding attention operations directly into circuits, Sohu eliminates the overhead of kernel launches, memory allocation, and scheduler decisions that limit GPU performance. The result is a reported 62,500 tokens/sec per chip on Llama 70B, compared to ~700 tokens/sec for an H100 at batch 1. This throughput advantage is real for autoregressive inference, but it comes at the cost of flexibility, Sohu cannot run any workload that doesn’t map to transformer attention.

The architectural trade-off is stark: Sohu’s performance is tied to the stability of transformer attention as the dominant AI architecture. If future models shift toward state-space models (SSMs), diffusion, or other non-attention mechanisms, Sohu’s fixed-function design becomes obsolete. For teams evaluating inference hardware, this means the ASIC is only viable if their workload is exclusively transformer-based and unlikely to change in the next few years. The lack of programmability also complicates debugging and optimization, as there’s no software layer to modify or profile.

Supply chain and toolchain risks add to the adoption hurdle. Sohu is not yet available for purchase or rent, and no independent benchmarks exist to verify Etched’s claims. The first racks are scheduled to ship in summer 2026, but teams must commit to the platform without hands-on testing. Additionally, migrating to Sohu requires retooling inference pipelines, as the chip lacks CUDA or ROCm compatibility. For hyperscalers or large enterprises with stable transformer workloads, the throughput gains may justify the migration cost, but smaller teams or those with diverse workloads may find the lock-in unacceptable.

The comparison to other custom inference chips highlights Sohu’s unique position. Groq’s LPU and AMD’s GPUs retain programmability, allowing them to adapt to new workloads. SambaNova’s SN40L, while also an ASIC, targets a broader range of models. Sohu’s bet is that transformer attention is so dominant that programmability is unnecessary. If the bet pays off, the chip could become a standard for high-throughput inference. If not, it risks becoming a niche solution for a shrinking workload class.