Presentation: Producing the World's Cheapest Tokens: A How-to Guide

The talk shifts the design goal from minimizing latency to minimizing token cost for workloads that can tolerate delay. Engineers are encouraged to re-evaluate the inference stack, selecting hardware and software that favor throughput over response time. This mindset change enables the use of techniques that are unnecessary in latency-critical services. The result is a pathway to run the same models at a fraction of the current expense.

Adopting the approach requires investing in batch-oriented infrastructure, such as job queues that can reorder requests based on readiness, and possibly choosing accelerators that excel at high batch sizes. Teams must also implement or configure speculative decoding and smart queue reordering logic, which may involve engineering effort. The trade-off is accepting higher latency, which may need to be communicated to downstream consumers of the generated tokens.

The method is unsuitable for interactive applications like chatbots or coding assistants where low latency is a primary requirement. It also loses effectiveness when the workload cannot be batched or when model quality must be maximized without any quantization or model-size reduction. In those scenarios the cost savings disappear and the original latency-focused stack remains preferable.