---
title: llm-d 1.0 launch under CNCF adds production features for distributed inference
url: https://www.elseif.net/llm-d-10-launch-under-cncf-adds-production-features-for-distributed-inference
published: 2026-10-23T18:03:51+00:00
language: en
section: Models
source: https://www.techtalkthai.com/llm-d-1-0-cncf-production-ready-distributed-ai-inference/
organizations: llm-d, CNCF, CoreWeave, Google Cloud, NVIDIA, IBM
publisher: elseif
---

# llm-d 1.0 launch under CNCF adds production features for distributed inference

The llm-d project team released version 1.0 of the open source software under CNCF governance for distributed LLM inference. The release targets Day 2 operations in production environments. Multi-tenant quality of service includes priority and fairness flow control, allowing latency-sensitive requests and bursty agent traffic to share a fleet without resource contention. Zero-downtime operations support router high availability with hot standby and rolling updates, while managing prefill and decode replicas as a logical unit with coordinated rollout and lockstep upgrades. A stable foundation consists of configuration API v1 and a request path tested across multiple clouds and accelerators. Agent-aware serving uses session-aware routing to deliver multi-turn conversations to points with cached context, combined with turn-priority scheduling and native Responses API state support to reduce redundant processing.

In a recent IBM Research case using an open frontier model with llm-d for quantum error correction hardware design, inference costs were 11.5 times lower than proprietary systems, with a prefix-cache hit rate of approximately 98% and an average service cost of 0.11 dollars per million tokens. In another benchmark, researchers deployed the GLM 5.2 open-weight mixture-of-experts model with 753 billion parameters on widely used NVIDIA GPUs. With hundreds of parallel agentic sessions, the system delivered over 6.6 million tokens per minute and supported 3,000 coding agents simultaneously without preemption. Current cloud pricing shows the cost per token is 5 to 10 times lower than equivalent commercial API rates, with the greatest savings occurring in heavy input traffic typical of agentic workloads.

Additional testing showed output throughput up to three times higher. The project now has over 500 contributors from more than 25 organizations. Partners including Google Cloud are integrating llm-d 1.0 into Google Kubernetes Engine and running it on Google TPU and NVIDIA GPU. CoreWeave offers the software on CoreWeave Kubernetes Service alongside NVIDIA Dynamo. NVIDIA plans to use llm-d on Vera Rubin, and AMD supports Instinct GPUs. Users such as DigitalOcean, JetBrains, Mistral, Cohere and Roblox operate the software in their model serving systems.