INFRA Signal 477
OpenCost 1.121.0: First-of-a-kind Kubernetes inference cost tracking
↗
OpenCost 1.121.0 integrates with llm-d to expose per-model and per-token cost metrics for LLM inference on Kubernetes, distinguishing between allocation-based costs (the cost of having a model available) and usage-based costs (the cost of actual inference work).
Platform teams running inference workloads currently have no way to connect GPU spend to token output, making build-vs-buy decisions and budget attribution guesswork. This release gives teams two complementary cost metrics that together reveal both the true cost of inference work and the cost of idle capacity, enabling data-driven decisions about model selection, utilization, and self-hosting versus SaaS APIs.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
OpenCost 1.121.0 introduces allocation-based and usage-based cost metrics per model, where the gap between them quantifies the cost of keeping a model warm but idle.
The integration works with vLLM metrics directly, so teams not using llm-d can still capture per-token inference costs.
Utilization can now be expressed as the ratio of usage-based to allocation-based cost per million tokens, eliminating the need for a separate utilization metric.
THE CLUSTER