State-Oriented Consistency: Why We Stopped Looking for One Right Answer

The article starts by pointing out that early design efforts asked which consistency model the whole cluster should use, assuming a single answer fits all state. It argues that the more useful question is which consistency guarantee each individual piece of state actually requires. This shift reveals that the default habit of applying one consistency strategy everywhere, called Uniform Consistency, is often unnecessary and leads to waste.

During routine operation two pods were killed by the kernel's OOM handler despite having a modest 512 MiB memory limit and showing no obvious load imbalance. Investigation showed that the pods were handling vastly different numbers of active connections while using almost the same amount of working-set memory, contradicting expectations from a load-balancer-induced spike.

Reading the code revealed that on every pod start a persistence hook loaded the entire fleet's client session state, turning each row into a live in-memory object regardless of whether that pod would ever serve the corresponding client. The design had been chosen because, behind a non-sticky load balancer, no pod could predict which clients would reconnect to it, so loading everything seemed the simplest way to guarantee availability.

The team realized that the session state did not need to be replicated everywhere; it only needed a single owner that any node could determine locally using a rule such as hashing the client identifier. By giving each piece of state its own owner, the unnecessary memory load disappeared and the OOM incidents stopped. They labeled the previous habit Uniform Consistency to contrast it with the new state-oriented approach.

The lesson extends beyond message brokers: any distributed system that manages more than one kind of state can face the same hidden cost when it assumes a uniform guarantee. However, the approach works only when state can be assigned a deterministic owner without requiring true replication; for state that must be available on multiple nodes simultaneously, a different consistency model will still be needed.