Presentation: From ms to µs: OSS Valkey Architecture Patterns for Modern AI

Dumanshu Goyal framed the latency problem with an analogy to NASA’s shuttle program, illustrating how an over-engineered design can balloon cost and complexity. In the data-layer context, the “tiles” and “wings” correspond to proxy components that were added to meet perceived requirements but ended up adding heat, i.e., CPU load, and maintenance burden. The presentation argues that shedding those extra layers yields a leaner, faster system.

The core issue identified is that proxy architectures hide CPU consumption from the primary service, causing unpredictable spikes in tail latency and expanding the failure surface. Because the proxy sits between the application and the cache, any slowdown or bug propagates to all downstream requests, creating a larger blast radius. Engineers must therefore account for the indirect cost of these intermediaries when sizing resources and setting latency targets.

By connecting applications directly to Valkey, the design eliminates the proxy’s processing layer, allowing requests to complete in microseconds rather than milliseconds. This direct path also simplifies failure isolation: a problem in one client does not cascade through a shared proxy, improving overall system resilience. The presenter demonstrated that such an architecture can reduce both latency and the total cost of ownership for AI feature stores.

Transitioning to a direct-access model is not free; teams need to refactor code to use Valkey’s native client APIs, remove or bypass existing proxy services, and possibly re-evaluate security or multi-tenant isolation strategies that the proxy previously handled. The new architecture may not support use cases that rely on request routing, request-level throttling, or other proxy-specific features, so those capabilities must be rebuilt elsewhere or accepted as trade-offs. In environments where those features are essential, a hybrid approach or selective proxy use might be required.