New Postgres sharding tool Neki builds on two decades of MySQL and Postgres solutions

Postgres sharding has long been a fragmented space, with companies often building custom solutions tailored to their specific workloads. Unlike MySQL, which benefited from early adoption by large-scale applications like Facebook and YouTube, Postgres lacked a dominant sharding framework. This led to a proliferation of one-off tools, each solving a narrow set of problems but failing to provide a general-purpose solution. Neki’s emergence suggests an attempt to unify these efforts, drawing from the successes and failures of prior tools like Vitess and PL/Proxy.

The operational cost of sharding Postgres has historically been high. Early solutions like PL/Proxy required manual function definitions for every query pattern, forcing tight coupling between application code and the sharding layer. This added complexity to both development and operations, as teams had to maintain routing logic alongside their application logic. Neki’s approach, while not fully detailed in the material, appears to aim for a more automated or declarative model, reducing the need for bespoke routing functions and lowering the barrier to adoption.

MySQL’s sharding ecosystem advanced more rapidly due to the LAMP stack’s dominance in the 2010s. Tools like Vitess, born out of YouTube’s scaling needs, provided a blueprint for horizontal scaling that Postgres lacked. Postgres users often resorted to vertical scaling or custom sharding layers, which were difficult to generalize. Neki’s development signals a shift toward bringing Postgres closer to MySQL’s level of sharding maturity, though it remains to be seen whether it can address the unique challenges of Postgres’ architecture, such as its reliance on extensions and its more rigid transaction model.

The material highlights that sharding is often a last resort, adopted only when a single node can no longer handle the workload. This reflects the trade-offs involved: sharding introduces complexity in query routing, data distribution, and operational overhead. Neki’s value proposition likely hinges on reducing this complexity, but it will still face limitations. For example, cross-shard transactions and joins remain challenging in any sharded system, and Postgres’ lack of native support for these operations means Neki will either need to work around them or accept their limitations.

The history of sharding tools underscores the tension between generality and specialization. Vitess succeeded because it was built for a specific use case (YouTube’s workload) but proved flexible enough for broader adoption. Postgres’ sharding solutions, like PL/Proxy, were often too tightly coupled to their original environments. Neki’s challenge will be to strike a balance: offering enough flexibility to handle diverse workloads while avoiding the pitfalls of over-customization that plagued earlier Postgres tools.