Concurrency vs. Throughput: why more parallelism can make databases slower

The incident started when a batch job opened a transaction on a hot table and held its locks for fifteen minutes without committing. Although the transaction blocked only writers, the read queries that arrived did not wait for those locks; instead InnoDB served them from a consistent snapshot. Building each snapshot required traversing the version history of every row the transaction had modified, and that history grew continuously for the full fifteen minutes.

As the version chains became longer, each read query took far longer than its usual millisecond latency, eventually hitting the 90-second execution ceiling. The application responded by retrying the queries in a tight loop, which caused more than ten thousand requests to accumulate inside MySQL. Processing each request forced the engine to reconstruct ever-longer version chains, leading to excessive page reads that outpaced the buffer pool’s ability to free memory.

Because the buffer pool was overwhelmed, even queries that touched completely unrelated tables began to fail. The system, which had been sized correctly for normal load, could not serve the sudden surge of read-only work generated by the long transaction. This shows how a single slow transaction can indirectly degrade performance across the entire database.

The environment differed from the previous Cloud SQL setup, which used a thread-pool style connection manager that queued excess statements when the pool was full. In contrast, Vitess’s vttablet employs a transaction pool that, once full, makes waiting clients fail after a preset timeout. Raising the transaction pool cap to ten thousand removed this backpressure, allowing the application to flood the database with more in-flight requests and worsening the version-chain overhead.

According to Little’s Law (N = X * W), adding in-flight requests only raises throughput when the average execution time per request stays constant. Here each new request increased the execution/wait time for all others by elongating the version chains they had to traverse, so the denominator grew alongside the numerator. Gunther’s Universal Scalability Law captures this effect: the γN term represents the overhead that limits scaling, and in this case the overhead came from version-chain reconstruction rather than lock contention.

The problem was resolved when Vitess’s configured transaction timeout killed the long-running transaction, releasing its locks and allowing the backlog of thirty seconds to drain. The episode illustrates that limiting concurrency, providing effective backpressure, and monitoring version-chain growth are critical practices for maintaining MySQL throughput under realistic workloads.