How we tracked down a 16-year-old SQLite bug

Tailscale’s control plane relies on SQLite databases, one per shard, with a single-writer design that aligns with SQLite’s recommended usage. This architecture is common in distributed systems, where SQLite’s simplicity and reliability are valued. However, the repeated corruption incidents exposed a flaw in this assumption, even in a well-configured deployment. The bug’s persistence over 16 years suggests it only manifests under specific, rare conditions, likely a combination of workload, timing, and environmental factors that Tailscale’s scale made visible.

The corruption had tangible operational costs. Each incident required halting the control plane process on the affected shard, leading to downtime for tailnets hosted there. Recovery times improved over time, but early incidents lasted over an hour, during which devices could not establish new connections or receive network updates. While existing peer-to-peer connections remained functional, the loss of control plane access disrupted admin console and API availability, creating a cascading effect on user experience. The broader impact on trust was equally significant, as global status page notifications eroded confidence even for unaffected users.

Debugging the issue was challenging because the bug did not correlate with recent code changes or obvious environmental triggers. Initial investigations focused on the backup pipeline, which snapshots databases to S3 every few minutes, but the corruption occurred independently of this process. The lack of clear patterns forced Tailscale to adopt a methodical, forensic approach, ruling out common causes like hardware failures or filesystem issues. The eventual resolution required deep collaboration with the SQLite community, highlighting how even well-documented systems can harbor edge cases that only emerge under production-scale stress.

The incident underscores the importance of proactive integrity checks. Tailscale’s use of `PRAGMA integrity_check` to detect corruption was critical, but the tool’s limitations, such as its inability to pinpoint the root cause, meant additional effort was needed to trace the bug. For engineers deploying SQLite, this serves as a reminder to build redundancy into recovery pipelines, as even rare corruption events can become frequent at scale. The fix also reinforces the value of transparency, as Tailscale’s public postmortem provides actionable insights for others running similar architectures.