Prometheus compression found roughly 10% better than VictoriaMetrics in ad-hoc testing

A claim at PromCon that VictoriaMetrics achieved 3-4x better compression than Thanos prompted scrutiny from attendees and ultimately this testing. The author, associated with Robust Perception, ran ad-hoc tests during the conference and followed up with a more detailed evaluation. The initial lightning talk finding was that Prometheus compression was roughly 10% better than VictoriaMetrics, and that VictoriaMetrics was not storing all the same information that Prometheus does.

The test setup forwarded data from Prometheus to VictoriaMetrics via remote write, both at default settings, using Prometheus 2.14.0 and VictoriaMetrics 1.29.3 on a machine with 32GB of RAM. The author deliberately excluded CPU and RAM comparisons, noting that Cortex, Thanos's receiver, or M3DB would be more appropriate peers for VictoriaMetrics since they also hook into remote write. The focus was specifically on compression and correctness, not broader operational characteristics like replication or fault tolerance.

Snapshot-based disk comparisons after half an hour showed Prometheus at 90M and VictoriaMetrics at 78M, though the author noted this was too early for compaction to have fully run. Query comparisons revealed that both systems returned the same values but with timestamp discrepancies beyond what remote-write ingest delay alone would explain. This timestamp difference is a correctness signal worth investigating when evaluating whether a third-party storage backend faithfully reproduces Prometheus query results.

The VictoriaMetrics developer corresponded with the author after the initial findings, suggesting adjustments to the test setup for realism, and was directed to the PromQL unittests to verify their third-party PromQL implementation rather than waiting for user-reported bugs. This exchange illustrates a constructive outcome: benchmark disputes can surface real gaps in correctness testing, and pointing a project to existing test suites is a concrete way to improve fidelity.

The broader takeaway for engineers is that evaluating software requires apples-to-apples setup, awareness of what data each system actually persists, and verification that query results match beyond just values. Compression ratios alone are insufficient when one system may be silently dropping metadata or timestamps. The author emphasizes reproducibility by publishing setup details and using stable versions, and notes that features like snapshot comparison help isolate storage efficiency from fixed overhead like Prometheus's WAL.