DATABASES Signal 454
Spotify introduces Random Access Parquet architecture for low-latency point queries on data lake
↗
Spotify introduced Random Access Parquet (RAP), an external indexing architecture over Apache Parquet files that enables low-latency point queries directly against data lake storage without replicating datasets into operational databases.
This eliminates the need to maintain duplicate storage systems for analytical and operational workloads, reducing cost and complexity for organizations with large-scale data lakes. Engineers running both batch analytics and latency-sensitive serving from the same Parquet datasets can now avoid the data replication and synchronization overhead that traditional architectures require.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
RAP maps lookup keys directly to Parquet file locations and row positions, enabling targeted reads from cloud object storage instead of full file scans.
An index builder generates append-only index fragments as new data is written to Apache Iceberg tables, keeping immutable Parquet files unchanged.
Storage layout optimizations including interleaved value columns and covering indexes allow some point queries to be served through a single ranged read of only a few kilobytes.
THE CLUSTER