Spotify Builds External Index to Enable Low Latency Point Queries on Its Data Lake

Details

Spotify's RAP (Random Access Parquet) adds an external index layer over Apache Parquet files, mapping lookup keys — like user IDs — directly to specific files and byte offsets. Instead of scanning thousands of files with Trino or BigQuery, a query resolves through the index then issues a targeted range read against Google Cloud Storage. GCS now provides millisecond access latency; the bottleneck was metadata traversal and file discovery overhead, which RAP eliminates.\n\nSpotify's motivation is scale: the company stores petabytes of serving data in Bigtable while exabytes reside in its GCS-based data lake, making wholesale replication into operational databases increasingly expensive. RAP allows the same Parquet datasets used for analytics, ML training, and notebooks to also serve low-latency online queries — including AI agents — without maintaining duplicate storage systems.\n\nRAP is compatible with Apache Iceberg: as data is written into Iceberg tables, an index builder generates append-only index fragments without modifying immutable Parquet files, so existing analytical pipelines require no changes. The approach follows similar architectural exploration by Google Cloud on Iceberg-based lakehouses for AI workloads, though RAP takes a different path via a dedicated external index rather than rethinking the table format.