I Compacted 1,000 Apache Iceberg Files Into 6. Here’s What Happened to Query Performance.

Large analytical datasets stored in table formats like Apache Iceberg can suffer from the "small files problem," where data is scattered across many tiny files, degrading query performance. To combat this, a maintenance process called compaction is used to combine small files into fewer, larger ones, reducing metadata overhead. An experiment benchmarks three SQL workloads on a 50-million-row Iceberg table before and after compacting 1,000 small files into six large ones. The results demonstrate that compaction significantly improves query performance by giving the query engine fewer files to open and manage. The process is demonstrated using Python and PySpark to create the table, run the benchmarks, and execute the compaction procedure.