Apache Iceberg has rapidly become a standard for open table formats in the big data ecosystem, offering ACID transactions, schema evolution, and time travel capabilities. However, as with any large-scale data lake solution, cost efficiency is a primary concern for engineering teams. The two most critical operations for controlling costs and performance in Iceberg are file compaction and snapshot expiration. Mismanaging these processes can lead to excessive storage bills and degraded query performance.
Understanding File Compaction
Over time, frequent small writes to an Iceberg table result in a fragmented set of small files. While small files are great for parallelism in ingestion, they are disastrous for analytics. Scanning thousands of tiny files incurs high API call costs on object stores like S3 or GCS and slows down query engines like Spark and Trino. File compaction (or "bin-packing") merges these small files into larger, optimal-sized files (typically 128MB to 256MB).
Regular compaction ensures that your data is stored in a format that is efficient for both storage and retrieval. You can schedule this as a maintenance job or trigger it programmatically.
// Example: Compaction using Spark Iceberg
import org.apache.iceberg.spark.SparkCatalog
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
.appName("Iceberg Compaction")
.config("spark.sql.catalog.demo", "org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.demo.type", "hadoop")
.config("spark.sql.catalog.demo.warehouse", "s3://my-bucket/iceberg/warehouse")
.getOrCreate()
// Rewrite small files into larger ones
spark.sql("REWRITE FILES TABLE demo.my_table")
// Optionally, specify a target file size
spark.sql("REWRITE FILES TABLE demo.my_table OPTIONS('target-file-size-bytes' 134217728)")
Managing Snapshot Expiration
Iceberg supports time travel, meaning it retains history of your table via snapshots. Each snapshot references a set of files. If you delete data or rewrite files without cleaning up old snapshots, the old data files remain on disk because the snapshots still reference them. This "zombie data" consumes storage space that you are still paying for.
Snapshot expiration removes old snapshots and, crucially, deletes the data files that are no longer referenced by any remaining snapshot. This is essential for cost control. However, you must balance the retention period with your business needs for auditing or rollback capabilities.
// Example: Expire snapshots older than 7 days
import org.apache.iceberg.Table
val table = spark.catalog.table("demo.my_table")
// Expire snapshots older than 7 days
table.expireSnapshots()
.retainLast(3) // Retain the last 3 snapshots regardless of age
.olderThan(System.currentTimeMillis() - (7L * 24L * 60L * 60L * 1000L))
.commit()
Best Practices for Cost Optimization
- Schedule Regular Maintenance: Use a scheduler like Airflow, Dagster, or cron to run compaction and expiration jobs daily or weekly.
- Monitor File Counts: Alert if the number of files in a table exceeds a threshold, indicating a need for immediate compaction.
- Test Expiration Policies: Simulate snapshot expiration in a staging environment to ensure you are not deleting data needed for compliance or rollback.
- Use ORC or Parquet: Choose columnar formats with high compression ratios to further reduce storage costs.
Conclusion
Optimizing Apache Iceberg for cost-effective cloud storage requires a proactive approach to data lifecycle management. By regularly compacting small files and expiring old snapshots, you can significantly reduce storage costs and improve query performance. Implementing automated maintenance jobs is not optional; it is a best practice for any production-grade Iceberg deployment. Start small, monitor your metrics, and adjust your policies to fit your specific workload requirements.