How-To Guides

Mastering the Modern Data Lake: How to Install and Configure Apache Iceberg

Oct 28, 2026 · 3 views
sqlBig DataHadoopapache icebergdata lakehouseSparkOpen Table FormatFlink

Apache Iceberg has rapidly become the standard for open table formats in the big data ecosystem. By providing ACID transactions, schema evolution, and versioning directly on top of data lakes, Iceberg bridges the gap between the flexibility of a data lake and the reliability of a data warehouse. However, getting started with Iceberg can be overwhelming due to the various components and dependencies involved.

This guide provides a comprehensive walkthrough for intermediate and advanced developers on how to install and configure Apache Iceberg using Apache Spark, one of the most common engines for processing Iceberg tables. We will cover the prerequisites, dependency installation, and creating your first table.

Prerequisites and Environment Setup

Before diving into the installation, ensure your environment meets the following requirements:

  • JDK 1.8 or higher: Iceberg supports Java 8+, but Java 11 or 17 is recommended for performance.
  • Apache Spark 3.x: We will use Spark 3.3+ for this example, as it has better native integration with Iceberg.
  • A Compatible File System: Local file system for development, or S3/GCS/HDFS for production.
  • Maven or Gradle: For managing dependencies in a Java/Scala project.

Adding Iceberg Dependencies

The first step in installing Iceberg is adding the necessary libraries to your project. Iceberg is modular, so you need the core library and the specific engine integration library (in this case, Spark).

Below is the pom.xml snippet for Maven. Note the version 1.2.0; you should check the official release page for the latest stable version.

<dependency>
  <groupId>org.apache.iceberg</groupId>
  <artifactId>iceberg-core</artifactId>
  <version>1.2.0</version>
</dependency>
<dependency>
  <groupId>org.apache.iceberg</groupId>
  <artifactId>iceberg-spark-runtime-3.3_2.12</artifactId>
  <version>1.2.0</version>
</dependency>

Important Note on Shading: The iceberg-spark-runtime artifact is shaded, meaning it repackages internal dependencies to avoid conflicts with your existing Spark libraries. This is the recommended approach for Spark applications.

Configuring the Spark Session

Once the dependencies are added, you need to configure the Spark session to recognize the Iceberg catalog. Iceberg supports multiple catalogs, including hive, jdbc, rest, and hadoop. For simplicity and local testing, we will use the HadoopCatalog which stores metadata in the file system itself.

Here is a Scala example for configuring the Spark session:

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog

val spark = SparkSession.builder()
  .appName("IcebergInstallationGuide")
  .master("local[*]")
  .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
  .config("spark.sql.catalog.iceberg", "org.apache.iceberg.spark.SparkCatalog")
  .config("spark.sql.catalog.iceberg.type", "hadoop")
  .config("spark.sql.catalog.iceberg.warehouse", "/tmp/iceberg-warehouse")
  .getOrCreate()

Key configurations explained:

  • spark.sql.extensions: Registers Iceberg-specific SQL extensions.
  • spark.sql.catalog.iceberg: Defines a named catalog called "iceberg".
  • spark.sql.catalog.iceberg.type: Specifies the catalog implementation. hadoop is easiest for local dev. For production, use hive or rest.
  • spark.sql.catalog.iceberg.warehouse: The root directory where table data and metadata will be stored.

Creating and Querying Your First Iceberg Table

With the catalog configured, you can now create an Iceberg table. Let's create a simple table named user_events in a namespace called analytics.

import org.apache.spark.sql.functions._

// Create namespace
spark.sql("CREATE NAMESPACE IF NOT EXISTS iceberg.analytics")

// Create table
spark.sql("""
  CREATE TABLE IF NOT EXISTS iceberg.analytics.user_events (
    id INT,
    name STRING,
    action STRING,
    timestamp TIMESTAMP
  )
  USING ICEBERG
""")

// Insert sample data
val data = spark.createDataFrame(Seq(
  (1, "Alice", "login", "2023-10-27 10:00:00".toTimestamp),
  (2, "Bob", "purchase", "2023-10-27 10:05:00".toTimestamp)
)).toDF("id", "name", "action", "timestamp")

data.writeTo("iceberg.analytics.user_events").append()

// Query the table
spark.sql("SELECT * FROM iceberg.analytics.user_events").show()

Verification and Best Practices

After running the above code, you should see the sample data printed in the console. To verify the installation was successful, check the file system at /tmp/iceberg-warehouse/analytics/user_events. You should see directories for data, metadata, and snapshots.

Best Practices for Production:

  1. Use a Metastore: While hadoop catalog is fine for testing, use Hive Metastore or a REST catalog in production for multi-engine support and metadata management.
  2. Commit Frequency: Iceberg commits can be expensive. Batch your writes to reduce the number of snapshots created.
  3. Snapshot Expiration: Configure snapshot expiration policies to manage storage costs. Old snapshots consume disk space until explicitly removed.

Conclusion

Installing Apache Iceberg is a straightforward process once you understand the catalog system and dependency management. By leveraging the shaded runtime artifacts and configuring the Spark session correctly, you can quickly start building reliable, ACID-compliant data pipelines on your existing data lake. As you move to production, consider migrating from the Hadoop catalog to Hive or REST catalogs to unlock advanced features like concurrent access and centralized metadata management.

Share:
Related Posts