Technical Tutorials

As data architectures evolve, the need for robust, ACID-compliant table formats becomes paramount. Apache Iceberg has emerged as a leading open-source table format for huge analytic datasets, providing high-performance, scalable metadata management for data lakes. Whether you are building a Lambda architecture or moving toward a modern Lakehouse pattern, getting Iceberg up and running is a critical first step. This guide walks you through the installation process for various environments, ensuring you can start leveraging Iceberg’s features like time travel, schema evolution, and row-level deletes.

Prerequisites and Environment Setup

Before diving into the installation, it is essential to understand that Apache Iceberg is a library, not a standalone service. It integrates with existing compute engines like Apache Spark, Trino, Presto, or Flink. Therefore, your environment must have a supported Java Runtime Environment (JRE) installed. Iceberg typically requires Java 11 or Java 17, depending on the version and the engine you are using.

Ensure you have Apache Maven or Gradle installed if you are integrating Iceberg into a Java-based project. For distributed environments, ensure your Hadoop distribution (CDH, HDP, or Apache Hadoop) is compatible with the Iceberg version you intend to install. Check the official Iceberg compatibility matrix to avoid version mismatches between your cluster software and the Iceberg libraries.

Installing Iceberg via Maven (Java/Scala Projects)

The most common way to use Iceberg is by adding it as a dependency in your build tool. If you are using Maven, you need to add the appropriate BOM (Bill of Materials) to ensure consistent versions across all Iceberg modules. Add the following to your pom.xml file:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.apache.iceberg</groupId>
      <artifactId>iceberg-bom</artifactId>
      <version>1.4.3</version> <!-- Check for the latest version -->
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>org.apache.iceberg</groupId>
    <artifactId>iceberg-spark-runtime-3.4_2.12</artifactId> <!-- Adjust for your Spark version -->
    <version>1.4.3</version>
    <scope>provided</scope>
  </dependency>
</dependencies>

For Scala projects using Spark 3.4, the artifact ID ends with -3.4_2.12. If you are using a different Spark minor version, update the suffix accordingly. The provided scope is used because the compute engine (like Spark) will provide the runtime classes, preventing jar conflicts.

Setting Up Iceberg on Standalone Clusters (Spark & Hadoop)

If you are running Apache Spark or Hadoop on a cluster (e.g., YARN or Kubernetes), you cannot rely on Maven dependencies alone. You must distribute the Iceberg jars to all nodes in the cluster. The easiest way to do this is by downloading the iceberg-spark-runtime jar from the official Apache Iceberg release page.

Place the downloaded jar in the jars/ directory of your Spark installation. For example:

cd $SPARK_HOME/jars
curl -LO https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.4_2.12/1.4.3/iceberg-spark-runtime-3.4_2.12-1.4.3.jar

After placing the jar, restart your Spark shell or submit your jobs. You can verify the installation by running a simple SQL command in the Spark SQL shell:

CREATE TABLE default.sample_table (
  id INT,
  data STRING
) USING iceberg;

If this command executes without errors, Iceberg is successfully installed and integrated with your Spark environment. You can now inspect the table metadata to confirm the table format is correct.

Using Docker for Quick Prototyping

For developers who want to test Iceberg without setting up a full local cluster, Docker provides a convenient solution. You can spin up a pre-configured environment using official or community-maintained Docker images that include Spark and Iceberg. This approach isolates dependencies and ensures a reproducible environment for testing.

For instance, you can pull a Docker image that has Iceberg pre-installed and run a Spark container with the appropriate volume mounts. This method is particularly useful for CI/CD pipelines where you need to validate data transformations against Iceberg tables.

Conclusion

Installing Apache Iceberg is a straightforward process that varies slightly depending on your deployment architecture. By correctly configuring dependencies in Maven, distributing jars in cluster environments, or utilizing Docker for quick setups, you can unlock the power of open table formats. Remember to always verify your installation by creating a sample table and checking the metadata. As the data lakehouse landscape continues to mature, mastering Iceberg installation is a foundational skill for any serious data engineer. Stay tuned for our next post, where we will dive into advanced configurations like catalog implementations and performance tuning for high-throughput workloads.