Technical Tutorials

In the evolving landscape of big data, the Data Lakehouse architecture has emerged as the gold standard for scalable, performant, and open analytics. At the heart of this revolution sits Apache Iceberg, a high-performance table format for huge analytic datasets. While Iceberg itself is a Java library, interacting with it typically requires setting up a runtime environment—whether that’s through a standalone CLI, Docker containers, or integration with engines like Spark or Trino. This guide will walk you through the practical steps to install and verify Apache Iceberg on a local development environment using Linux and Docker.

Prerequisites: Understanding the Environment

Before diving into installation, it is crucial to understand that Apache Iceberg is not a standalone service you simply "install" like a database. It is a table format specification. Therefore, to use it, you need a runtime engine that supports the Iceberg format. The most common entry points for developers are:

Apache Spark: The most popular engine for writing and reading Iceberg tables.
Trino (formerly PrestoSQL): Ideal for interactive SQL queries.
Apache Hive: For legacy compatibility.

For this guide, we will focus on the Iceberg CLI (Command Line Interface) and a Docker-based Spark setup, as these provide the quickest path to getting started without managing a full Hadoop cluster.

Method 1: Installing the Iceberg CLI via Homebrew

The Iceberg CLI provides a powerful tool for managing catalog metadata, inspecting tables, and performing basic operations. If you are on macOS or Linux with Homebrew installed, this is the fastest method.

First, ensure you have Homebrew installed. Then, run the following commands to install the latest version of Iceberg:

# Update your package list
brew update

# Install the Apache Iceberg CLI
brew install apache-iceberg

Once installed, you can verify the installation by checking the version:

iceberg --version

This command should return the current version number, confirming that the CLI is available in your system path. You can now initialize a local Hadoop catalog by setting the appropriate configuration files in your home directory or current working directory.

Method 2: Running Iceberg with Docker and Spark

For a more robust testing environment, or if you are using a system without Homebrew, Docker offers an isolated and reproducible way to run Iceberg. The official Apache Iceberg Docker images include pre-configured Spark instances with Iceberg dependencies baked in.

Start by pulling the official Iceberg image. You can choose a specific Spark version; here we use Spark 3.3:

docker pull apache/iceberg:spark-3.3

Once the image is downloaded, you can launch a Spark container that connects to your local Hadoop file system or S3-compatible storage. Below is an example command to run a Spark shell with Iceberg enabled:

docker run -it \
  --name iceberg-spark \
  -v ${PWD}:/home/iceberg/data \
  -e AWS_REGION=us-east-1 \
  -e AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY \
  -e AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY \
  apache/iceberg:spark-3.3 \
  spark-sql \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
  --conf spark.sql.catalog.spark_catalog.catalog-impl=org.apache.iceberg.hive.HiveCatalog

Note: In a real-world scenario, you would typically mount your local data directory to the container and configure the catalog to point to a metadata store like Hive Metastore or a simple Hadoop file system.

Verifying the Installation

After installation, verification is key. If you used the CLI, try creating a simple namespace:

iceberg catalog create --uri file:///tmp/hive_catalog

If you are using Spark, run a simple SQL command inside the Spark Shell:

CREATE NAMESPACE my_database;
SHOW NAMESPACES;

If you see your namespace listed, your Iceberg installation is successful and ready for use.

Conclusion

Installing Apache Iceberg is less about downloading a binary and more about configuring the runtime environment that supports it. By leveraging either the Homebrew CLI for quick local tests or Docker for containerized development, developers can easily integrate Iceberg into their data engineering workflows. Whether you are building a modern data lakehouse on AWS, GCP, or Azure, mastering the installation and configuration of Iceberg is the first critical step toward unlocking scalable, ACID-compliant data analytics.