Apache Iceberg is a high-performance table format designed for massive analytic datasets, developed under the Apache Software Foundation. The project provides a unified approach to managing large-scale data by enabling multiple query engines including Spark, Trino, Flink, Presto, Hive, and Impala to safely access and modify the same tables concurrently. The core Java library in this repository serves as the reference implementation for Iceberg across all language bindings and integrations.
The repository is organized into modular components that handle different aspects of table management and engine integration. The foundational modules include iceberg-common for shared utilities, iceberg-api for the public interface, and iceberg-core which contains the primary API implementations and Avro file support. Optional modules extend functionality for specific file formats: iceberg-parquet handles Parquet-backed tables, iceberg-arrow enables reading Parquet data into Arrow memory, and iceberg-orc provides ORC file support. The iceberg-hive-metastore module implements table backing through the Hive metastore Thrift client, while iceberg-data allows direct table access from JVM applications.
Engine-specific integration is handled through dedicated modules. The iceberg-spark module implements Spark's Datasource V2 API with submodules for different Spark versions, iceberg-flink contains Apache Flink integration classes, and iceberg-mr provides InputFormat and related classes for Apache Hive integration. The project maintains compatibility across multiple versions of these engines, with detailed compatibility information available through the Multi-Engine Support documentation.
The repository demonstrates significant community engagement and active maintenance. GitGenius tracking shows a median issue and pull request response latency of zero hours with a mean of approximately 2703 hours, indicating rapid initial triage followed by longer resolution periods for complex issues. The most active labels tracked are stale with 878 occurrences, bug with 732, and improvement with 494, reflecting ongoing maintenance and feature development. Key contributors nastra, RussellSpitzer, and manuzhang have driven 544, 480, and 472 tracked events respectively, establishing a core team managing the project's direction.
The project uses Gradle for building with Java 17 or 21 as requirements. The build system supports standard operations including full builds with tests, test skipping for faster builds, and code style fixes through spotlessApply commands. Testing infrastructure requires Docker, with specific considerations noted for macOS Docker Desktop users and SELinux configurations on certain systems.
Iceberg's format specification is documented as stable, with new features added in each version while maintaining backward compatibility. The project tracks issues and pull requests through GitHub and encourages contributions via pull requests, with community discussions occurring on the dev mailing list. Beyond the Java reference implementation, the Iceberg ecosystem includes implementations in Go, Python, Rust, and C++, each maintained as separate Apache projects. The project's classification spans schemas evolution, Hadoop compatibility, schema management, ACID transactions, partitioning, metadata management, and data lake functionality, positioning it as a comprehensive solution for scalable analytics on large datasets.
