Delta Lake is an open-source storage framework that enables building a Lakehouse architecture by combining data lake and data warehouse capabilities. Written primarily in Scala, it provides ACID transaction support and integrates with multiple compute engines including Apache Spark, PrestoDB, Flink, Trino, and Hive. The project offers APIs for Scala, Java, Rust, Ruby, and Python, making it accessible to a wide range of development environments and use cases.
The repository implements a transaction protocol that guarantees serializability for concurrent reads and writes. Delta Lake achieves ACID guarantees through specific requirements on underlying storage systems, including atomic visibility of files, mutual exclusion for writers, and consistent directory listings. The framework maintains backward compatibility with all Delta Lake tables, ensuring that newer versions can always read tables written by older versions, though forward compatibility is not guaranteed as new protocol features are introduced.
Delta Lake provides multiple integration pathways for different systems. The Apache Spark connector allows reading from and writing to Delta Lake tables. The Delta Standalone library enables Scala and Java-based projects, including Apache Flink, Apache Hive, Apache Beam, and PrestoDB, to interact with Delta tables without requiring Spark. The Delta Rust API provides low-level access to Delta tables with Python and Ruby bindings for use with data processing frameworks. Trino and PrestoDB connectors support reading and writing capabilities, while the Apache Flink connector focuses on write operations.
The codebase is built using SBT and requires Java 17 or later. Development setup is supported through IntelliJ, which is the recommended IDE. The project includes comprehensive test suites and provides both Scala and Python testing environments, with Python tests managed through Conda.
Activity tracking shows the repository maintains active engagement with its community.
Delta Lake is licensed under Apache License 2.0 and maintains community engagement through public Slack channels, a mailing list, LinkedIn, and YouTube presence.
