Delta Sharing is an open protocol for secure real-time exchange of large datasets that enables organizations to share data regardless of which computing platforms they use. The repository contains the protocol specification, reference server implementation, and connectors for multiple languages and systems. The protocol itself is a simple REST interface that securely shares access to portions of cloud datasets and leverages modern cloud storage systems such as S3, ADLS, or GCS to transfer data reliably.
The repository includes several core components. The protocol specification defines how data sharing works at the technical level. The Python Connector is a library implementing the Delta Sharing Protocol to read tables from a Delta Sharing Server, allowing users to load shared tables as pandas DataFrames or Apache Spark DataFrames. The Apache Spark Connector implements the same protocol for reading shared tables in SQL, Python, Java, Scala, and R. The Delta Sharing Server is a reference implementation for development purposes that allows users to share existing tables in Delta Lake and Apache Parquet format on cloud storage systems.
The Python Connector requires Python 3.8 or higher for version 1.1 and later. It accesses shared tables through profile files, which are JSON files containing user credentials to access a Delta Sharing Server. The connector supports loading data as pandas DataFrames and integrates with Apache Spark for distributed processing. It also supports querying table changes when history sharing is enabled on the server side.
The Apache Spark Connector requires Java 8 or higher, Scala 2.12.x, and Apache Spark 3 or Databricks Runtime 9 and later. Users can set it up interactively through Spark shells or as part of standalone Maven or SBT projects. The connector supports Change Data Feed queries starting from release 0.5.0 and Spark Structured Streaming as a data source from release 0.6.0 onward.
The repository overlaps with contributors from databricks/terraform-provider-databricks, duckdb/duckdb, and jetbrains/compose-multiplatform.
The repository is classified across numerous categories including data sharing, delta format, authentication, cloud storage, API interfaces, version control, authorization, privacy-preserving technologies, fine-grained access control, secure data exchange, data governance, and compliance. The primary language is Scala, and the project is associated with topics including big-data, data-sharing, delta-lake, pandas, and spark.
The ecosystem around Delta Sharing extends beyond the core repository through community-contributed connectors. Power BI has a Databricks-owned connector that supports query table version, metadata, and latest snapshot operations. Community members have contributed connectors for Clojure, Node.js, and Java, with the Clojure connector supporting advanced features like Change Data Feed queries, time travel queries, delta format queries, and predicate pushdown. This ecosystem allows organizations to integrate Delta Sharing with their existing tools and platforms.
