databricks/zerobus-sdk

Zerobus SDK is a monorepo of client libraries that enables high-throughput streaming data ingestion into Databricks Delta tables.

Zerobus addresses the need for real-time, high-volume data pipelines by providing a dedicated streaming service optimized for direct ingestion into Delta tables. The SDK approach works by offering language-specific bindings across multiple platforms, allowing developers to integrate streaming ingestion from their preferred language without building custom solutions. The tool handles the complexity of managing connections, authentication, and data serialization to Databricks infrastructure.

The SDK is production-ready and suitable for teams building real-time data pipelines that require high throughput into Databricks. It supports Rust, Python, Go, TypeScript, Java, C++, and C#, with prebuilt binaries available for Linux, Windows, and macOS across x86_64 and aarch64 architectures. Adoption requires a Databricks workspace with appropriate permissions, a Delta table target, and a service principal with credentials. The tool is backed by Databricks and carries a general availability designation, meaning minor and patch updates maintain backward compatibility while major versions may introduce breaking changes.

The project maintains active issue tracking and welcomes user feedback. Development spans multiple language implementations with platform-specific binary distribution, indicating sustained engineering effort across the SDK ecosystem. The team acknowledges platform limitations, particularly around macOS binary availability due to CI constraints, and provides guidance for building from source when prebuilt binaries are unavailable.