Apache DataFusion Ballista is a distributed query execution engine built in Rust that extends Apache DataFusion by enabling parallelized execution of workloads across multiple nodes. The project allows existing DataFusion applications to be distributed with minimal code changes, making it accessible for users who want to scale their query processing without major architectural rewrites.
The Ballista architecture consists of scheduler processes and executor processes that can run as native binaries or Docker containers, with deployment options including Docker Compose and Kubernetes. Clients submit jobs to the scheduler, which coordinates task distribution to executors that report back on task status and completion. The system is designed to handle complex SQL queries including CTEs, joins, and subqueries at scale, though the project documentation acknowledges an ongoing gap between DataFusion and Ballista functionality that the community is actively working to close.
Performance benchmarks derived from TPC-H queries demonstrate significant optimization progress. Testing at scale factor 100 with 100 GB of data on a single node with one executor and eight concurrent tasks shows an overall speedup of 2.9x compared to Apache Spark. Individual query performance varies, with some queries showing substantially higher relative speedups than others, indicating that optimization efforts have been particularly effective in certain query patterns.
The codebase is organized into multiple Cargo feature-gated components. The ballista client crate includes a standalone mode feature for in-process scheduler and executor operation. The ballista-core crate provides Arrow IPC optimizations for shuffle performance and optional Spark compatibility mode. The ballista-scheduler component supports optional features including Substrait plan support, Prometheus metrics collection, execution graph visualization, Kubernetes Event Driven Autoscaling integration, REST API endpoints, and stage plan caching control. The ballista-executor crate includes the mimalloc memory allocator for performance optimization alongside Arrow IPC improvements. The ballista-cli component provides a terminal user interface for REST client interactions.
The project shares contributors with apache/datafusion, pingcap/tidb, and nvidia/cudf-spark, indicating cross-pollination with other major distributed data processing systems.
