Apache DataFusion is an extensible query engine written in Rust that leverages Apache Arrow as its in-memory columnar format. The project provides libraries and binaries for developers building fast and feature-rich database and analytic systems, with the ability to customize the engine for particular workloads. The core engine offers SQL and DataFrame APIs, a full query planner, a columnar streaming multi-threaded vectorized execution engine, and support for partitioned data sources. Built-in support includes CSV, Parquet, JSON, and Avro formats, with extensive customization capabilities at nearly all points including data sources, query languages, functions, and custom operators.
The repository maintains active development with significant community engagement.
DataFusion's ecosystem extends beyond the core Rust implementation through multiple language bindings and specialized projects. DataFusion Python provides a Python interface for SQL and DataFrame queries, DataFusion Java offers Java bindings, and DataFusion Comet serves as an accelerator for Apache Spark based on the DataFusion engine. The project is classified across multiple analytical domains including analytics platforms, data integration, data processing, scalable queries, distributed computing, streaming data, ETL workflows, batch processing, query optimization, and real-time analytics.
The crate provides extensive customization through configurable features. Default features include nested expressions for working with complex types, compression support for multiple formats, cryptographic and datetime functions, encoding functions, Parquet and SQL support, regular expression functions, Unicode-aware operations, and logical plan unparsing. Optional features add Apache Avro support, backtrace information in error messages, Parquet Modular Encryption, and serialization capabilities. The project follows Apache Software Foundation licensing under the Apache License 2.0 and maintains a committed Cargo.lock file with regular dependency updates via Dependabot, ensuring reproducible builds and managed dependency evolution.
