Apache DataFusion is an extensible query engine written in Rust that leverages Apache Arrow as its in-memory columnar format. The project provides libraries and binaries for developers building fast and feature-rich database and analytic systems, with the ability to customize the engine for particular workloads. The core engine offers SQL and DataFrame APIs, a full query planner, a columnar streaming multi-threaded vectorized execution engine, and support for partitioned data sources. Built-in support includes CSV, Parquet, JSON, and Avro formats, with extensive customization capabilities at nearly all points including data sources, query languages, functions, and custom operators.
The repository maintains active development with significant community engagement. According to GitGenius tracking data, the project has processed 5519 issues and pull requests with a median response latency of 0.0 hours and a mean latency of 4061.1 hours. The most frequently applied issue labels are enhancement with 2682 occurrences, bug with 1791 occurrences, and good first issue with 429 occurrences, indicating a healthy balance between feature development and bug fixes with explicit pathways for new contributors. The project's top contributor alamb has logged 8094 events, followed by Jefffrey with 854 events and adriangb with 825 events, demonstrating concentrated expertise among key maintainers.
DataFusion's ecosystem extends beyond the core Rust implementation through multiple language bindings and specialized projects. DataFusion Python provides a Python interface for SQL and DataFrame queries, DataFusion Java offers Java bindings, and DataFusion Comet serves as an accelerator for Apache Spark based on the DataFusion engine. The project is classified across multiple analytical domains including analytics platforms, data integration, data processing, scalable queries, distributed computing, streaming data, ETL workflows, batch processing, query optimization, and real-time analytics. GitGenius identifies overlapping contributors with trinodb/trino, lance-format/lance, and apache/datafusion-ballista, indicating cross-pollination within the broader data processing ecosystem.
The crate provides extensive customization through configurable features. Default features include nested expressions for working with complex types, compression support for multiple formats, cryptographic and datetime functions, encoding functions, Parquet and SQL support, regular expression functions, Unicode-aware operations, and logical plan unparsing. Optional features add Apache Avro support, backtrace information in error messages, Parquet Modular Encryption, and serialization capabilities. The project follows Apache Software Foundation licensing under the Apache License 2.0 and maintains a committed Cargo.lock file with regular dependency updates via Dependabot, ensuring reproducible builds and managed dependency evolution.
