Apache Airflow is a Python-based platform for programmatically authoring, scheduling, and monitoring workflows. The project enables users to define workflows as directed acyclic graphs (DAGs) in code, making them maintainable, versionable, testable, and collaborative. The Airflow scheduler executes tasks on worker arrays while respecting specified dependencies, and the platform provides command-line utilities and a rich user interface for visualizing pipelines, monitoring progress, and troubleshooting issues in production environments.
The platform is designed for workflows that are mostly static and slowly changing, where the DAG structure remains similar from one run to the next. Airflow emphasizes that tasks should ideally be idempotent and should not pass large quantities of data between tasks, though metadata can be exchanged using the XCom feature. For high-volume, data-intensive operations, the platform recommends delegating to external services. While Airflow is not a streaming solution, it is frequently used to process real-time data by pulling from streams in batches.
Three core principles guide Airflow's design: Dynamic pipelines are defined in code, enabling dynamic DAG generation and parameterization. The framework is extensible with a wide range of built-in operators and can be extended to fit specific needs. Flexibility is achieved through Jinja templating engine integration, allowing rich customizations.
The platform supports Python versions 3.10 through 3.14 on the main development version and stable 3.2.0 release, with the deprecated 2.11.2 version supporting 3.10 through 3.12. Both AMD64 and ARM64 platforms are supported. Kubernetes compatibility spans versions 1.30 through 1.35 for current releases. Database support includes PostgreSQL 14-18, MySQL 8.0 and 8.4, and SQLite 3.15.0 and later, though SQLite is recommended only for development and testing, not production use.
Airflow is commonly used for data processing tasks in data engineering, data integration, ETL and ELT workflows, machine learning operations, and general workflow orchestration. The platform is distributed via PyPI, Docker, and Artifact HUB, with significant community adoption evidenced by contributor counts and commit activity metrics. The project operates under the Apache License 2.0 and maintains semantic versioning with defined version life cycles and support policies for Python and Kubernetes versions.
