kubeflow/pipelines

Kubeflow Pipelines is a machine learning workflow orchestration system built on Kubernetes that enables users to construct, deploy, and manage end-to-end ML workflows. Written primarily in Python, it serves as a core component of the Kubeflow ML toolkit and provides the Kubeflow Pipelines SDK for building reusable ML pipelines that can run on Kubernetes clusters with portability and scalability.

The repository addresses three primary objectives outlined in its documentation: enabling end-to-end orchestration of ML pipelines, facilitating easy experimentation through trial and experiment management, and promoting component and pipeline reusability to accelerate solution development. Users can install Kubeflow Pipelines either as part of the complete Kubeflow Platform or as a standalone service. The system has evolved to support modern Kubernetes environments by defaulting to the Emissary Executor from version 1.8 onward, which is container runtime agnostic and works with any Kubernetes-supported container runtime, replacing the deprecated Docker container runtime that was incompatible with Kubernetes 1.20 and later.

The project maintains compatibility with specific dependency versions including Argo Workflows v3.7 and v4.0, and MySQL v8. Under the hood, Kubeflow Pipelines leverages Argo Workflows to orchestrate Kubernetes resources, a relationship the project acknowledges and credits in its documentation. The repository includes comprehensive documentation covering SDK usage, API specifications, and Python SDK references, along with architectural details and developer guides for contributors.

The project maintains an active community with biweekly meetings held on Wednesdays from 10-11 AM PST and a dedicated Slack channel on the Cloud Native Computing Foundation workspace. The repository includes optional developer tooling through a just command runner that wraps existing make targets for convenience, though all CI and release workflows continue to use the underlying make infrastructure. Contributing guidelines and developer setup documentation are provided to facilitate community participation in the project's ongoing development and maintenance.