The Distributed Workloads repository is part of the Open Data Hub project and provides artifacts for managing distributed workloads on OpenShift AI. Written primarily in Go, the repository focuses on enabling large-scale machine learning and data processing tasks across Kubernetes clusters. The project is classified across multiple domains including frameworks, big data analytics, workload management, cluster management, resource orchestration, and distributed computing, reflecting its role as infrastructure for coordinating complex computational tasks.
The repository includes practical examples demonstrating real-world use cases for distributed machine learning workloads. These examples cover fine-tuning large language models using Ray and DeepSpeed on OpenShift AI, fine-tuning Stable Diffusion models with DreamBooth and Ray Train, and performing hyperparameter optimization through Ray Tune integrated with OpenShift AI. These examples serve as templates for users implementing similar distributed training and optimization workflows.
The core functionality centers on integration testing for distributed workload components. The testing infrastructure requires administrative access to an OpenShift cluster, with support for local development using CRC, along with installed OpenDataHub or RHOAI with all Distributed Workload components enabled. The test suite is written in Go 1.21 and provides configurable environment variables for controlling test behavior, including output directories and timeout durations for short, medium, and long-running tasks.
The repository supports multiple specialized test suites targeting different aspects of the distributed workloads stack. The Ray integration tests allow customization of Ray cluster images, with a default image of quay.io/modh/ray:2.47.1-py312-cu128. The fms-hf-tuning test suite handles model training workflows with support for GPU-accelerated testing, including integration with HuggingFace tokens for accessing gated models and PersistenceVolumeClaim resources for managing model data. Tests can upload trained models to S3-compatible storage using configurable endpoint and credential environment variables.
Additional test suites cover ODH integration testing with Workbench environments, requiring notebook user credentials and custom notebook images. The Kubeflow Trainer v2 and TrainingHub SDK test suites support downloading models and datasets from S3-compatible storage, with optional prefixes for supervised fine-tuning, open-source fine-tuning, and LoRA training data. The repository includes specific documentation for pre-staging S3 data in disconnected environments, enabling testing in air-gapped deployments.
Tests are executed as standard Go unit tests, with support for test tiering through the TEST_TIER environment variable, allowing users to run Smoke, Tier1, Tier2, Tier3, Pre-Upgrade, and Post-Upgrade test categories based on their validation requirements.
