Horovod is a distributed deep learning training framework designed to simplify and accelerate training across multiple GPUs and machines for TensorFlow, Keras, PyTorch, and Apache MXNet. The framework is hosted by the LF AI & Data Foundation and addresses a core challenge in deep learning: making it straightforward to scale single-GPU training scripts to run efficiently across many GPUs in parallel without requiring extensive code modifications.

The primary motivation behind Horovod stems from Uber's internal experience finding that the MPI model provides a more straightforward approach than previous solutions like Distributed TensorFlow with parameter servers. Once a training script is written for Horovod, it can execute on a single GPU, multiple GPUs, or multiple hosts without any further code changes. The framework achieves significant performance gains, with benchmarks on 128 servers containing 4 Pascal GPUs each connected by RoCE-capable 25 Gbit/s networks demonstrating 90 percent scaling efficiency for both Inception V3 and ResNet-101, and 68 percent scaling efficiency for VGG-16.

Horovod's core principles are based on MPI concepts including size, rank, local rank, allreduce, allgather, broadcast, and alltoall operations. Using the framework requires minimal modifications to existing training code: initializing Horovod with hvd.init(), pinning GPUs to processes by local rank, scaling learning rates by the number of workers, wrapping optimizers in hvd.DistributedOptimizer, broadcasting initial variable states from rank 0 to all other processes, and ensuring checkpoints are saved only on worker 0.

The repository supports multiple deployment scenarios including direct execution on machines with multiple GPUs via the horovodrun command, distributed training across multiple servers, execution within Docker containers, deployment on Kubernetes clusters using Helm Charts and the Kubeflow MPI Operator, integration with Spark for distributed training, execution on Ray clusters, containerization through Singularity, and deployment on LSF HPC clusters such as Summit.

Installation requires CMake and appropriate compiler versions depending on the deep learning framework used. For GPU support with NCCL, users set the HOROVOD_GPU_OPERATIONS environment variable during installation. The framework also supports MPI-based execution and provides Conda and Docker installation options.

According to GitGenius activity tracking across 52 monitored items, the repository shows a median issue and pull request response latency of 402.3 hours with a mean of 12522.8 hours, indicating variable response times across different types of requests. Bug reports represent the most active issue category with 29 tracked items, followed by enhancement requests with 8 items and questions with 6 items. The most active contributors tracked include EnricoMi and laytonjbgmail with 5 events each, and Parvez-Khan-1 with 4 events. The repository shares overlapping contributors with microsoft/vscode, microsoft/typescript, and rust-lang/rust, suggesting cross-project collaboration among maintainers.