triton-inference-server/server

Triton Inference Server is an open source inference serving platform developed by NVIDIA that enables deployment of AI models across multiple frameworks and hardware platforms. The server supports deep learning frameworks including TensorRT, PyTorch, ONNX, and OpenVINO, as well as machine learning frameworks like RAPIDS FIL. It can run on NVIDIA GPUs, x86 and ARM CPUs, and AWS Inferentia, making it suitable for cloud, data center, edge, and embedded deployment scenarios.

The platform provides multiple execution engines called backends, with support for concurrent model execution and various batching strategies. Dynamic batching and sequence batching capabilities allow optimization for different query patterns, while implicit state management supports stateful models. The server enables model pipelining through ensemble models and Business Logic Scripting, allowing teams to create complex inference workflows. A Backend API allows developers to add custom backends and preprocessing or postprocessing operations, with support for Python-based backends.

Triton Inference Server exposes inference capabilities through HTTP/REST and gRPC protocols based on the community-developed KServe protocol. For in-process use cases, it provides C and Java APIs that allow direct linking into applications. The platform includes comprehensive metrics for monitoring GPU utilization, server throughput, and latency. Model configuration and repository management features enable explicit control over which models are available, with tools like the Model Analyzer supporting optimization through profiling.

The repository shows active maintenance and community engagement. The repository shares contributors with microsoft/vscode, microsoft/typescript, and rust-lang/rust, suggesting cross-project collaboration within the broader software ecosystem.

The codebase is primarily written in Python and is distributed under the BSD 3-Clause license. The current release version is 2.70.0, corresponding to the 26.06 container release on NVIDIA GPU Cloud. The main branch tracks development progress toward the next release. Documentation covers building and deploying Triton through Docker containers or from source, preparing models for serving, and configuring the server for various use cases. The project includes tutorials and examples for popular models like ResNet, BERT, and DLRM, with deployment examples for Kubernetes and Helm on GCP, AWS, and NVIDIA FleetCommand. Triton Inference Server is part of NVIDIA AI Enterprise and offers enterprise support through NVIDIA global support channels.