ModelMesh Serving is a Kubernetes controller written in Go that manages ModelMesh, a general-purpose model serving management and routing layer. The repository has been archived with users directed to the main KServe repository for ongoing development. The project functions as a central orchestration point for deploying and managing machine learning models at scale across Kubernetes clusters.
The architecture comprises several interconnected components distributed across multiple repositories. The core consists of the ModelMesh Serving controller itself, which this repository contains, and the ModelMesh containers that handle model placement orchestration and request routing. The modelmesh-runtime-adapter serves as an intermediary layer, running in model serving pods to bridge ModelMesh with third-party model servers. This adapter also incorporates puller logic responsible for retrieving models from storage and managing their lifecycle through load and unload operations.
ModelMesh Serving provides out-of-the-box integration with multiple established model serving runtimes including Nvidia's Triton Inference Server, Seldon's MLServer, OpenVINO Model Server, and TorchServe. The system supports extensibility through custom ServingRuntime resources, allowing operators to integrate additional model servers beyond the default offerings. A KServe V2 REST Proxy component translates RESTful HTTP APIs into gRPC, enabling inference requests using the KServe V2 REST Predict Protocol to reach models that only support the V2 gRPC Predict Protocol.
The project relies on helper libraries including kv-utils for key-value store operations abstracted over etcd and Zookeeper, and litelinks-core, an RPC and service discovery library based on Apache Thrift used for internal ModelMesh communications. Issues across all components are tracked centrally in this repository despite the distributed nature of the codebase.
The project is classified across multiple domains reflecting its comprehensive scope: model management, deployment, scalability, Kubernetes orchestration, inference serving, microservices architecture, MLOps workflows, and API gateway functionality. This classification underscores ModelMesh Serving's role as a sophisticated platform for managing machine learning model deployment and serving at enterprise scale within containerized environments.
