GuideLLM is a benchmarking and evaluation platform designed specifically for assessing language model performance under real-world production conditions. Written in Python, it enables teams to simulate end-to-end interactions with OpenAI-compatible and vLLM-native servers, generate realistic workload patterns, and produce detailed reports that reveal system behavior, resource requirements, and operational limits. The platform supports both real and synthetic datasets, multimodal inputs including text, images, audio, and video, and flexible execution profiles to give engineering and ML teams a consistent framework for evaluating model behavior and planning capacity.
The core functionality centers on SLO-aware benchmarking that captures complete latency and token-level statistics. GuideLLM measures critical metrics like time-to-first-token (TTFT), inter-token latency (ITL), and end-to-end behavior across full distributions rather than simple averages. It generates realistic, configurable traffic patterns through synchronous, concurrent, and rate-based modes, including reproducible sweeps to identify safe operating ranges. The platform produces standardized, exportable reports in multiple formats—JSON, CSV, and HTML—enabling consistency across teams and workflows while supporting integration with dashboards and regression tracking systems.
GuideLLM distinguishes itself from competing tools by focusing exclusively on LLM-specific workloads rather than generic endpoint benchmarking. Unlike simpler scripts or research-only pipelines, it delivers high-throughput benchmarking through multiprocessing, threading, and async execution. The platform uses standard Python interfaces and HuggingFace datasets instead of custom formats, making it accessible for everyday engineering tasks. It supports multiple backends including OpenAI-compatible endpoints and vLLM-native servers, with endpoints covering completions, chat completions, audio translation, and audio transcription.
Recent additions to the platform include a new CLI interface with improved configuration and validation, backends for in-process vLLM Python API and websocket audio transcription, multi-turn conversation capabilities for benchmarking chat agents, full tool calling support on both client and server sides, synthetic video and image datasets for controlled experimentation, and replay of Mooncake trace files for realistic load testing. The platform also supports benchmarking Geospatial LLMs. Active development areas include replay of OTEL and WEKA trace files, improved scenarios for standard workflows, ability to stack scenario files for complex benchmarking, constraint overrides for individual benchmarks, and a gRPC backend for vLLM-native servers.
The platform requires Python 3.10 through 3.13 and runs on Linux or MacOS. Installation is available through PyPI, source installation, or container images from ghcr.io/vllm-project/guidellm. Output from benchmark runs includes console summaries with high-level statistics, benchmarks.json containing the authoritative record with full configuration and per-request timings, benchmarks.csv for compact tabular reporting suitable for spreadsheets and BI tools, and benchmarks.html for visual exploration with charts of latency distributions and throughput behavior.
