TruLens is a Python-based evaluation and tracking framework designed for LLM experiments and AI agents. The project provides systematic evaluation capabilities that go beyond informal testing, enabling developers to understand performance, identify failure modes, and iteratively improve their applications as they develop prompts, models, retrievers, and knowledge sources.
The framework is built on OpenTelemetry-based tracing, making every function call, LLM generation, retrieval, and tool invocation visible as structured OTEL spans. This architecture ensures interoperability with existing observability infrastructure, allowing users to export traces to Jaeger, Grafana Tempo, Datadog, or any OTLP-compatible backend. The instrumentation is described as fine-grained and stack-agnostic, providing comprehensive visibility across different application architectures.
TruLens introduces seven purpose-built evaluators specifically designed for agentic systems. These evaluators measure distinct aspects of agent behavior including LogicalConsistency for reasoning coherence and hallucination detection, ExecutionEfficiency for identifying redundant steps and wasted computation, PlanAdherence for tracking whether execution followed stated plans, PlanQuality for assessing intrinsic strategy quality, ToolSelection for verifying correct tool choices, ToolCalling for validating argument validity and output interpretation, and ToolQuality for measuring external tool reliability. The framework also supports Model Context Protocol tool calls through dedicated MCP span types that capture tool names, arguments, output, and latency.
The evaluation system operates in multiple modes, supporting batch evaluation on existing data, inline evaluation alongside running applications, and offline evaluation workflows. A flexible Selector API allows developers to target any span attribute for evaluation. Core concepts documented in the project include Feedback Functions, the RAG Triad, and Honest, Harmless and Helpful Evals frameworks.
TruLens supports a broad range of LLM providers through dedicated packages including OpenAI and Azure OpenAI, LiteLLM for Anthropic and other providers, Google Gemini, AWS Bedrock, Snowflake Cortex, HuggingFace, and LangChain models. The project is available on PyPI and includes interactive examples runnable in Google Colab.
The project shares contributors with major ecosystem projects including run-llama/llama_index, mastra-ai/mastra, and langchain-ai/langchain, indicating integration within the broader LLM application development ecosystem. GitGenius classifies the repository across multiple domains including AI model transparency, fairness assessment, bias detection, AI risk management, explainable AI techniques, and algorithmic accountability.
