Principal Solutions Architect

Solo.io launches agentevals to solve agentic AI's "biggest unsolved problem" Solo.io launches agentevals, an open-source framework for evaluating agentic AI systems, announced at KubeCon Europe alongside a CNCF agent registry donation. So many agents, so little time to evaluate them. Solo.io's new projects can help. Agentic AI has blown up. These tools have become hotter than hot. But, there's this little problem. How do you evaluate them? Solo.io, best known for its cloud-native networking and API gateway platform, Gloo, has launched a new open-source initiative called agentevals. It's designed to help developers evaluate and benchmark "agentic AI" systems. Solo.io announced the project at KubeCon Europe in Amsterdam. According to Solo.io founder and CEO Idit Levine, autonomous AI systems pose new challenges for cloud operations. "Enterprises are experimenting with AI copilots and infrastructure agents, but they lack visibility into how these systems behave when given open-ended goals. gentBench helps teams understand not only what the models can do, but where their reasoning breaks down," Levine tells The New Stack. Levine continues, "Evaluation is the biggest unsolved problem in agentic infrastructure today. Organizations have frameworks for building agents, gateways for connecting them, and registries for governing them, but no consistent way to know whether an agent is actually reliable enough to trust in production." Aye, there's the rub. Agentevals provides a framework for testing the effectiveness of AI agents in real-world workflows, such as infrastructure automation, API orchestration, and service management. The goal is to give enterprise teams a standardized way to measure the reliability, latency, and success rates of autonomous agents before deploying them in production. "Evaluation is the biggest unsolved problem in agentic infrastructure today. Organizations have frameworks for building agents, gateways for connecting them, and registries for governing them, but no consistent way to know whether an agent is actually reliable enough to trust in production." The framework integrates with Solo.io's Gloo Platform and Envoy Proxy. This enables you to simulate multi-step tasks, such as configuring microservices, updating routing policies, or troubleshooting Kubernetes clusters under controlled conditions. Each run generates reproducible logs, metrics, and outcome data that can be used to compare different AI backends or agent architectures. The company claims that agentevals is the first benchmark designed to evaluate LLM-as-Agent across a diverse spectrum of different environments. To do this, the program relies on OpenTelemetry. "Whether you're using commercial APIs or open LLMs like Llama 3, you need transparent metrics for decision-making... We want agentevals to become a common reference point for the AI operations community." In addition, Solo says the open-source project is part of a broader effort to make AI-driven operations auditable and trustworthy. Levine says, "Whether you're using commercial APIs or open LLMs like Llama 3, you need transparent metrics for decision-making." We want agentevals to become a common reference point for the AI operations community." Agentevals is available on GitHub under the Apache 2.0 license. Solo.io plans to collaborate with other cloud-native vendors and AI research groups to expand the test library and integrate with common ML evaluation tools. In addition, Solo.io donated its agentregistry an AI-native open source registry for AI agents, MCP tools, and Agent Skills to the Cloud Native Computing Foundation (CNCF). This program enables you to standardize how AI capabilities are catalogued, discovered, and governed across the enterprise. As everyone and their uncle swiftly moves to Agentic computing, I expect both programs will find many fans.

Solo.io launches agentevals open source project, contributes agentregistry to CNCF. AMSTERDAM - Solo.io announced the launch of agentevals, a new open source project for evaluating and benchmarking agentic AI behavior, and the contribution of its agentregistry project to the Cloud Native Computing Foundation at KubeCon + CloudNativeCon Europe 2026. The company said the two initiatives address gaps in production reliability and governance for agentic AI workloads. Solo.io said agentevals uses OpenTelemetry to capture and correlate individual invocations from distributed agentic interactions, then scores them against golden evaluation sets using an extensible evaluation engine. The project supports offline and online evaluation modes, ships with built-in evaluators for trajectory matching and LLM-as-judge scoring, and includes a CLI, web interface and Model Context Protocol server. The company said the tool works with any model and framework that emits OpenTelemetry spans, with no requirement for agent reruns. "Evaluation is the biggest unsolved problem in agentic infrastructure today," said Idit Levine, founder and CEO of Solo.io. "Organizations have frameworks for building agents, gateways for connecting them, and registries for governing them, but no consistent way to know whether an agent is actually reliable enough to trust in production." The agentregistry project, originally introduced by Solo.io in November 2025, provides a centralized registry where AI agents, MCP tools and agent skills are catalogued, discovered and governed. Solo.io said the contribution to CNCF governance will enable community growth alongside kagent, a CNCF sandbox project for running AI agents in Kubernetes, and agentgateway, which is housed in the Linux Foundation. The registry integrates with Kubernetes, AWS AgentCore and Google Vertex AI for deployment, and includes runtime discovery to detect agents deployed outside governed workflows.