It's 2026, and We're Still Talking Evals

In this episode of MLOps.community, the host and guest dive deep into the evolving challenges of evaluating AI agents in 2026, emphasizing that evaluation (evals) is not a one-time testing phase but a continuous, foundational practice embedded from the very beginning of product development. The discussion highlights how traditional software testing falls short for LLMs due to their non-deterministic behavior, context drift, and emergent failure modes—such as agents losing track after repeated success or failing in unexpected ways when users ask open-ended questions like 'surprise me.' The speakers stress that effective evals require simulating diverse user personas, designing intentional failure scenarios, and conducting thorough error analysis to uncover hidden pain points. They critique over-reliance on generic metrics like accuracy and advocate for business-aligned, context-specific evaluation frameworks tied to real-world outcomes like conversion and user satisfaction. The episode also explores the limitations of current eval tooling, with the host expressing frustration over tools that are slow, inflexible, and poorly designed for multi-turn analysis, ultimately favoring custom-built solutions that enable faster iteration and deeper insight. A recurring theme is that evals should be a team-wide, iterative discipline—not a chore—because they directly impact product quality and user trust.