Why Software Is Never “Done” Anymore with Iccha Sethi, SVP of Engineering at Vanta

In this episode of Modern CTO, Joel Beasley sits down with Iccha Sethi, Senior Vice President of Engineering at Vanta, to explore why software is no longer 'done' in the modern era. Sethi explains that the rise of AI and agentic features has fundamentally changed software development, introducing constant drift and non-determinism from underlying models—vendors like OpenAI and Anthropic continuously evolve their models without clear announcements. This means engineering teams can never truly 'ship and forget' a feature. Instead, they must maintain ongoing quality through continuous evaluation, both offline using golden datasets and online via real customer data. Vanta has built a robust AI evaluation maturity model with five levels, including traceability, golden data, offline and online evaluation, experimentation, and self-improving feedback loops. Sethi emphasizes that engineering leadership now requires a new mindset: staying ahead of drift, managing AI adoption responsibly, and fostering a culture of continuous learning and improvement. The conversation also dives into practical tools and leadership philosophies. Sethi shares how Vanta uses SPAN to track onboarding success, investment mix, and AI tool adoption—especially to monitor 'AI slop' (overly large PRs). She reflects on her career journey from Atlassian to GitHub to Vanta, highlighting how each role offered unique learning opportunities and shaped her leadership style. Key themes include the importance of customer exposure, giving direct feedback with empathy, and adopting proven operational frameworks from mentors. Sethi’s advice to aspiring leaders centers on observing great behaviors and integrating them into your own toolkit—building a personal leadership 'tool belt' over time.