Is AI Actually Making Your Team Better? Q&A Episode 4

In this Q&A episode of the Scrum.org Community Podcast, Eric Naberg and guest Daryl Fernandez dive into real questions from their webinar on 'Managing Your AI Teammate: Turning AI from Experiment to Strategic Partner.' The conversation centers on the challenge of measuring AI's impact on team productivity, value delivery, and overall performance. Daryl emphasizes that while traditional metrics like velocity and flow are still relevant, the key lies in understanding AI's role within specific workflows—ideation, development, validation, and deployment—and aligning measurement to those contexts. He stresses that success isn't about chasing universal benchmarks but about setting clear expectations through job descriptions, using AI to help define those metrics, and holding the hiring manager accountable for ensuring AI delivers tangible value. The discussion also explores the human side of AI integration: team disruption, fear of replacement, and the balance between quantitative data and qualitative feedback. Drawing on frameworks like Daniel Pink’s 'Drive,' they acknowledge that perceived productivity and morale matter, even when hard data is scarce. Ultimately, the episode positions AI not as a replacement but as a new team member requiring intentional onboarding, continuous evaluation, and empathetic leadership. Key takeaways include: (1) Use flow velocity and team-level metrics like work-in-progress and technical debt as baselines to measure AI’s impact; (2) Write AI job descriptions that define expected outcomes and use AI to help identify measurable KPIs; (3) The hiring manager—not the team—must be accountable for AI’s success; (4) Balance quantitative data with qualitative feedback, especially around team morale and perceived productivity; (5) Leverage AI to analyze and improve your own measurement practices; (6) Expect disruption when adding AI—treat it like onboarding a new team member; (7) Use evidence from reputable sources (McKinsey, Microsoft, Gartner) but adapt findings to your context; (8) Continuously ask AI: 'What should I measure to know if this is working?'