We Cut LLM Latency by 70% in Production

In this episode of MLOps.community, a senior engineering leader shares their journey of building and optimizing AI in production at an HR tech company, achieving up to 70% latency reduction through strategic infrastructure and model optimization. Starting as a non-AI expert, they embraced a 'learning alongside the team' mindset, leading to the development of a 'flywheel framework' for AI adoption that balances business impact with technical feasibility. Key technical wins include using TensorRT LLM for model optimization, implementing scheduled and dynamic GPU scaling based on usage patterns, and baking models into container images to eliminate cold start delays. The speaker emphasizes the importance of focusing on a 'sweet spot' in the AI pyramid—balancing cost, performance, latency, and throughput—rather than optimizing all metrics simultaneously. They also discuss the evolution of AI adoption within their engineering team, from skepticism to near-universal use, and the cultural shift toward responsible AI with guardrails, human-in-the-loop workflows, and a self-hosted AI platform that supports both product and internal agentic engineering. The episode concludes with reflections on the future of self-hosted coding agents and the critical role of transparency, data governance, and customer trust in scaling AI responsibly.