Open Source Self-Driving with Comma AI

In this episode of Practical AI, host Daniel Whitenack and co-host Chris Benson welcome Harold Schaefer, CTO of Comma AI (formerly Kama AI), to discuss the company's open-source self-driving technology, OpenPilot. Schaefer traces the evolution of OpenPilot from a niche hobbyist project to a widely adopted autonomy stack, now used in over 50% of miles driven by users on highways. The conversation dives into the technical architecture of OpenPilot, emphasizing its end-to-end machine learning approach—where raw video input directly produces steering and acceleration commands—unlike traditional systems that rely on intermediate steps like lane detection or traffic light recognition. A key innovation highlighted is the use of a learned simulation (a 'world model') trained via diffusion models to generate photorealistic, physically accurate video sequences that enable the system to learn recovery from mistakes, such as drifting off the lane. The system runs entirely on-device, with training and simulation handled in a centralized data center. Schaefer also discusses the challenges of working with legacy car hardware, the importance of open source for community-driven car support, and the company’s vision for expanding into indoor robotics and general-purpose AI agents. The episode closes with reflections on the future of accessible, open-source robotics that empower users rather than lock them into corporate ecosystems. Key takeaways include: 1) End-to-end autonomy via learned simulation enables robust, scalable training without manual labeling; 2) Open source is essential for community-driven hardware support and user trust; 3) On-device inference ensures privacy and reliability; 4) Future advancements will come from solving low-level control, reinforcement learning, and continual learning; 5) The long-term goal is to create simple, open-source AI tools that make daily life easier—like smarter vacuums and dishwashers—without corporate lock-in. The episode maintains a positive, forward-looking tone, celebrating open innovation and practical AI progress.