Events

Language-Driven Learning for Interactive Robotics

Building and deploying broadly capable robots requires systems that can efficiently learn from and work with people. To achieve this, robots must balance capability — the fundamental tools necessary to enable real-world deployment — and sustainability — the ability to grow and adapt through human feedback. In this talk, Siddharth Karamcheti will motivate language-driven learning to tackle these axes, providing robots with better abstractions for perception, action, and human-robot interaction. Towards capability, Karamcheti will present Voltron, his approach for using language to learn visual representations that can be efficiently adapted for diverse robotics tasks. Building on these ideas, Karamcheti will discuss Prismatic, his experimental framework for developing visually-conditioned language models and vision-language-action policies at scale. Towards sustainability, Karamcheti will next introduce Vocal Sandbox, a new framework that integrates these models to develop collaborative robots that can work alongside human partners, using language to express uncertainty and learn new behaviors from real-time interactions. Finally, Karamcheti will conclude with open challenges for enhancing both the capability and sustainability of modern robots, with directions for future work.