Events

Rethinking Data Use in Large Language Models

Large language models (LMs) such as ChatGPT have revolutionized natural language processing and artificial intelligence more broadly. In this talk, Sewon Min will discuss her research on understanding and advancing these models, centered around how they use the very large text corpora they are trained on. First, Min will describe her efforts to understand how these models learn to perform new tasks after training, demonstrating that their so-called in context learning capabilities are almost entirely determined by what they learn from the training data. Next, Min will introduce a new class of LMs—nonparametric LMs—that repurpose this training data as a data store from which they retrieve information for improved accuracy and updatability. Min will describe her work on establishing the foundations of such models, including one of the first broadly used neural retrieval models and an approach that simplifies a traditional, two-stage pipeline into one. Min will also discuss how nonparametric models open up new avenues for responsible data use, e.g., by segregating permissive and copyrighted text and using them differently. Finally, Min will envision the next generation of LMs we should build, focusing on efficient scaling, improved factuality, and decentralization.