Large language models (LLMs) are increasingly capable of completing knowledge intensive tasks by recalling information from a static pretraining corpus. Here we are concerned with LLMs in the context of evolving data requirements. For instance: batches of new data that are introduced periodically; subsets of data with user-based access controls; or requirements on dynamic removal of documents with guarantees that associated knowledge cannot be recalled. We wish to satisfy these requirements while at the same time ensuring a model does not forget old information when new data becomes available. To address these issues, we introduce AdapterSwap, a training and inference scheme that organizes knowledge from a data collection into a set of low-rank adapters, which are dynamically composed during inference. Our experiments demonstrate AdapterSwap's ability to support efficient continual learning, while also enabling organizations to have fine-grained control over data access and deletion.
Network Digital Untwinning: Towards Backward Optimization of Digital Twins
North Carolina State University(北卡罗来纳州立大学);University of Maryland(马里兰大学);University of Louisville(路易斯维尔大学);University of Michigan(密歇根大学);University of Miami(迈阿密大学)
A Certified Unlearning Approach without Access to Source Data
Department of Electrical and Computer Engineering, University of California, Riverside, CA, USA(加州大学河滨分校电子与计算机工程系);Brookhaven National Laboratory, Upton, NY, USA(布鲁克海文国家实验室)
Department of Electrical Engineering and Computer Science University of California Irvine(电气工程与计算机科学系加州大学伊维特分校);Department of Electrical and Computer Engineering University of Rochester(电气与计算机工程系罗切斯特大学)
Certifying the Right to Be Forgotten: Primal-Dual Optimization for Sample and Label Unlearning in Vertical Federated Learning
College of Computing and Data Science (CCDS), Nanyang Technological University, Singapore and Digital Trust Centre (DTC), Singapore(南洋理工大学计算与数据科学学院(CCDS)及新加坡数字信任中心(DTC));College of Computing and Data Science (CCDS), Nanyang Technological University, Singapore(南洋理工大学计算与数据科学学院(CCDS));School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院)