Large Language Models (LLMs) have demonstrated impressive capability in different tasks and are bringing transformative changes to many domains. However, keeping the knowledge in LLMs up-to-date remains a challenge once pretraining is complete. It is thus essential to design effective methods to both update obsolete knowledge and induce new knowledge into LLMs. Existing locate-and-edit knowledge editing (KE) method suffers from two limitations. First, the post-edit LLMs by such methods generally have poor capability in answering complex queries that require multi-hop reasoning. Second, the long run-time of such locate-and-edit methods to perform knowledge edits make it infeasible for large scale KE in practice. In this paper, we explore Parameter-Efficient Fine-Tuning (PEFT) techniques as an alternative for KE. We curate a more comprehensive temporal KE dataset with both knowledge update and knowledge injection examples for KE performance benchmarking. We further probe the effect of fine-tuning on a range of layers in an LLM for the multi-hop QA task. We find that PEFT performs better than locate-and-edit techniques for time-sensitive knowledge edits.
Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse
Shandong University（山东大学）；School of Computer Science, Beijing University of Posts and Telecommunications（北京邮电大学计算机学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
Context-Robust Knowledge Editing for Language Models
Graduate School of Data Science, Seoul National University（首尔国立大学数据科学研究生院）；School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology（全州科学技术院电子工程与计算机科学学院）
Consistency-Aware Editing for Entity-level Unlearning in Language Models
Shanxi University, China（山西大学）；Cardiff University, UK（卡地夫大学）；Singapore University of Technology and Design, Singapore（新加坡科技设计大学）；ILCC, School of Informatics, University of Edinburgh, Edinburgh, UK（爱丁堡大学信息学院ILCC）
DualEdit: Dual Editing for Knowledge Updating in Vision-Language Models
Harvard University（哈佛大学）；City University of Hong Kong（香港城市大学）；Amazon（亚马逊）；Fudan University（复旦大学）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能关键实验室，上海交通大学）
Are We Evaluating the Edit Locality of LLM Model Editing Properly?
National University of Singapore（新加坡国立大学）；Huazhong University of Science（华中科技大学）；Central China Normal University；University of Maryland, College Park（马里兰大学学院市分校）
WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
Tübingen AI Center, University of Tübingen（图宾根人工智能中心，图宾根大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Technical University, Munich（慕尼黑技术大学）；University of Virginia（弗吉尼亚大学）
We investigate input-conditioned hypernetworks for multi-tasking in NLP, generating parameter-efficient adaptations for a decoder using a hypernetwork conditioned on the output of an encoder. This approach produces a unique decoder adaptation for every input instance, allowing the network a larger degree of flexibility than prior work that only produces one decoder adaptation per task. We apply our method to sequence classification tasks, extractive QA, and summarisation and find that it surpasses previous parameter efficient fine-tuning methods and often outperforms fully finetuning the under…
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation
Department of Computer Science（计算机科学系）；University of North Carolina at Chapel Hill（北卡罗来纳大学 Chapel Hill 分校）；School of Artificial Intelligence and Data Science（人工智能与数据科学学院）；University of Science and Technology of China（中国科学技术大学）
Tsinghua University（清华大学）；Sogou Inc.（搜狗公司）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；Institute for AI Industry Research, Tsinghua University（清华大学智能产业研究院）
Multi-source sequence generation (MSG) is an important kind of sequence generation tasks that takes multiple sources, including automatic post-editing, multi-source translation, multi-document summarization, etc. As MSG tasks suffer from the data scarcity problem and recent pretrained models have been proven to be effective for low-resource downstream tasks, transferring pretrained sequence-to-sequence models to MSG tasks is essential. Although directly finetuning pretrained models on MSG tasks and concatenating multiple sources into a single long sequence is regarded as a simple method to tra…
Merging Methods for Multilingual Knowledge Editing for Large Language Models: An Empirical Odyssey
Department of Computer Science and Engineering, POSTECH（POSTECH计算机科学与工程系）；Designovel Co., Ltd.（Designovel公司）；LLSOLLU；Graduate School of Artificial Intelligence, POSTECH（POSTECH人工智能研究生院）
MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models
University of Science and Technology of China（中国科学技术大学）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，北京通用人工智能研究院）；Northeast Forestry University（东北林业大学）；C-FAIR&school of software, Shandong University（山东大学软件学院C-FAIR）；State Key Lab. for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；The University of Sydney（悉尼大学）；Anhui Polytechnic University（安徽工程大学）；Ludwig Maximilian University of Munich（慕尼黑大学）
Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers
University of Science and Technology of China（中国科学技术大学）；School of Computer Science, Fudan University（复旦大学计算机学院）；Tsinghua University（清华大学）；Hefei University of Technology（合肥工业大学）
Precise In-Parameter Concept Erasure in Large Language Models
Blavatnik School of Computer Science and AI, Tel Aviv University（塔尔斯基大学计算机科学与人工智能学院）；New York University（纽约大学）；University of Oxford & WhiteBox（牛津大学及WhiteBox）
