Modern language models capture a large body of factual knowledge. However, some facts can be incorrectly induced or become obsolete over time, resulting in factually incorrect generations. This has led to the development of various editing methods that allow updating facts encoded by the model. Evaluation of these methods has primarily focused on testing whether an individual fact has been successfully injected, and if similar predictions for other subjects have not changed. Here we argue that such evaluation is limited, since injecting one fact (e.g. ``Jack Depp is the son of Johnny Depp'') introduces a ``ripple effect'' in the form of additional facts that the model needs to update (e.g.``Jack Depp is the sibling of Lily-Rose Depp''). To address this issue, we propose a novel set of evaluation criteria that consider the implications of an edit on related facts. Using these criteria, we then construct RippleEdits, a diagnostic benchmark of 5K factual edits, capturing a variety of types of ripple effects. We evaluate prominent editing methods on RippleEdits, showing that current methods fail to introduce consistent changes in the model's knowledge. In addition, we find that a simple in-context editing baseline obtains the best scores on our benchmark, suggesting a promising research direction for model editing.
MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,北京通用人工智能研究院);Northeast Forestry University(东北林业大学);C-FAIR&school of software, Shandong University(山东大学软件学院C-FAIR);State Key Lab. for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);The University of Sydney(悉尼大学);Anhui Polytechnic University(安徽工程大学);Ludwig Maximilian University of Munich(慕尼黑大学)
ChartSync: A Benchmark for Visuo-Logical Cascading Chart Editing
Jianghan University(江汉大学);HiThink Research(海思睿研);University of Science and Technology of China(中国科学技术大学);Southeast University(东南大学);Zhejiang University(浙江大学)
National Engineering Research Center of Speech and Language Information Processing(语音与语言信息处理国家级工程研究中心);University of Science and Technology of China(中国科学技术大学);iFLYTEK Research(iFLYTEK研究院);University of California, Los Angeles(美国加州大学洛杉矶分校)
ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editing
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Harbin Institute of Technology(哈尔滨工业大学);Tianjin University(天津大学);Tsinghua University(清华大学)
REPAIR: Robust Editing via Progressive Adaptive Intervention and Reintegration
Engineering Center for Space Utilization of Chinese Academy of Sciences(中国科学院空间利用工程技术中心);University of Chinese Academy of Sciences(中国科学院大学);School of Computer Science and Engineering(计算机科学与工程学院);Northeastern University(东北大学);New York University(纽约大学);School of Computing(计算机学院);National University of Singapore(新加坡国立大学);The Chinese University of Hong Kong(香港中文大学);Technical University of Munich(慕尼黑技术大学)
CodeEditorBench: Evaluating Code Editing Capability of Large Language Models
Multimodal Art Projection Research Community(多模态艺术投影研究社区);University of Waterloo(滑铁卢大学);HKUST(香港科技大学);University of Manchester(曼彻斯特大学);Tongji University(同济大学);Vector Institute(矢量研究所)
MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
BIGAI(北京智源人工智能研究院);Shandong University(山东大学);University of Science and Technology of China(中国科学技术大学);Peking University(北京大学);Beijing Institute of Technology(北京理工大学)
Output Vector Editing for Memorization Mitigation in Large Language Models
Center for Information and Language Processing, LMU Munich(慕尼黑大学语言与信息处理中心);Department of Computer Science, University of Copenhagen(哥本哈根大学计算机科学系);Munich Center for Machine Learning(慕尼黑机器学习中心);Pioneer Centre for AI(人工智能先锋中心)