Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, Arjun Guha
A significant amount of research is focused on developing and evaluating large language models for a variety of code synthesis tasks. These include synthesizing code from natural language, synthesizing tests from code, and synthesizing explanations of code. In contrast, the behavior of instructional code editing with LLMs is understudied. These are tasks in which the model is provided a block of code and an instruction to modify the code. The editing instruction may ask for a feature to be added or removed, describe a bug and ask for a fix, or ask for a different kind of solution. We introduce a carefully crafted benchmark of code editing tasks and use it to evaluate several cutting edge LLMs. Our evaluation exposes a significant gap between the capabilities of state-of-the-art open and closed models. For example, even GPT-3.5-Turbo is better than the best open model at code editing tasks. We also introduce a new, carefully curated, permissively licensed training dataset of code editing tasks coupled with natural language instructions. Using this training dataset, we show that we can fine-tune open Code LLMs to significantly improve their code editing capabilities, closing the gap between open and closed models. All code, data, and models are available at https://github.com/nuprl/CanItEdit.
Constraining Sequential Model Editing with Editing Anchor Compression
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China(中国科学技术大学语音及语言信息处理国家工程研究中心);Zhejiang University(浙江大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs
School of Computer Science and Technology, Tianjin University(天津大学计算机科学与技术学院);Kexin Technology(可心科技);University of Chinese Academy of Sciences(中国科学院大学);Tencent Hunyuan(腾讯混元);College of Computer Science, Chongqing University(重庆大学计算机学院)
Potential and Challenges of Model Editing for Social Debiasing
Zhejiang University School of Engineering(浙江大学工程学院);Westlake University(西湖大学);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院先进技术研究所)
Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language Models
Hefei University of Technology(合肥工业大学);University of Science and Technology of China(中国科学技术大学);Hefei Comprehensive National Science Center(合肥综合性国家科学中心)
KEditVis: A Visual Analytics System for Knowledge Editing of Large Language Models
Zhejiang University(浙江大学);State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD&CG国家重点实验室);Hong Kong University of Science and Technology(香港科技大学);School of Software Engineering, South China University of Technology(华南理工大学软件学院);School of Software Technology, Zhejiang University(浙江大学软件学院)