COMPKE: Complex Question Answering under Knowledge Editing
Provable Responsible AI and Data Analytics (PRADA) Lab(可证明负责任的人工智能与数据分析实验室);King Abdullah University of Science and Technology(卡布斯大学);Peking University(北京大学);South China University of Technology(华南理工大学);Sun Yat-sen University(中山大学)
Keyuan Cheng, Zijian Kan, Zhixian He, Zhuoran Zhang, Muhammad Asif Ali, Ke Xu, Lijie Hu, Di Wang
Knowledge Editing, which efficiently modifies the knowledge in large language models, has gathered great attention. Current benchmarks primarily use multi-hop question answering to assess and analyze newly injected or updated knowledge. However, we argue that these benchmarks fail to effectively evaluate how well the updated models apply this knowledge in real-life scenarios, particularly when questions require complex reasoning, involving one-to-many relationships or multi-step logical intersections. To fill in this gap, we introduce a new benchmark, COMPKE: Complex Question Answering under Knowledge Editing, which includes 11,924 complex questions that reflect real-life situations. We conduct an extensive evaluation of four knowledge editing methods on COMPKE, revealing that their effectiveness varies notably across different models. For instance, MeLLo attains an accuracy of 39.47 on GPT-4O-MINI, but this drops sharply to 3.83 on QWEN2.5-3B. We further investigate the underlying causes of these disparities from both methodological and model-specific perspectives. The datasets are available at https://github.com/kzjkzj666/CompKE.
Multi-hop Question Answering under Temporal Knowledge Editing
King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);South China University of Technology(华南理工大学);Ant Group(蚂蚁集团);SDAIA-KAUST AI(沙特数据与人工智能局-阿卜杜拉国王科技大学人工智能项目);Provable Responsible AI and Data Analytics (PRADA) Lab(可信赖负责任人工智能与数据分析实验室)
DRESSing Up LLM: Efficient Stylized Question-Answering via Style Subspace Editing
School of Computer Science, Peking University(北京大学计算机学院);Center on Frontiers of Computing Studies, Peking University(北京大学计算前沿研究中心);National Research and Engineering Center of Software Engineering, Peking University(北京大学国家软件工程研究与工程中心)
Avoiding Knowledge Edit Skipping in Multi-hop Question Answering with Guided Decomposition
State Key Laboratory for Novel Software Technology, Nanjing University, China(新型软件技术国家重点实验室,南京大学,中国);National Institute of Healthcare Data Science, Nanjing University, China(医疗数据科学国家研究院,南京大学,中国)
PokeMQA: Programmable knowledge editing for Multi-hop Question Answering
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);North Carolina State University(北卡罗来纳州立大学);Texas A&M University(德克萨斯农工大学);University of Georgia(佐治亚大学)
MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language
King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);King AbdulAziz University(阿卜杜勒阿齐兹国王大学);University of Science and Technology, China(中国科学技术大学);Shenzhen University(深圳大学)
StruEdit: Structured Outputs Enable the Fast and Accurate Knowledge Editing for Large Language Models
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of Science and Technology of China(中国科学技术大学)
LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments
Institute of High Performance Computing (IHPC)(高性能计算研究所);Nanyang Technological University(南洋理工大学);Centre for Frontier AI Research(前沿人工智能研究中心);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);Texas A&M University(德克萨斯农工大学);University of Stuttgart(斯图加特大学);University of California San Diego(加利福尼亚大学圣迭戈分校)
KEditVis: A Visual Analytics System for Knowledge Editing of Large Language Models
Zhejiang University(浙江大学);State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD&CG国家重点实验室);Hong Kong University of Science and Technology(香港科技大学);School of Software Engineering, South China University of Technology(华南理工大学软件学院);School of Software Technology, Zhejiang University(浙江大学软件学院)
FABLE: Fine-grained Fact Anchoring for Unstructured Model Editing
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络与信息安全学院)
Towards Human-Level Understanding of Complex Process Engineering Schematics: A Pedagogical, Introspective Multi-Agent Framework for Open-Domain Question Answering
School of Computer and Information Technology, Shanxi University(山西大学计算机与信息学院);Beijing Key Laboratory of Multimedia and Intelligent Software Technology, Beijing Institute of Artificial Intelligence, School of Information Science and Technology, Beijing University of Technology(北京人工智能研究院多媒体与智能软件技术重点实验室,北京理工大学信息科学技术学院);University of Manchester(曼彻斯特大学);Singapore University of Technology and Design(新加坡科技设计大学);ILCC, School of Informatics, University of Edinburgh(爱丁堡大学信息学院ILCC)
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
Nanjing University of Science and Technology(南京理工大学);Baidu Inc(百度公司);AIML, Adelaide University(阿德莱德大学AIML);SUTD(新加坡科技设计大学);Southeast University(东南大学);East China Normal University(华东师范大学);University of Oxford(牛津大学)
DySK-Attn: A Framework for Efficient, Real-Time Knowledge Updating in Large Language Models via Dynamic Sparse Knowledge Attention
Department of Computer Science, San Francisco State University, San Francisco, CA 94132, India(计算机科学系,圣何塞州立大学);Department of Computer Science and Engineering, Indian Institute of Technology Bombay, Mumbai 400076, India(印度班加罗尔理工学院计算机科学与工程系);Department of Computer Science and Engineering, Indian Institute of Technology Delhi, New Delhi 110016, India(印度德里理工学院计算机科学与工程系);Department of Computer Science and Automation, Indian Institute of Science, Bengaluru 560012, India(印度班加罗尔科学研究所计算机科学与自动化系);Machine Learning Lab, International Institute of Information Technology Hyderabad (IIIT-H), Hyderabad 500032, India(国际信息科技研究所海得拉巴分所机器学习实验室)
SSE-Bio: A Structured Self-Evolving Agent with Agentic Retrieval Policy for Multi-Hop Biomedical Reasoning
School of Computing Science, University of Glasgow(格拉斯哥大学计算科学学院);Brigham and Women’s Hospital, Harvard Medical School, Harvard University(哈佛大学医学院附属布莱根妇女医院);Language Technology Lab, University of Cambridge(剑桥大学语言技术实验室);School of Natural and Computing Science, University of Aberdeen(阿伯丁大学自然与计算科学学院);School of Cancer Sciences, University of Glasgow(格拉斯哥大学癌症科学学院);Cancer Research UK Scotland Institute(英国癌症研究苏格兰研究所)