arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.19872cs.CV
收藏

面向增强知识协作的统一多模态编辑

Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration

Zhejiang University(浙江大学);National University of Singapore(新加坡国立大学);Hefei University of Technology(合肥工业大学);Nanyang Technological University(南洋理工大学);Singapore Management University(新加坡管理大学)

Kaihang Pan, Zhaoyu Fan, Juncheng Li, Qifan Yu, Hao Fei, Siliang Tang, Richang Hong, Hanwang Zhang, Qianru Sun

AI总结:

针对多模态大语言模型知识编辑难以平衡可靠性、通用性与局部性的问题,提出UniKE统一编辑范式,将两类知识建模为向量化键值记忆并解耦表示促进协作,实验验证其能同时保障三大特性。

摘要:

多模态大语言模型(Multimodal LLMs, MLLMs)的快速发展也给有效的知识编辑带来了重大挑战。现有方法包括内在知识编辑和外部知识调用两类,各有优劣,应用于MLLMs时难以平衡可靠性、通用性和局部性这三大理想特性。本文提出了一种新型多模态编辑方法UniKE,为内在知识编辑与外部知识调用建立了统一的视角与范式。两类知识都被概念化为向量化的键值记忆,对应的编辑过程类似于人类认知中的同化与顺应阶段,在相同的语义层级上执行。在这一统一框架内,我们进一步通过将知识表示解耦为语义空间和真实性空间来促进知识协作。大量实验验证了该方法的有效性,确保编辑后的MLLM能同时保持出色的可靠性、通用性和局部性。UniKE的代码可在https://github.com/beepkh/UniKE获取。

英文摘要:

The swift advancement in Multimodal LLMs (MLLMs) also presents significant challenges for effective knowledge editing. Current methods, including intrinsic knowledge editing and external knowledge resorting, each possess strengths and weaknesses, struggling to balance the desired properties of reliability, generality, and locality when applied to MLLMs. In this paper, we propose UniKE, a novel multimodal editing method that establishes a unified perspective and paradigm for intrinsic knowledge editing and external knowledge resorting. Both types of knowledge are conceptualized as vectorized key-value memories, with the corresponding editing processes resembling the assimilation and accommodation phases of human cognition, conducted at the same semantic levels. Within such a unified framework, we further promote knowledge collaboration by disentangling the knowledge representations into the semantic and truthfulness spaces. Extensive experiments validate the effectiveness of our method, which ensures that the post-edit MLLM simultaneously maintains excellent reliability, generality, and locality. The code for UniKE is available at \url{https://github.com/beepkh/UniKE}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑