Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration
Zhejiang University(浙江大学);National University of Singapore(新加坡国立大学);Hefei University of Technology(合肥工业大学);Nanyang Technological University(南洋理工大学);Singapore Management University(新加坡管理大学)
The swift advancement in Multimodal LLMs (MLLMs) also presents significant challenges for effective knowledge editing. Current methods, including intrinsic knowledge editing and external knowledge resorting, each possess strengths and weaknesses, struggling to balance the desired properties of reliability, generality, and locality when applied to MLLMs. In this paper, we propose UniKE, a novel multimodal editing method that establishes a unified perspective and paradigm for intrinsic knowledge editing and external knowledge resorting. Both types of knowledge are conceptualized as vectorized key-value memories, with the corresponding editing processes resembling the assimilation and accommodation phases of human cognition, conducted at the same semantic levels. Within such a unified framework, we further promote knowledge collaboration by disentangling the knowledge representations into the semantic and truthfulness spaces. Extensive experiments validate the effectiveness of our method, which ensures that the post-edit MLLM simultaneously maintains excellent reliability, generality, and locality. The code for UniKE is available at \url{https://github.com/beepkh/UniKE}.
Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language Models
Hefei University of Technology(合肥工业大学);University of Science and Technology of China(中国科学技术大学);Hefei Comprehensive National Science Center(合肥综合性国家科学中心)
UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing
Center for Data Science, Peking University(北京大学数据科学中心);Alibaba Group(阿里巴巴集团);CASIA;Center for Machine Learning Research, Peking University(北京大学机器学习研究中心);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室)
InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing
Shanghai AI Laboratory(上海人工智能实验室);Fudan University(复旦大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);South China University of Technology(华南理工大学);Nanjing University(南京大学);Xiamen University(厦门大学);CUHK MMLab(香港中文大学MMLab)
Uncovering Entity Identity Confusion in Multimodal Knowledge Editing
New Laboratory of Pattern Recognition (NLPR)(模式识别新实验室);State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)(多模态人工智能系统国家重点实验室);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing University of Posts and Telecommunications(北京邮电大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);Huazhong University of Science and Technology(华中科技大学);Shandong University(山东大学)
Multimodal Image Synthesis and Editing: The Generative AI Era
Max Planck Institute for Informatics(马克斯·普朗克信息研究所);S-Lab, Nanyang Technological University(南洋理工大学S实验室);Nanyang Technological University(南洋理工大学);Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing
Harbin Institute of Technology(哈尔滨工业大学);Pengcheng Laboratory(鹏城实验室);Shenzhen University of Advanced Technology(深圳先进技术大学);Northwestern Polytechnical University(西北工业大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editing
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
Towards Efficient Exemplar Based Image Editing with Multimodal VLMs
Indian Institute of Technology, Bombay(印度理工学院班加罗尔分校);Indian Institute of Technology, Roorkee(印度理工学院罗尔基分校);Microsoft Research(微软研究院);Stanford University(斯坦福大学);Adobe MDSR(Adobe MDSR实验室);Carnegie Mellon University(卡内基梅隆大学)
SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models
University of Macau(澳门大学);ARC Lab, Tencent PCG(腾讯平台与内容事业群ARC实验室);Tencent AI Lab(腾讯人工智能实验室);Shenzhen Institute of Advanced Technology(深圳先进技术研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);Tsinghua University(清华大学)
Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing
Media Integration and Communication Center (MICC), University of Florence(佛罗伦萨大学媒体集成与传播中心(MICC));University of Florence(佛罗伦萨大学);University of Modena and Reggio Emilia(摩德纳-雷焦艾米利亚大学);University of Pisa(比萨大学)
FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing
College of Computer Science and Artificial Intelligence, Wenzhou University(温州大学计算机与人工智能学院);Memorial University of Newfoundland(纽芬兰纪念大学);State Key Laboratory of CAD&CG, Zhejiang University(浙江大学CAD&CG国家重点实验室)
DPDEdit: Detail-Preserved Diffusion Models for Multimodal Fashion Image Editing
Shenzhen Technology University(深圳科技大学);University of Washington(华盛顿大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院)
ReEdit: Multimodal Exemplar-Based Image Editing with Diffusion Models
Indian Institute of Technology, Roorkee(鲁尔基印度理工学院);Adobe MDSR;Microsoft Research(微软研究院);Indian Institute of Technology, Bombay(孟买印度理工学院);Stanford University(斯坦福大学);Carnegie Mellon University(卡内基梅隆大学)