Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models
State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);Tencent(腾讯);Center for Data Science, Peking University(北京大学数据科学中心);Center for Machine Learning Research, Peking University(北京大学机器学习研究中心)
Sen Ye, Mengde Xu, Shuyang Gu, Di He, Liwei Wang, Han Hu
Current research in multimodal models faces a key challenge where enhancing generative capabilities often comes at the expense of understanding, and vice versa. We analyzed this trade-off and identify the primary cause might be the potential conflict between generation and understanding, which creates a competitive dynamic within the model. To address this, we propose the Reason-Reflect-Refine (R3) framework. This innovative algorithm re-frames the single-step generation task into a multi-step process of "generate-understand-regenerate". By explicitly leveraging the model's understanding capability during generation, we successfully mitigate the optimization dilemma, achieved stronger generation results and improved understanding ability which are related to the generation process. This offers valuable insights for designing next-generation unified multimodal models. Code is available at https://github.com/sen-ye/R3.
Navigating Gigapixel Pathology Images with Large Multimodal Models
Department of Biomedical Informatics, Harvard Medical School(哈佛医学院生物医学信息学系);Department of Pathology, Massachusetts General Hospital(麻省总医院病理学系);Department of Pathology and Laboratory Medicine, Brown University(布朗大学病理学与实验室医学系)
Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model
Zhejiang University(浙江大学);Tsinghua University(清华大学);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences)(教育部计算电力网络与信息安全重点实验室,山东计算机科学中心(国家超算中心济南中心),齐鲁工业大学(山东科学院));Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science(山东省计算电力互联网与服务计算重点实验室,山东省计算机科学基础研究中心);University of Electronic Science and Technology of China(电子科技大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);RWTH Aachen University(亚琛工业大学)
Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Models
College of Intelligent Robotics and Advanced Manufacturing(智能机器人与先进制造学院);Fudan University(复旦大学);Fysics Intelligence Technologies Co., Ltd. (Fysics AI)(Fysics智能技术有限公司(Fysics AI));School of Computer Science and Engineering(计算机科学与工程学院);Central South University(中南大学)
3UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
School of Information and Communication Engineering, Dalian University of Technology(大连理工大学信息与通信工程学院);School of Artificial Intelligence, Dalian University of Technology(大连理工大学人工智能学院)
Scalable Residual Feature Aggregation Framework with Hybrid Metaheuristic Optimization for Robust Early Pancreatic Neoplasm Detection in Multimodal CT Imaging
XVTP3D: Cross-view Trajectory Prediction Using Shared 3D Queries for Autonomous Driving
University of Chinese Academy of Sciences(中国科学院大学);SenseTime(商汤科技);Intuitive Surgical(直觉外科公司);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)