When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs
Peking University(北京大学);Provable Responsible AI and Data Analytics (PRADA) Lab(可证明负责任的人工智能和数据分析实验室);King Abdullah University of Science and Technology(卡布斯大学);South China University of Technology(华南理工大学);Tsinghua University(清华大学);University of Georgia(佐治亚大学);MBZUAI(马克斯·普朗克人工智能研究所)
Zhuoran Zhang, Tengyue Wang, Xilin Gong, Yang Shi, Haotian Wang, Di Wang, Lijie Hu
Multimodal large language models (MLLMs) must resolve conflicts when different modalities provide contradictory information, a process we term modality following. Prior work measured this behavior only with coarse dataset-level statistics, overlooking the influence of model's confidence in unimodal reasoning. In this paper, we introduce a new framework that decomposes modality following into two fundamental factors: relative reasoning uncertainty (the case-specific confidence gap between unimodal predictions) and inherent modality preference( a model's stable bias when uncertainties are balanced). To validate this framework, we construct a controllable dataset that systematically varies the reasoning difficulty of visual and textual inputs. Using entropy as a fine-grained uncertainty metric, we uncover a universal law: the probability of following a modality decreases monotonically as its relative uncertainty increases. At the relative difficulty level where the model tends to follow both modalities with comparable probability what we call the balance point, a practical indicator of the model's inherent preference. Unlike traditional macro-level ratios, this measure offers a more principled and less confounded way to characterize modality bias, disentangling it from unimodal capabilities and dataset artifacts. Further, by probing layer-wise predictions, we reveal the internal mechanism of oscillation: in ambiguous regions near the balance point, models vacillate between modalities across layers, explaining externally observed indecision. Together, these findings establish relative uncertainty and inherent preference as the two governing principles of modality following, offering both a quantitative framework and mechanistic insight into how MLLMs resolve conflicting information.
Correct When Paired, Wrong When Split: Decoupling and Editing Modality-Specific Neurons in MLLMs
School of Information Science and Engineering, Yunnan University(云南大学信息科学与工程学院);School of Software, Yunnan University(云南大学软件学院);National University of Singapore(新加坡国立大学);School of Engineering, Yunnan University(云南大学工程学院)
OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models
Harbin Institute of Technology(哈尔滨工业大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));BIAI, ZJUT & D5Data.ai(BIAI、浙江工业大学及D5Data.ai);National University of Singapore(新加坡国立大学)
Hybrid RAG-empowered Multi-modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-based Contract Approach
Guangdong University of Technology(广东工业大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Guangzhou University of Traditional Chinese Medicine(广州中医药大学);The Second Affiliated Hospital of Guangzhou University of Chinese Medicine(广州中医药大学第二附属医院)
Visual Question Decomposition on Multimodal Large Language Models
Munich Center for Machine Learning(慕尼黑机器学习中心);Technical University of Munich(慕尼黑工业大学);Amazon Web Services(亚马逊云计算服务);LMU Munich(慕尼黑大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Oxford(牛津大学)
Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing
School of Computer Science, Peking University(北京大学计算机科学学院);MOE Key Laboratory of Computational Linguistics, Peking University(北京大学教育部计算语言学重点实验室);National Key Laboratory for Multimedia Information Processing, Peking University(北京大学国家多媒体信息处理重点实验室)
PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models
Harbin Institute of Technology(哈尔滨工业大学);Pengcheng Laboratory(鹏城实验室);The Hong Kong Polytechnic University(香港理工大学);Sichuan University(四川大学);Zhejiang Normal University(浙江师范大学)
MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues
University of Notre Dame(诺丁汉大学);University of California, Los Angeles(加州大学洛杉矶分校);Georgia Institute of Technology(佐治亚理工学院);University of Montreal(蒙特利尔大学)
Darwinian Memory: A Training-Free Self-Regulating Memory System for GUI Agent Evolution
Didichuxing Co. Ltd(滴滴出行有限公司);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Tianjin University(天津大学);Sun Yat-sen University(中山大学);Fudan University(复旦大学)
True Multimodal In-Context Learning Needs Attention to the Visual Context
LMU Munich(慕尼黑大学);Technical University of Munich(慕尼黑技术大学);Siemens AG(西门子股份公司);University of Science and Technology of China(中国科学技术大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Konrad Zuse School of Excellence in Reliable AI (relAI)(Konrad Zuse可靠性人工智能卓越学院);University of Oxford(牛津大学)
Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT
Central South University(中南大学);University of Electronic Science and Technology of China(电子科技大学);Peng Cheng Laboratory(鹏城实验室);Nanjing University(南京大学);Microsoft(微软公司)
Proteo-R1: Reasoning Foundation Models for De Novo Protein Design
Stanford University(斯坦福大学);Shanghai Jiao Tong University(上海交通大学);Yale University(耶鲁大学);University of Notre Dame(圣母大学);University of Waterloo(滑铁卢大学);University of Chicago, USA(美国芝加哥大学);University of Hamburg, Germany(德国汉堡大学);University of New South Wales(新南威尔士大学);University of Tokyo(东京大学);Harvard University(哈佛大学);AWS AI, Amazon (work outside of Amazon)(亚马逊AWS人工智能(亚马逊外工作));Chinese University of Hong Kong(香港中文大学)