arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.02243cs.AI
收藏

当模态冲突时:单模态推理不确定性如何主导多模态大语言模型中的偏好动态

When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs

Peking University(北京大学);Provable Responsible AI and Data Analytics (PRADA) Lab(可证明负责任的人工智能和数据分析实验室);King Abdullah University of Science and Technology(卡布斯大学);South China University of Technology(华南理工大学);Tsinghua University(清华大学);University of Georgia(佐治亚大学);MBZUAI(马克斯·普朗克人工智能研究所)

Zhuoran Zhang, Tengyue Wang, Xilin Gong, Yang Shi, Haotian Wang, Di Wang, Lijie Hu

AI总结:

本文提出将多模态大模型解决模态冲突的模态跟随行为分解为相对推理不确定性和固有模态偏好,通过可控数据集和熵指标揭示模态跟随概率随相对不确定性单调递减的规律,并探查逐层预测解释了模型在平衡点附近的振荡机制。

摘要:

多模态大语言模型(MLLMs)在不同模态提供矛盾信息时必须解决冲突,我们将此过程称为模态跟随。先前的工作仅通过粗略的数据集级统计数据来衡量这一行为,忽略了模型对单模态推理置信度的影响。在本文中,我们引入了一个新框架,将模态跟随分解为两个基本因素:相对推理不确定性(单模态预测之间特定于案例的置信度差距)和固有模态偏好(不确定性平衡时模型的稳定偏差)。为了验证该框架,我们构建了一个可控数据集,系统地改变视觉和文本输入的推理难度。使用熵作为细粒度不确定性指标,我们揭示了一个普遍规律:跟随某一模态的概率随着其相对不确定性的增加而单调递减。在模型倾向于以相近概率跟随两种模态的相对难度水平(我们称之为平衡点),这是模型固有偏好的实用指标。与传统的宏观比率不同,该指标提供了一种更有原则且混淆更少的方法来表征模态偏差,将其与单模态能力和数据集伪影解耦。此外,通过探查逐层预测,我们揭示了振荡的内部机制:在平衡点附近的模糊区域,模型在各层之间在模态间摇摆不定,解释了外部观察到的犹豫不决。这些发现确立了相对不确定性和固有偏好作为模态跟随的两大主导原则,为 MLLMs 解决冲突信息提供了定量框架和机制见解。

英文摘要:

Multimodal large language models (MLLMs) must resolve conflicts when different modalities provide contradictory information, a process we term modality following. Prior work measured this behavior only with coarse dataset-level statistics, overlooking the influence of model's confidence in unimodal reasoning. In this paper, we introduce a new framework that decomposes modality following into two fundamental factors: relative reasoning uncertainty (the case-specific confidence gap between unimodal predictions) and inherent modality preference( a model's stable bias when uncertainties are balanced). To validate this framework, we construct a controllable dataset that systematically varies the reasoning difficulty of visual and textual inputs. Using entropy as a fine-grained uncertainty metric, we uncover a universal law: the probability of following a modality decreases monotonically as its relative uncertainty increases. At the relative difficulty level where the model tends to follow both modalities with comparable probability what we call the balance point, a practical indicator of the model's inherent preference. Unlike traditional macro-level ratios, this measure offers a more principled and less confounded way to characterize modality bias, disentangling it from unimodal capabilities and dataset artifacts. Further, by probing layer-wise predictions, we reveal the internal mechanism of oscillation: in ambiguous regions near the balance point, models vacillate between modalities across layers, explaining externally observed indecision. Together, these findings establish relative uncertainty and inherent preference as the two governing principles of modality following, offering both a quantitative framework and mechanistic insight into how MLLMs resolve conflicting information.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑