缓解多模态推理中的模态不平衡问题
Mitigating Modal Imbalance in Multimodal Reasoning
AI总结：
该研究针对多模态推理中的模态不平衡问题，发现FMs跨模态冲突识别能力远低于单模态，源于跨模态注意力不对称，提出在训练样本中显式结合多模态的方法可降低不平衡并提升视觉-语言基准性能。
摘要：
部署在计算机使用智能体等现实任务中的基础模型（Foundation Models, FMs）必须整合多种模态。FMs在执行联合推理、同时对多种模态进行推理时表现如何？尤其是当模态之间相互作用、相互关联形成跨模态语境时。为更好地理解这一问题，我们研究了FMs在跨模态冲突下的表现：即不同模态呈现相互矛盾证据的场景。这让我们得以检验FMs是会优先考虑某一模态，还是通过联合推理来调和冲突。实验表明，FMs在单模态语境（由单一模态构成）中识别冲突的准确率可达90%，但当证据分散在不同模态时，这一比例低至3%——在由多种语言构成的跨语言语境中也观察到了类似结果。我们将这一失效归因于跨模态注意力不平衡，发现FMs的注意力得分存在极端不对称性，不成比例地优先关注特定模态。研究表明，盲目扩大多模态或多语言数据集规模并不能消除跨模态注意力不平衡，因为这些数据集缺乏明确需要跨模态推理的训练样本。我们证明，即便是一种简单且可扩展的方法——在每个训练样本中显式结合多种模态——也能显著降低注意力不平衡。注意力不平衡的降低直接转化为多个视觉-语言基准测试上的下游性能提升。我们的研究结果强调了系统解决跨模态语境问题对于构建可靠基础模型的重要性。
英文摘要：
Foundation models (FMs) deployed in real-world tasks such as computer-use agents must integrate diverse modalities. How good are FMs at performing joint reasoning, simultaneously reasoning over multiple modalities, especially when the modalities interact and relate to each other to form cross-modal context? To better understand this problem, we study FMs on cross-modal conflicts: scenarios where conflicting evidence is presented across modalities. This allows us to examine whether FMs prioritize one modality over another or reason jointly to reconcile the conflict. Our experiments reveal that FMs can recognize conflicts in unimodal contexts, composed of a single modality, 90% of the time, but the ratio falls as low as 3% when evidence is split across modalities -- similar observations hold in cross-lingual contexts, composed of multiple languages. We trace this failure to cross-modal attention imbalance, showing that FMs exhibit extreme asymmetry in attention scores, disproportionately prioritizing certain modalities. We show that cross-modal attention imbalance does not go away by simply scaling up multimodal or multilingual datasets blindly, since they lack training examples that explicitly require cross-modal reasoning. We demonstrate that even a simple and scalable method of explicitly combining multiple modalities within each training instance significantly reduces attention imbalance. Reduced attention imbalance directly translates to improved downstream performance on several vision-language benchmarks. Our findings underscore the importance of systematically addressing cross-modal contexts to build reliable foundation models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
