arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.15580cs.AI
收藏

视觉如何转化为语言:多模态推理的逐层信息论分析

How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning

Duke kunshan University, Duke University(杜克昆山大学、杜克大学);The Chinese University of Hong Kong, Hong Kong, China(香港中文大学);Fudan University, Shanghai, China(复旦大学)

Hongxuan Wu, Yukun Zhang, Xueqing Zhou

AI总结:

本研究通过逐层信息论分析揭示多模态Transformer中视觉信息如何转化为语言,发现视觉独特信息早期峰值,语言独特信息在晚期层主导预测,跨模态协同较低,且任务依赖性强。

摘要:

当一个多模态Transformer回答一个视觉问题时,预测是由视觉证据、语言推理还是真正的跨模态计算驱动的?这种结构在不同层之间是如何演变的?我们通过基于部分信息分解(PID)的逐层框架来回答这个问题,该框架将每个Transformer层的预测信息分解为冗余、视觉独特、语言独特和协同组件。为了使PID在高维神经表示中可行,我们引入了PID流,一种结合降维、归一化流高斯化和闭式高斯PID估计的流程。将此框架应用于LLaVA-1.5-7B和LLaVA-1.6-7B在六个GQA推理任务上的表现,我们揭示了一种一致的模态转换模式:视觉独特信息在早期峰值并随深度衰减,语言独特信息在晚期层中激增,以解释最终预测的大约82%,而跨模态协同仍低于2%。这种轨迹在模型变体中高度稳定(逐层相关性>0.96),但强烈依赖任务,语义冗余决定了详细的信息指纹。为了建立因果性,我们进行了针对性的Image→Question注意力敲除,并发现破坏主要转换路径会引发可预测的视觉独特信息增加、补偿协同和总信息成本增加——这些效果在视觉依赖任务中最强,在高冗余任务中最弱。这些结果提供了信息论和因果层面的解释,说明如何在多模态Transformer中将视觉转化为语言,并为识别信息丢失的架构瓶颈提供了定量指导。

英文摘要:

When a multimodal Transformer answers a visual question, is the prediction driven by visual evidence, linguistic reasoning, or genuinely fused cross-modal computation -- and how does this structure evolve across layers? We address this question with a layer-wise framework based on Partial Information Decomposition (PID) that decomposes the predictive information at each Transformer layer into redundant, vision-unique, language-unique, and synergistic components. To make PID tractable for high-dimensional neural representations, we introduce \emph{PID Flow}, a pipeline combining dimensionality reduction, normalizing-flow Gaussianization, and closed-form Gaussian PID estimation. Applying this framework to LLaVA-1.5-7B and LLaVA-1.6-7B across six GQA reasoning tasks, we uncover a consistent \emph{modal transduction} pattern: visual-unique information peaks early and decays with depth, language-unique information surges in late layers to account for roughly 82\% of the final prediction, and cross-modal synergy remains below 2\%. This trajectory is highly stable across model variants (layer-wise correlations $>$0.96) yet strongly task-dependent, with semantic redundancy governing the detailed information fingerprint. To establish causality, we perform targeted Image$\rightarrow$Question attention knockouts and show that disrupting the primary transduction pathway induces predictable increases in trapped visual-unique information, compensatory synergy, and total information cost -- effects that are strongest in vision-dependent tasks and weakest in high-redundancy tasks. Together, these results provide an information-theoretic, causal account of how vision becomes language in multimodal Transformers, and offer quantitative guidance for identifying architectural bottlenecks where modality-specific information is lost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑