How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning
Duke kunshan University, Duke University(杜克昆山大学、杜克大学);The Chinese University of Hong Kong, Hong Kong, China(香港中文大学);Fudan University, Shanghai, China(复旦大学)
When a multimodal Transformer answers a visual question, is the prediction driven by visual evidence, linguistic reasoning, or genuinely fused cross-modal computation -- and how does this structure evolve across layers? We address this question with a layer-wise framework based on Partial Information Decomposition (PID) that decomposes the predictive information at each Transformer layer into redundant, vision-unique, language-unique, and synergistic components. To make PID tractable for high-dimensional neural representations, we introduce \emph{PID Flow}, a pipeline combining dimensionality reduction, normalizing-flow Gaussianization, and closed-form Gaussian PID estimation. Applying this framework to LLaVA-1.5-7B and LLaVA-1.6-7B across six GQA reasoning tasks, we uncover a consistent \emph{modal transduction} pattern: visual-unique information peaks early and decays with depth, language-unique information surges in late layers to account for roughly 82\% of the final prediction, and cross-modal synergy remains below 2\%. This trajectory is highly stable across model variants (layer-wise correlations $>$0.96) yet strongly task-dependent, with semantic redundancy governing the detailed information fingerprint. To establish causality, we perform targeted Image$\rightarrow$Question attention knockouts and show that disrupting the primary transduction pathway induces predictable increases in trapped visual-unique information, compensatory synergy, and total information cost -- effects that are strongest in vision-dependent tasks and weakest in high-redundancy tasks. Together, these results provide an information-theoretic, causal account of how vision becomes language in multimodal Transformers, and offer quantitative guidance for identifying architectural bottlenecks where modality-specific information is lost.
Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoning
The Hong Kong Polytechnic University(香港理工大学);Singapore Management University(新加坡国立大学);Shandong University(山东大学);University of Science and Technology of China(中国科学技术大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Faithful-First Reasoning, Planning, and Acting for Multimodal LLMs
Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology(香港科技大学);The Australian National University(澳大利亚国立大学);Fudan University(复旦大学);University of Sydney(悉尼大学)
PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Data Science and Artificial Intelligence Research Institute, China United Network Communications Group Co., Ltd.(中国联合网络通信集团有限公司数据科学与人工智能研究院)
NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs
College of Intelligent Robotics and Advanced Manufacturing, Fudan University(复旦大学智能机器人与先进制造学院);Zhongguancun Academy(中关村学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Tianjin University(天津大学);Nankai University(南开大学)
Dependency, Compression, and Synergy: A Unified Information-Theoretic View of Multimodal Learning
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics(西南财经大学计算机与人工智能学院);Beijing Humanoid Robot Innovation Center (X-Humanoid)(北京人形机器人创新中心(X-Humanoid));Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Electronic Science and Technology of China(电子科技大学)
UkrMeme: Multimodal Explainable Persuasion Detection in Ukrainian Memes
Ukrainian Catholic University(乌克兰天主教大学);Johannes Gutenberg University Mainz(美因茨约翰内斯·古滕贝格大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Centre for European Research in Trusted AI (CERTAIN)(欧洲可信人工智能研究中心)
ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources
Electrical and Computer Engineering University of California, Los Angeles(电气与计算机工程大学加州大学洛杉矶分校);DEVCOM Army Research Laboratory(国防部陆军研究实验室);University of California, Los Angeles(加州大学洛杉矶分校);Amazon(亚马逊)
Efficient Quantification of Multimodal Interaction at Sample Level
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China(中国人民大学语言学院);Tencent Data Platform(腾讯数据平台);Tsinghua University, Beijing, China(清华大学);Beijing Key Laboratory of Research on Large Models(北京大型模型研究关键实验室);Engineering Research Center of Next-Generation Intelligent Search(下一代智能搜索工程研究中心)
MILR: Improving Multimodal Image Generation via Test-Time Latent Reasoning
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室);Peking University(北京大学);Tsinghua University(清华大学);University of California, Los Angeles(加州大学洛杉矶分校)
Information-Theoretic Decomposition for Multimodal Interaction Learning
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing(中国人民大学人工智能学院,北京);Beijing Key Laboratory of Research on Large Models(北京大模型研究关键实验室);Engineering Research Center of Next-Generation Intelligent Search(下一代智能搜索与推荐工程研究中心);Beihang University, Beijing(北航,北京);Gaotu Techedu Inc.(高图科技有限公司)