Numerous studies have shown that multimodal LLMs process speech and images well but fail in non-intuitive ways rendering trivial tasks such as object counting unreliable. We investigate this behavior from an information-theoretic perspective by framing multimodal LLM inference as a mismatched decoder problem: a decoder trained primarily on text can only extract information along text-aligned directions (removing up to 98% of the variation in modality-specific (non-text) directions improves decoder loss) and the amount of accessible information is bounded by the Generalized Mutual Information (GMI). We show that information loss is bounded as the distributional mismatch between the source data and the text data increases, and as the sensitivity of the decoder increases. This bound is a function of the model's scoring rule not its architecture. We validate the predictions across five models spanning speech and vision. A controlled study (two Prismatic VLMs differing only in encoder text-alignment) shows that the bottleneck lies in the scoring rule of the decoder rather than the text-alignment of the encoder or the learned projection. A LoRA intervention demonstrates that simply training with an emotion-related objective improves emotion detection from 17.3% to 61.8% task accuracy without affecting other attributes, confirming that the training objective determines what becomes accessible.
Cross-Modal Attention Calibration for LVLM Hallucination Mitigation
Sun Yat-sen University（中山大学）；The University of Hong Kong（香港大学）；Meituan（美团）；Inspur Database Technology（Inspur数据库技术）；Guilin University of Electronic Technology（桂林电子科技大学）；Shenzhen Loop Area Institute（深圳河套学院）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）
Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder
School of Computer Science and Technology, Harbin Institute of Technology Shenzhen（哈尔滨工业大学（深圳）计算机科学与技术学院）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science, Fudan University（复旦大学计算机科学学院）
Large-scale pre-trained multi-modal models (e.g., CLIP) demonstrate strong zero-shot transfer capability in many discriminative tasks. Their adaptation to zero-shot image-conditioned text generation tasks has drawn increasing interest. Prior arts approach to zero-shot captioning by either utilizing the existing large language models (e.g., GPT-2) or pre-training the encoder-decoder network in an end-to-end manner. In this work, we propose a simple framework, named DeCap, for zero-shot captioning. We introduce a lightweight visual-aware language decoder. This decoder is both data-efficient and …
Jailbreaking Vision-Language Models Through the Visual Modality
Warsaw University of Technology（华沙理工大学）；NASK National Research Institute（波兰国家研究研究院）；University of Liverpool（利物浦大学）；Indian Institute of Technology Roorkee（印度理工学院拉胡尔分校）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models
Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Kuaishou Technology（快手科技）；School of Computer Science and Technology, Beihang University（北航计算机科学与技术学院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Chongqing University（重庆大学）；Wuhan University（武汉大学）
Simultaneous translation involves translating a sentence before the speaker's utterance is completed in order to realize real-time understanding in multiple languages. This task is significantly more challenging than the general full sentence translation because of the shortage of input information during decoding. To alleviate this shortage, we propose multimodal simultaneous neural machine translation (MSNMT), which leverages visual information as an additional modality. Our experiments with the Multi30k dataset showed that MSNMT significantly outperforms its text-only counterpart in more ti…
Decoding Student Minds: Leveraging Conversational Agents for Psychological and Learning Analysis
STIH Laboratory, Sorbonne University（索邦大学STIH实验室）；LIPAH-LR11ES14, National Engineering School of Tunis（突尼斯国家工程学院LIPAH-LR11ES14）；Faculty of Sciences of Tunis（突尼斯科学学院）；LINFI Laboratory（LINFI实验室）
A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings
Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong Baptist University（香港 Baptist大学）
Transferable Decoding with Visual Entities for Zero-Shot Image Captioning
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Tencent（腾讯）；Shanghai Jiao Tong University（上海交通大学）；Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
