隐式多模态对齐：关于冻结LLM向多模态输入的泛化性研究
Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
AI总结：
本研究探究冻结大语言模型的多模态泛化机制，发现其内部存在隐式多模态对齐现象，该现象与架构相关，可用于评估模型性能、解释幻觉问题，还能支撑推理加速与模型压缩方案。
摘要：
大语言模型（LLM）在未经过任何多模态微调的情况下，已在多模态任务中展现出出色性能，是构建大多模态模型的基础，但目前学界对其成功的原因仍缺乏清晰认知。本研究将冻结LLM暴露于图像、视频、音频和文本输入下，分析其内部表征，旨在理解LLM在文本输入之外的泛化机制。\n研究发现：第一，感知token在LLM内部与文本token极易区分，二者表征存在显著差异，且不存在感知token到文本token的完全转换；第二，感知token与文本token会激活相似的LLM权重；第三，尽管存在差异，二者在LLM内部是隐式对齐的，研究者将该机制命名为隐式多模态对齐（IMA），并认为其与模型架构设计相关，是帮助LLM实现泛化的关键，这也为“LLM向多模态输入的泛化主要源于其架构”这一观点提供了更多证据。\n研究启示：第一，隐式对齐分数与任务性能呈正相关，可作为模型评估与选择的代理指标；第二，该分数与幻觉现象呈负相关，说明幻觉问题主要源于内部感知表征与文本表征的不对齐；第三，感知token在模型各层间变化极小，因此研究者提出了跳过（如FFN层中）部分计算的不同方案，可显著降低推理成本；第四，由于嵌入在各层间变化缓慢，且文本与多模态激活的权重高度重叠，研究者通过仅保留1个可在广泛多模态任务中良好运行的子网络实现了LLM压缩。论文代码：https://github.com/mshukor/ima-lmms。
英文摘要：
Large Language Models (LLMs) have demonstrated impressive performance on multimodal tasks, without any multimodal finetuning. They are the building block for Large Multimodal Models, yet, we still lack a proper understanding of their success. In this work, we expose frozen LLMs to image, video, audio and text inputs and analyse their internal representation aiming to understand their generalization beyond textual inputs. Findings. Perceptual tokens (1) are easily distinguishable from textual ones inside LLMs, with significantly different representations, and complete translation to textual tokens does not exist. Yet, (2) both perceptual and textual tokens activate similar LLM weights. Despite being different, (3) perceptual and textual tokens are implicitly aligned inside LLMs, we call this the implicit multimodal alignment (IMA), and argue that this is linked to architectural design, helping LLMs to generalize. This provide more evidence to believe that the generalization of LLMs to multimodal inputs is mainly due to their architecture. Implications. (1) We find a positive correlation between the implicit alignment score and the task performance, suggesting that this could act as a proxy metric for model evaluation and selection. (2) A negative correlation exists regarding hallucinations, revealing that this problem is mainly due to misalignment between the internal perceptual and textual representations. (3) Perceptual tokens change slightly throughout the model, thus, we propose different approaches to skip computations (e.g. in FFN layers), and significantly reduce the inference cost. (4) Due to the slowly changing embeddings across layers, and the high overlap between textual and multimodal activated weights, we compress LLMs by keeping only 1 subnetwork that works well across a wide range of multimodal tasks. Paper code: https://github.com/mshukor/ima-lmms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
