大语言模型能够弥补视觉表征的不足
LLMs Can Compensate for Deficiencies in Visual Representations
AI总结：
本文通过自注意力消融实验证明，基于CLIP的VLM中语言主干能够在视觉表征语境化不足时动态弥补其缺陷并恢复性能，为将更多视觉处理卸载到语言解码器的未来架构提供了依据。
摘要：
许多在一系列多模态任务中表现非常出色的视觉-语言模型（VLM），都建立在基于CLIP的视觉编码器之上，而已知这些编码器存在各种局限性。我们研究了这样一个假设：VLM中强大的语言主干通过将视觉特征语境化或加以丰富，来弥补可能较弱的视觉特征。我们使用三个基于CLIP的VLM，在一个精心设计的探测任务上进行了受控的自注意力消融实验。我们的研究结果表明，尽管存在已知的局限性，CLIP视觉表征仍能为语言解码器提供可直接读取的语义信息。然而，在视觉表征中语境化程度降低的场景下，语言解码器能够在很大程度上弥补这一不足并恢复性能。这表明VLM中存在一种动态的分工机制，并激励未来将更多视觉处理任务卸载给语言解码器的架构设计。
英文摘要：
Many vision-language models (VLMs) that prove very effective at a range of multimodal task, build on CLIP-based vision encoders, which are known to have various limitations. We investigate the hypothesis that the strong language backbone in VLMs compensates for possibly weak visual features by contextualizing or enriching them. Using three CLIP-based VLMs, we perform controlled self-attention ablations on a carefully designed probing task. Our findings show that despite known limitations, CLIP visual representations offer ready-to-read semantic information to the language decoder. However, in scenarios of reduced contextualization in the visual representations, the language decoder can largely compensate for the deficiency and recover performance. This suggests a dynamic division of labor in VLMs and motivates future architectures that offload more visual processing to the language decoder.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
