arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.08017cs.CVcs.AI
收藏

文本和图像的表示从第一层对齐

Representations of Text and Images Align From Layer One

D-INFK, ETH Zurich, Switzerland(苏黎世联邦理工学院信息与知识系统研究所);Aisle Research(Aisle研究)

Evžen Wybitul, Javier Rando, Florian Tramèr, Stanislav Fort

AI总结:

该研究通过合成方法证明,视觉-语言模型中图像和文本表示在第一层即可实现对齐,为模型可解释性提供了新路径。

摘要:

我们证明,在基于适配器的视觉-语言模型中,各种概念的图像和文本描述的表示在非常第一层就具有有意义的对齐。这与传统观点相矛盾,即这种图像-文本对齐只出现在后期层。我们通过一种新的合成方法来展示这一点,该方法受DeepDream启发:给定一个文本概念,如“木星”,我们提取其在特定层的概念向量,然后通过优化合成一个图像,其表示与该向量对齐。我们对Gemma 3中的7个层的数百个概念应用了我们的方法,并发现合成的图像往往描绘出目标文本概念的显著视觉特征:例如,在第一层,超过50%的图像描绘出可识别的动物、活动或季节特征。我们的方法因此提供了直接且具有建设性的证据,证明在概念和层的基础上的图像-文本对齐。与以往测量多模态对齐的方法不同,我们的方法简单、快速,不需要辅助模型或数据集。它还提供了一种新的路径,通过回溯模型的图像处理组件来可视化模型的表示空间,从而提高模型的可解释性。

英文摘要:

We show that for a variety of concepts in adapter-based vision-language models, the representations of their images and their text descriptions are meaningfully aligned from the very first layer. This contradicts the established view that such image-text alignment only appears in late layers. We show this using a new synthesis-based method inspired by DeepDream: given a textual concept such as "Jupiter", we extract its concept vector at a given layer, and then use optimisation to synthesise an image whose representation aligns with that vector. We apply our approach to hundreds of concepts across seven layers in Gemma 3, and find that the synthesised images often depict salient visual features of the targeted textual concepts: for example, already at layer 1, more than 50 % of images depict recognisable features of animals, activities, or seasons. Our method thus provides direct, constructive evidence of image-text alignment on a concept-by-concept and layer-by-layer basis. Unlike previous methods for measuring multimodal alignment, our approach is simple, fast, and does not require auxiliary models or datasets. It also offers a new path towards model interpretability, by providing a way to visualise a model's representation space by backtracing through its image processing components.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑