arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.07474cs.CLcs.AI
收藏

跨模态分类泛化在(视觉-)语言模型中

Cross-Modal Taxonomic Generalization in (Vision-) Language Models

Toyota Technological Institute at Chicago(芝加哥丰田技术研究所);The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Tianyang Xu, Marcelo Sandoval-Castaneda, Karen Livescu, Greg Shakhnarovich, Kanishka Misra

AI总结:

研究探讨了语言模型在无显性证据情况下通过跨模态泛化恢复超范畴知识的能力,发现语言模型能基于语言线索实现泛化。

摘要:

什么是语言模型通过表面形式学习的语义表示与通过更基础证据学习的语义表示之间的相互作用?我们研究了在输入来自不同模态(如视觉-语言模型)的场景中,其中预训练语言模型与预训练图像编码器对齐的情况。作为案例研究,我们聚焦于预测图像中物体的超范畴(hypernym)。我们采用VLM设置,保持图像编码器和语言模型冻结,仅学习中间映射。我们逐步剥夺VLM对超范畴的显性证据,并测试语言模型是否能恢复超范畴知识。我们发现,所研究的语言模型即使在最极端的实验版本(训练时无超范畴证据)中也能恢复此知识并实现泛化。额外实验表明,这种跨模态分类泛化仅在反事实图像-标签映射具有每个类别内高视觉相似性时持续存在。综合来看,这些发现表明,语言模型中的跨模态泛化源于外部非语言输入的一致性和从语言线索中获得的知识。

英文摘要:

What is the interplay between semantic representations learned by language models (LM) from surface form alone to those learned from more grounded evidence? We study this question for a scenario where part of the input comes from a different modality -- in our case, in a vision-language model (VLM), where a pretrained LM is aligned with a pretrained image encoder. As a case study, we focus on the task of predicting hypernyms of objects represented in images. We do so in a VLM setup where the image encoder and LM are kept frozen, and only the intermediate mappings are learned. We progressively deprive the VLM of explicit evidence for hypernyms, and test whether knowledge of hypernyms is recoverable from the LM. We find that the LMs we study can recover this knowledge and generalize even in the most extreme version of this experiment (when the model receives no evidence of a hypernym during training). Additional experiments suggest that this cross-modal taxonomic generalization persists under counterfactual image-label mappings only when the counterfactual data have high visual similarity within each category. Taken together, these findings suggest that cross-modal generalization in LMs arises as a result of both coherence in the extralinguistic input and knowledge derived from language cues.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑