arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.06555cs.CLcs.AIcs.CVcs.LG
收藏

视觉模型与语言模型共享概念吗?一项向量空间对齐研究

Do Vision and Language Models Share Concepts? A Vector Space Alignment Study

University of Copenhagen(哥本哈根大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)

Jiaang Li, Yova Kementchedjhieva, Constanza Fierro, Anders Søgaard

AI总结:

针对预训练语言模型缺乏世界心智模型、难关联话语与现实的观点,该研究通过多类语言与视觉模型的实证评估,发现二者表征存在部分同构收敛性,为多模态处理及语言模型理解研究提供参考。

摘要:

大规模预训练语言模型(LMs)被认为“缺乏将话语与现实世界关联的能力”(Bender和Koller,2020),因为它们没有“世界的心智模型”(Mitchell和Krakauer,2023)。若果真如此,语言模型的表征应与视觉模型生成的表征无关联。我们对四类语言模型(BERT、GPT-2、OPT和LLaMA-2)与三种视觉模型架构(ResNet、SegFormer和MAE)开展了实证评估。实验表明,受离散性、多义性和频率影响,语言模型的表征会部分收敛至与视觉模型表征同构的状态。这一发现对多模态处理及语言模型理解相关争论(Mitchell和Krakauer,2023)均具有重要意义。

英文摘要:

Large-scale pretrained language models (LMs) are said to ``lack the ability to connect utterances to the world'' (Bender and Koller, 2020), because they do not have ``mental models of the world' '(Mitchell and Krakauer, 2023). If so, one would expect LM representations to be unrelated to representations induced by vision models. We present an empirical evaluation across four families of LMs (BERT, GPT-2, OPT and LLaMA-2) and three vision model architectures (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge towards representations isomorphic to those of vision models, subject to dispersion, polysemy and frequency. This has important implications for both multi-modal processing and the LM understanding debate (Mitchell and Krakauer, 2023).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑