arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2104.08066cs.CLcs.AI
收藏

视觉和语言模型中视觉扩展对自然语言理解的影响

Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models

National Institute of Informatics(国立信息学研究所);Graduate University for Advanced Studies(高等研究生院)

Taichi Iki, Akiko Aizawa

摘要:

一种用于创建视觉和语言(V&L)模型的方法是通过结构修改和V&L预训练来扩展语言模型。这种扩展旨在使V&L模型继承原始语言模型的自然语言理解(NLU)能力。为了看看这是如何实现的,我们建议使用NLU基准(GLUE)来评估V&L模型。我们比较了五个V&L模型,包括单流和双流模型,使用相同的预训练进行训练。双流模型,其更高的模态独立性,实现了约一倍的参数数量,预计将保持NLU的能力更好。我们的主要发现是,双流分数并没有太大的差异,比单流分数,与预期相反。进一步的分析表明,预训练导致NLU任务的性能下降,很少有例外。这些结果表明,采用单流结构和设计的预训练可能是一种有效的方法,以提高语言知识的维护V&L扩展。

英文摘要:

A method for creating a vision-and-language (V&L) model is to extend a language model through structural modifications and V&L pre-training. Such an extension aims to make a V&L model inherit the capability of natural language understanding (NLU) from the original language model. To see how well this is achieved, we propose to evaluate V&L models using an NLU benchmark (GLUE). We compare five V&L models, including single-stream and dual-stream models, trained with the same pre-training. Dual-stream models, with their higher modality independence achieved by approximately doubling the number of parameters, are expected to preserve the NLU capability better. Our main finding is that the dual-stream scores are not much different than the single-stream scores, contrary to expectation. Further analysis shows that pre-training causes the performance drop in NLU tasks with few exceptions. These results suggest that adopting a single-stream structure and devising the pre-training could be an effective method for improving the maintenance of language knowledge in V&L extensions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑